Everything becomes easier with standardization and process improvement. One of the first things you can do is create a list of pain points which you sort of started. Follow up with thinking about a process you can put in to help address some of the pain points. Don’t fix everything. Also come up with some way to measure your improvement and pin points. An easier way to think about this is people, process, and tools.
You are in a kobiyashi maru where you are being set up to fail. Your upper manager is indifferent to the situation, the fact your shop is 90% contractors means upper management is not willing to build a coherent group of dedicated people to who have skin in the game.
Switch from scrum to waterfall and force the PM to create a complete end-to-end soecification of how the damned thing is supposed to work. And don’t let PM prioritize what work is done first on the project.
Also, is product eventually being produced and successfully sold and money is made, and it’s just people bitching about velocity and story points?