API calls cannot have deterministic progress calculated in advance due to various factors like current system resources available, load, amount of data stored and various other things that can impact the time it takes to finish an API call. So I don't see how you can get a progress percentage out of such call. Spinners are the only way as of now. It is not always like a File upload where you know the size of the file and the chunks uploaded to the server already.
This is too complex, difficult, and case dependent to make it a standard that requires implementation by frameworks. Nothing in the system can know how long each other part will take, let alone what downstream parts will be used to serve a request... so what float value should it be sending back when?
I think you might be confusing application level and protocol level features? Of course you could always add custom headers (X-Wants-Progress) in your application. But the protocol (HTTP) doesn't really have insight into (or care about) any of your application context per se :)
at some point in the stack it is / should be fairly well
known how long something is going to take
Wasn't there a deprecation of the X- prefix in protocol headers?
Other thoughts:
You could probably make this play better with HTTP libraries by returning the header on an existing HTTP message, instead of changing the overall reply semantics. It's a little less efficient, but probably much more broadly compatible.
You could also have min-est-completion-time and max-est-completion-time rather than a percentage (partly to address people's concerns about the uncertainty or unknowability of overall progress as a percentage). Perhaps these could customarily be described as 90% intervals (or they could explicitly include a confidence estimate! like "90% of requests at this approximate degree of completion were completed within 7 and 32 seconds").