It's not that the big players make it difficult for you to send/receive from "your own domain". It's more the underlying provider infrastructure. If you pick basically any mail provider, it'll work out just fine also with your own domain. And yes, even if you were to completely run your own mail stack it works okay if you follow all the best practices.
I run Mailcast (https://mailcast.io) which will forward email on your domain to your Gmail account. We have domain aliasing to make it easy to add multiple domains with the same addresses, and you can configure sending on your domain as well.
I'd love to leave the Gmail/Outlook universe as much as the next guy, but I don't think a Google Workspace account at $8/month counts as "big bucks" does it? And a custom-domain in Office365 with Outlook is $4/month(?) for the Web-only version (without the Office suite apps). Seems like a reasonable price for reliable email.
Fastmail, Proton, and Tuta exist as well.