Have a look at Graphlit (https://www.graphlit.com). Managed cloud-native platform for knowledge ingestion and retrieval. Simple GraphQL API, and serverless backend to handle all the data ingestion, embeddings, and RAG LLM patterns. Free tier to try out without CC. (Disclaimer: I'm the founder)

We're working on an open source chat gpt that runs on kubernetes. https://github.com/bionic-gpt/bionic-gpt

And you're right many companies are looking at running LLMs in house.