The bulleted list of what constitutes “ideal” is missing one of the most important types of questions: questions that aren’t answered by the knowledge set, but which seem like they should/might be.
This is where RAG systems consistently fall down. The end user, by definition, doesn’t know what you’ve got in your data. They won’t ask questions carefully cherry-picked from it. They’ll ask questions they need to know the answer to, and more often than you think, those answers won’t be in your data. You absolutely must know how your system behaves when they do that.
How does this differ from Ragas? https://docs.ragas.io/en/latest/index.html
Can it generate HTML as part of prompt?
Can I plug this into ragas pipeline
This is super cool!
I want to hack
For the people wondering, the Github repo is only hosting a couple of lines of Python to connect to their API.
If you have your own LLM, you may have sensitive/private data "in" it from your training. You may not be allowed to use this service from a legal point of view.