The following steps describe the process of performing a RAG evaluation test
- Initialization of Datasources: Setting up datasources that describe collections of documents.
- Document Handling: Adding documents to the datasources which will be used for generating questions.
- QASet Creation: Establishing QA sets that contain questions and their correct answers linked to specific documents.
- Question Generation: Questions are either loaded from a QASet source or generated from the Document. This is done once before the test runs.
- Execution of Test Runs: Running the tests to generate responses.
- Evaluation Setup: Configuring evaluation functions and test evaluations to assess the responses based on predefined metrics.
- Response Generation: Utilizing the RAG system to generate responses to the questions during test runs.
- Context Management: Associating responses with relevant document contexts.
- Execute Evaluation: Running the configured evaluation functions to generate scores and feedback for the responses.