Prompt Evaluator is an AI assistant tool specifically designed for product managers and developers. With a strong emphasis on versatility, our tool empowers users to effectively solve a wide range of problems tailored to their unique use cases. By enabling prompt template engineering and enhancement, we ensure that prompts are both easily testable and maintainable. We are constantly working towards addressing broader, more generic issues to provide a clear and user-centric solution that unleashes your full potential. Stay tuned for exciting updates as we continue to enhance our tool.
Table of Content
The experiment feature in our product allows users to create collections of prompt templates. These templates are for user-defined conversations that may include variables. Users can define their own conversations with various roles and prompts, incorporating variables where necessary. Users can evaluate the performance of prompts by executing them with different OpenAI models and associated test cases. By running prompt templates with different models and test cases, users gain valuable insights into the performance and suitability of their prompts for different scenarios.
Select the desired experiment to view its details. Read the experiment title, description, and other related details to understand its purpose and context.
To start a new experiment, just click on the "Create experiment" button. A brand new experiment will be created with a default title and description. The title and description are basic details that briefly explain what the experiment is about.
We can update the title for the experiment in two ways:
- Using the rename button associated with each experiment cell.
- Simply click on the title located at the top of the page.
We can add a clear and concise description in this section to explicitly state the experiment's goals and objectives by modifying the default description as shown below.
Step 1: Navigate to the Prompt Template section.
Step 2: Click on the "Add New Template" option.
Step 3: A page for creating a new template will appear, allowing users to define and customize the content, format, and variables used in the prompt templates, aligning them with the experiment's objectives and requirements.
To customize the prompt template, you have the flexibility to modify the content, format, and variables according to your requirements. In this process, you need to provide both the system role and user role based on your specific needs. When declaring variables in the conversation, please use the format {{variable_name}}. For a better understanding, refer to the example provided below.
To add a new role, simply click on the "Add message" button located above the create button.
To change the role of a conversation, you can toggle the role from within the conversation itself.
Step 4: Click the "Create" button at the bottom of the page to create the template.
You can see the newly created prompt template under the prompt template list.
Step 1: Click the prompt template cell you wish to update.
Step 2: Then, provide the necessary changes to the template, as demonstrated earlier when creating a new template.
Step 3: Save the template after making any changes or edits.
Step 1: Please click on the "Clone" button provided under each prompt template cell.
Step 2: After making the necessary changes to the template, clone it using the clone button.
Below, you will see a newly created clone template.
Step 1: Access the Test Case section or tab.
Step 2: Click on the "Add new test case".
Step 3: Specify the variable values, expected outputs, and additional parameters for each test case.
Add title and description as shown below:
Now, specify the template that should replace it, along with the corresponding value for the given variable name.
Add acceptable results which will be used to compare the actual and expected result to generate a report.
Step 4: Create the test case after defining the necessary details.
Step 1: Click on the run button provided under each prompt template cell.
Step 2: Choose the required model and evaluation from the drop down menu
Step 3: To initiate the process, please click the "Run" button.
Step 4: After a few seconds, a report will be generated, which you can then examine.
To view the report, click the view report button located under the desired prompt template cell.
Please review the side-by-side comparison of the expected and actual outcomes. Pay attention to any differences, problems with performance, or areas that need improvement. Using this analysis, make informed decisions and adjustments to optimize the experiment.
You can use the "Share Report" button found below the test case within the report. Alternatively, you can copy the link directly from the Google tab.