add 4o-mini and timing script #94
Conversation
![mentatai[bot]](https://avatars.githubusercontent.com/in/850097?s=60&v=4){kind=small}
There was a problem hiding this comment.
The pull request adds new GPT-4o models and introduces a timing script for performance comparison. The changes look good overall, with a few minor suggestions for improvement in the timing script to enhance its flexibility and output. The duplicate line in the models.py file should be addressed. Otherwise, this PR seems to provide valuable additions to the project.
Thanks for using MentatBot. Give comments a 👍 or 👎 to help me improve!
| "gpt-4o-mini-2024-07-18", OPEN_AI, input_cost=15, output_cost=60, context_length=128000 | ||
| ) | ||
|
|
||
| GPT_4o_2024_05_13 = TextModel("gpt-4o-2024-05-13", OPEN_AI, input_cost=500, output_cost=1500, context_length=128000) |
There was a problem hiding this comment.
This line appears to be a duplicate of line 75. Consider removing one of these declarations to avoid confusion and maintain code clarity.
| {"role": "user", "content": "list 100 random words"}, | ||
| ] | ||
|
|
||
| models = ["gpt-4o-mini", "gpt-4o"] |
There was a problem hiding this comment.
Consider making the list of models configurable, either through command-line arguments or environment variables. This would make the script more flexible for testing different model combinations.
| ] | ||
|
|
||
| models = ["gpt-4o-mini", "gpt-4o"] | ||
| runs = 5 |
There was a problem hiding this comment.
Similarly to the models, consider making the number of runs configurable. This would allow for easier adjustment of the test parameters without modifying the code.
| start = timer() | ||
| response = await client.get_response(messages=messages, model=model) | ||
| end = timer() | ||
| print(f" Run {i + 1}: {end - start:.1f}s, {response.characters_per_second:.0f} characters/s") |
There was a problem hiding this comment.
It might be helpful to add more statistics, such as the average time and characters per second across all runs for each model. This would provide a quick summary of the performance comparison.
No description provided.