-
Notifications
You must be signed in to change notification settings - Fork 534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update openai wrapper to work with tiktoken interface and newest openai version #794
Conversation
…ry into update_openai_wrapper
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! Could you please add a note to the yaml noting which tasks are supported with which models? (some are not supported because we can't get logits, right?)
with the changes, LGTM 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving to unblock the setup.py
change. Deferring the full review to Max.
Some changes to the OpenAI tokenizer (tiktoken) made the previously working e2e OpenAI eval wrapper no longer work with the eval script.
Some changes to the newest version of OpenAI API include deprecating the
openai.ChatCompletion.create
method in favor of first constructing a client and callingclient.chat.completion.create
. Additionally, return results are no longer dictionaries , but insteadopenai.types.Completion
objectsTested with
composer eval/eval.py eval/yamls/openai_eval.yaml
Ran on all of lm_tasks_v0.2:
openai-eval-O5OK4O