-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Features/registry #162
Features/registry #162
Conversation
The way I see it, a Task depends on a Dataset to be created and a Reward model to be evaluated. Imho a config file could be a better place to position those configurations than raw python code, where the validator loads the definition of the "task framework" into the generic code that knows how to handle it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Besides minor logging and formatting changes, I believe this approach is less user friendly than relying in configs. This approach also don’t relate reward models to the task to be performed, which could be something to address in the future. In any case, it seems to be functional, if it’s currently working on prod and we have tests for it as it apparently is the case, we should be fine moving on with this approach.
The goal of this PR is to introduce a registry data structure to manage both tasks and datasets.