Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow passing a GenerationConfig for generative evals #16

Open
lewtun opened this issue Feb 6, 2024 · 1 comment · May be fixed by #428
Open

Allow passing a GenerationConfig for generative evals #16

lewtun opened this issue Feb 6, 2024 · 1 comment · May be fixed by #428
Assignees
Labels
feature request New feature/request

Comments

@lewtun
Copy link
Member

lewtun commented Feb 6, 2024

For some benchmarks, it can be preferable to generate responses via sampling with non-zero temperature. Although this affect reproducibility, it could be useful for power users who are constructing an internal benchmark with lighteval

One way to do this would be by allowing one to pass a generation_config field in the task metadata which is ultimately parsed by transformers.GenerationConfig (docs) before being passed to the generate() method

@clefourrier
Copy link
Member

@lewtun can you check #428 to tell me if there are other params you'd want to see?

@clefourrier clefourrier linked a pull request Dec 10, 2024 that will close this issue
clefourrier added a commit that referenced this issue Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature/request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants