Allow passing a GenerationConfig
for generative evals
#16
Labels
feature request
New feature/request
GenerationConfig
for generative evals
#16
For some benchmarks, it can be preferable to generate responses via sampling with non-zero temperature. Although this affect reproducibility, it could be useful for power users who are constructing an internal benchmark with
lighteval
One way to do this would be by allowing one to pass a
generation_config
field in the task metadata which is ultimately parsed bytransformers.GenerationConfig
(docs) before being passed to thegenerate()
methodThe text was updated successfully, but these errors were encountered: