launch lighteval using lighteval --args

[NathanHB]

• removes uneeded script files to launch lighteval
• moves parsers function to lighteval.commands.parsers
• creates a lighteval cli executable with 3 subcommands (list-tasks, nanotron, accelerate)
• removes the code used to list tasks and puts it in the Registry class

how to use:

with nanotron

nothing changes except for the use of the cli executable

 lighteval nanotron  {nanotron args}

with accelerate

accelerate args now need to be passed to the lighteval script.

accelerate launch --multi_gpu --num_processes=8 -m \
lighteval accelerate \
--model_args dummy \
--tasks "original|mmlu:anatomy|0|0" \
--override_batch_size 1  \
--output_dir="tmp/" \

[clefourrier]

LGTM, super useful

[clefourrier]

I'm not fond of accelerate ... lighteval accelerate

[NathanHB]

I agree but not sure how we could really differenciate between nanotron and accelerate launchers

[clefourrier]

well, can you launch nanotron with accelerate?

[NathanHB]

wdym ?

[clefourrier]

I don't think we'll ever get accelerate launch ... lighteval nanotron

[clefourrier]

So we could assume that if accelerate launch is used, then it's lighteval accelerate?