Parallel Dataset preparation

[parsa-ra]

I wonder how I could load a dataset using load_dataset function in a parallel fashion. Is there an automatic way to use the load_dataset in a parallel form or I have to modify the _generate_examples function to be parallel itself, I just wrote a customized loading script that tends to load and preprocess raw data and store them in pyarrow format ...

[mariosasko]

Hi! You can use num_proc in load_dataset to parallelize the generation process.