Add use_tokenizer_eos option to convert text to mds script

[irenedea]

Manual Test

(1) Make some test files

import os
os.mkdir('in')
for i in range(5):
    with open(f'in/file{i}.txt', 'w') as f:
        f.write(f' HELLO{i} ' * 250)

(2) Run conversion with --use_tokenizer_eos

`python convert_text_to_mds.py --input_folder in --output_folder out --concat_tokens 2048 --tokenizer mosaicml/mpt-7b --use_tokenizer_eos --processes 1`

(3) Print out 0th sample to see the <|endoftext|> token between text sequences

    tokenizer = AutoTokenizer.from_pretrained('mosaicml/mpt-7b')
    dataset = StreamingDataset(local='out', num_canonical_nodes=1)
    sample = dataset[0]
    tokens = np.frombuffer(sample['tokens'], dtype=int)
    decoded = tokenizer.decode(tokens)
    print(decoded)