Refactor Binary File Operations

[mali-git]

Feature request

Implement a centralized, reusable utility or module for handling binary file operations which can be used across modules. This utility should:

1. Standardize reading and writing headers, binary data, and indices.
2. Support modular integration with existing components like EmbeddedStreamData and others.
3. Reduce code duplication while improving readability and maintainability.

Motivation

Currently, there is duplicated code for reading and writing binary files across multiple modules and functions, including:

• EmbeddedStreamData
• PackedDataGenerator
• LargeFileLinesReader
• shuffle_tokenized_data()

This redundancy increases maintenance overhead and the risk of inconsistencies. For example, reading headers, writing index data, and handling binary streams are repeated in different forms, leading to potential bugs and inefficiencies.

[le1nux]

First version implemented here:
https://github.com/Modalities/modalities/blob/65477cdcd20a33a66cc30d5220def47b42e5b27f/src/modalities/dataloader/preprocessing/tokenization/tokenized_file_writer.py

Note that write_tokenized_dataset expects the full dataset as a list of numpy arrays already.
Iteratively adding rows is not possible for now.