Efficient Addition of New Columns in Large-Scale Feature Datasets

[liugs0213]

Feature Request / Improvement

In large-scale feature datasets, adding new columns (or features) can be inefficient if the entire dataset needs to be rewritten. A more efficient method would involve associating only the new feature data with existing records, without rewriting the whole dataset. This could significantly optimize storage and improve iteration speeds, especially in machine learning and ETL pipelines.

Key Considerations:

1. Storage Efficiency: Rather than rewriting the entire dataset when adding new columns, only the newly added columns should be written. This avoids duplication of existing data and minimizes storage consumption.
2. Efficient Query and Write Performance: Queries and writes should be optimized by linking new feature data to existing records without the need to reprocess or modify the entire dataset. This will maintain query performance while reducing unnecessary data movement.

Current Approaches:

Bytedance scheme: Whether the way of sorting according to the primary key is the best scheme.
https://developer.volcengine.com/articles/7260058755952279606

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time