Dynamic columns, writing only part of the data

[pavlexander]

Presumably I have a data of format:

public class MyData {
   public long Price { get; set; }
   public long Amount { get; set; }
   public string Name { get; set; }
}

How would you write only a part of the columns?

In example above I would like to write to file a Price column, but not Amount and Name. In other scenario I would like to write Price and Name, but not the Amount column data.. Is there a way of doing it? Presumably the feature that I am looking for is dynamic objects writing (or objects with dynamic sets of properties)...

Currently I am using the following approach which writes the full set of properties into file:

var builder = new WriterPropertiesBuilder();
builder.DisableDictionary();
builder.Compression(Compression.Snappy);
builder.Encoding(ParquetSharp.Encoding.DeltaBinaryPacked);
var properties = builder.Build();

using var writer = ParquetFile.CreateRowWriter<MyData>(fileFullPath, properties);
writer.WriteRows(data);
writer.Close();

I also tried using the dynamic objects like so:

     var records = new List<dynamic>();
     foreach (var item in data)
     {
         dynamic record = new ExpandoObject();
         record.price = item.Price;

                //dynamic record = new
                //{
                //    price = item.Price
                //};

         records.Add(record);
     }

     using var writer = ParquetFile.CreateRowWriter<dynamic>(fileNewPath, properties);
     writer.WriteRows(records);
     writer.Close();

but get the exception:

System.ArgumentException: 'Type 'System.Object' does not have any public fields or properties to map to Parquet columns, or any private fields or properties annotated with 'MapToColumnAttribute' (Parameter 'type')'

Preferably I would like to avoid initializing and creating a new collection just to have some of the columns exported. The best scenario would be to reuse the same MyData object when writing to file and avoiding some of the properties..

[adamreeve]

Hi @pavlexander, we don't currently have support for this with the row-oriented API. There's an open issue about skipping some properties of an object at #190, although the solution proposed there involves adding an annotation to properties so that wouldn't solve your problem of wanting to change which set of properties to include at runtime, and that issue is also about reading rather than writing.

I think for this problem it makes sense to use the lower-level column oriented API instead.