mixed types in columns?

[matt-long]

Should we be setting a default type in DataFrame columns?

[andersy005]

@matt-long, is this still an issue?

[andersy005]

@matt-long,

While working on #66, I realized that this issue causes a lot of other issues when it comes to the .search() method (when the default value for a column is not compatible with the column data type). For example, if the column is supposed to contain integers, and None is used as a sentinel value, the column data type is cast to float. Assuming that this column is meant to contain years, a value of 2000 becomes 2000.0. As a result, the search() method becomes unreliable.

Should we be setting a default type in DataFrame columns?

This sounds like a step in the right direction though It may not be an easy task as the number of collections supported in intake-esm increases.

• Can we replace .csv with a different file format such as parquet or hdf5 that maintains the original column data types instead of infering them at runtime?
• One disadvantage of these formats is that they are not human-readable. Would this be a problem to the users?

[matt-long]

I don't see a problem with non-human-readable formats.