Tabular files: semicolon is not supported as a separator by ".csv" files

[vkush]

Researches of catalysis are using semicolon ";" as a separator within .csv files, what is not supported during ingest of tabular files, where only comma "," is currently supported (because of "Comma Separated Values"). Based on Wikipedia, a .csv files can have also a different separators, not only a comma.

Issues are known and under discussion:

• semicolon separated values file with .csv extension fails ingest IQSS/dataverse#8990 (see also all related issues)
• Refactor and rethink the tabular ingest subsystem in v6. IQSS/dataverse#8526 (see also a comment about separator types and quoting mechanism)
• conversation in user group
  • looks like "comma" is hardcoded: https://github.com/IQSS/dataverse/blob/v6.4/src/main/java/edu/harvard/iq/dataverse/ingest/tabulardata/impl/plugins/csv/CSVFileReader.java#L93
  • see similar conversation about semicolon and missing first row

• Also we have to check, if "additional empty spaces" and "multiple-underscores" are interpreted properly, especially when all issues above are solved. With proper .csv with commas it is ok now, use test-file-spaces-underscores.csv as an example.

[dalito]

German researchers with not much data background may create files with ; as separator (and , as decimal separator). But it should be avoided!

The ; is unfortunately created by Excel with German locale setting. I also saw problems with date and time formats in the past when German locale is used.

Instead of adapting to such problematic csv files I would prefer a validator that rejects or at least flags incompatible csv files. Do the BasCat validation tools maybe address this? @khatamirad