Table file I/O

[apjanke]

Matlab's table support includes I/O functions for reading and writing tables to and from various file types:

I/O Functionality

• readtable - datetime support #27
• writetable - datetime support #27
• detectImportOptions
• getvaropts
• setvaropts
• setvartype - datetime support #27

It supports CSV, TSV/delimited, fixed-width fields, and Excel files. (And Open Office .ods files?)

Excel files are going to be the big issue here. They're a complex file format that's hard to parse, so we'll probably need to pull in a library to do this. Apache FOP would be the obvious choice for me. But that's Java, and Java is not available in all Octave builds, so Octave prefers not to have anything in core that depends on Java. And I'd like to see Tablicious make it in to core Octave some day.

This is going to be a big task.

References

• io package on Octave Packages
• o package on Octave Forge
  • io package doco on Octave Forge
  • io package news
• Savannah #44571 - table feature request for core Octave
• Savanna #15419 - Implement Table I/O

TODO

• Research C/C++ Excel I/O libraries.
• Research the Octave Forge io package to see if I can build on top of that.
  • For both Excel and regular csv/dlm reading.

[apjanke]

There was some discussion of this on the Octave Maintainers mailing list: https://lists.gnu.org/archive/html/octave-maintainers/2019-10/msg00071.html. Looks like there's an appetite for readtable/writetable. I'm bumping up the priority on this to High.

[eebrown]

Would the io package (https://octave.sourceforge.io/io/index.html) help provide some intermediate functions for readtable? E.g. to load a csv file, using csv2cell() from io and your cell2table() ?

[apjanke]

Would the io package (https://octave.sourceforge.io/io/index.html) help provide some intermediate functions for readtable?

Yes and no. ;) In theory and in a broad sense, yes, the io package should provide lower-level things that Tablicious can build off of. But in practice, Tablicious has specific I/O needs, I generally like to write my own low-level shizz, and depending on the io package introduces the problem of synchronizing versions between the two packages (especially because the stuff Tablicious would need only appears in recent versions of io). So who knows?

I'm open to either approach: build on top of io and take it as a dependency, or just build our own I/O stuff from scratch.

[eebrown]

Thanks for considering. I am not too familiar with the Octave package system (I mostly work in R), but another option would be to borrow the needed (GPL) code within your package to avoid a dependency issue, and then you could also adapt it to your needs. I think ultimately all of this belongs in base Octave, since they are core Matlab features and in high demand (e.g. bug https://savannah.gnu.org/bugs/?44571). In any event thank you for all your work on this package.

[apjanke]

I think ultimately all of this belongs in base Octave, since they are core Matlab features

I agree. My ultimate goal is to contribute the entirety of Tablicious up into the core Octave code base and have it just be part of core Octave. Though I'm going to maintain it, at least in parallel, as a package for a while, because I am stuck on Octave 4.4.x due to the Octave 5.x GUI being broken on macOS, so if it makes it in to an upcoming Octave release I still won't be able to use it there.

and in high demand

Is it? I've been working on this library for like a year and gotten almost no users or bug reports/feature requests. ;)

In any event thank you for all your work on this package.

You're very welcome! Happy to have someone who finds it useful.

[XVilka]

I am stuck on Octave 4.4.x due to the Octave 5.x GUI being broken on macOS

It probably worth to retry Octave 6.2.0 nowadays:

• https://www.gnu.org/software/octave/news/release/2021/02/20/octave-6.2.0-released.html
• https://wiki.octave.org/Octave_for_macOS

[apjanke]

It probably worth to retry Octave 6.2.0 nowadays

You are correct! The Octave 6.x release fixed the GUI problems we were having on macOS, so this is no longer a blocker here, and the importance of having a back-compatible Tablicious package is lower IMHO. (Though still useful, because there are some people out there, especially Linux users running the distribution-supplied Octave, who are still on older Octave versions.)

We're actually having a discussion on the Octave Maintainers mailing list right now about getting all this Tablicious stuff into core Octave eventually.