Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated Arrow files unreadable by JS libs #655

Closed
jheer opened this issue Dec 29, 2024 · 2 comments · Fixed by #656
Closed

Updated Arrow files unreadable by JS libs #655

jheer opened this issue Dec 29, 2024 · 2 comments · Fixed by #656
Labels
bug

Comments

@jheer
Copy link
Member

jheer commented Dec 29, 2024

It looks like recent changes to flights-200k.arrow include compression. Unfortunately compressed data is not supported by either the Apache Arrow JS lib or Flechette. I recommed reverting to uncompressed data.

Here are the results of trying to load the file with both libraries within an Observable notebook:

Image

(Originally raised in @uwdata/flechette#26)
@jheer jheer added the bug label Dec 29, 2024
@jheer jheer mentioned this issue Dec 29, 2024
Closed
@dangotbanned
Copy link
Member

Sorry to have broken this for you @jheer!

vega-datasets/scripts/flights.py

Lines 503 to 504 in 369b462

case ".arrow":
df.write_ipc(fp, compression="zstd")

Just to confirm, from the options in polars.dataframe.write_ipc; only "uncompressed" is expected to work for you?

I think I was concerned about the significant size difference in flights-3m when I chose that option.
But any reduction in size is useless if you can't read it anyway 😅

We should definitely fix this before #654

@dangotbanned dangotbanned mentioned this issue Dec 21, 2024
Open
3 tasks
@domoritz
Copy link
Member

The 3m is less of a concern but we should not compress the 200k arrow file.

dangotbanned added a commit that referenced this issue Dec 31, 2024
@dangotbanned
fix: Change default .arrow compression to "uncompressed"
36b7573 
Fixes #655

Adds support for overriding any defaults on a per-`Spec` basis.
This may be helpful if we later run into issues with vega/vega#3961

**The file size of `flights-200k.arrow` has a 7.2x increase.**
However, this is still roughly half the size of `flights-200k.json`.
@dangotbanned dangotbanned mentioned this issue Dec 31, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Change default .arrow compression to "uncompressed" vega/vega-datasets
3 participants
@jheer @domoritz @dangotbanned