Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better save_to_zarr #266

Merged
merged 6 commits into from
Oct 16, 2023
Merged

Better save_to_zarr #266

merged 6 commits into from
Oct 16, 2023

Conversation

aulemahal
Copy link
Collaborator

@aulemahal aulemahal commented Oct 2, 2023

Pull Request Checklist:

  • This PR addresses an already opened issue (for bug fixes / features)
    • This PR fixes #xyz
  • (If applicable) Documentation has been added / updated (for bug fixes / features).
  • (If applicable) Tests have been added.
  • This PR does not seem to break the templates.
  • HISTORY.rst has been updated (with summary of main changes).
    • Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

  • New func xs.io.round_bits to round a floating point variable up to a number of bits, allowing for a much better compression afterwards.
  • New "bitround" arg to save_to_zarr and save_to_netcdf, allowing to combine the bit rounding and the saving. The argument can take many forms for more flexibility.

Does this PR introduce a breaking change?

No.

Other information:

Default keepbits value is guessed from the variable name, with a default of 12 (0.0122% of rel error). Better guesses should be given in xs.io.KEEPBITS, but I have none for now.

Copy link
Collaborator

@RondeauG RondeauG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I'll let @coxipi approve this, since this is his work.

xscen/io.py Outdated Show resolved Hide resolved
xscen/io.py Outdated Show resolved Hide resolved
xscen/io.py Outdated Show resolved Hide resolved
tests/test_io.py Outdated Show resolved Hide resolved
Copy link
Contributor

@coxipi coxipi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, looks good to me!

@aulemahal aulemahal merged commit fd867f8 into main Oct 16, 2023
6 checks passed
@aulemahal aulemahal deleted the compress branch October 16, 2023 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants