Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default behaviour for write_map is to use lossy compression with floating-point numbers #3

Open
ziotom78 opened this issue Jan 14, 2016 · 1 comment

Comments

@ziotom78
Copy link
Contributor

The default value for the compress parameter to write_map is True, meaning that the code will use one of the compression algorithms specified by the FITS standard. Since these algorithms are lossy when applied to maps containing floating-point values, the code will silently introduce an error in the data written to disk, as the following example shows:

import numpy
import qubic.io

FILE_NAME = '/tmp/compressed_map.fits'

INPUT_PIXELS = numpy.random.normal(size=12 * 64 * 64)
qubic.io.write_map(FILE_NAME, INPUT_PIXELS, dtype=numpy.float64)
OUTPUT_PIXELS = qubic.io.read_map(FILE_NAME)

DIFF = OUTPUT_PIXELS - INPUT_PIXELS
print('Stddev of the difference: {0:.3f}'
      .format(numpy.std(DIFF)))

# Result:
# Stddev of the difference: 0.018

(Note that the code uses float64 instead of the default float32 type when calling write_map.) Setting compress=False in the above call to qubic.io.write_map makes the difference between INPUT_PIXELS and OUTPUT_PIXELS to vanish.

Since this behaviour might be surprising for people not versed with the FITS standard, I am suggesting a couple of workarounds:

  1. Make compress default to False, and specify in the docstring for write_map that using True with floating-point maps will introduce an error in the pixel values (currently, the docstring does not mention the compress parameter anywhere but in the declaration);
  2. Make the code emit a warning when compress is True and the datatype of the parameter map is a floating-point value.
@ziotom78
Copy link
Contributor Author

Just for the record, using healpy.write_map and healpy.read_map with .fits.gz works seamlessy and produces relevant compression ratios for partial-coverage maps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant