Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete data references and "hollow out" DataModel on close. #311

Open
braingram opened this issue Jul 24, 2024 · 0 comments
Open

Delete data references and "hollow out" DataModel on close. #311

braingram opened this issue Jul 24, 2024 · 0 comments

Comments

@braingram
Copy link
Collaborator

braingram commented Jul 24, 2024

There is nothing that prevents accessing DataModel attributes after closing when read from a fits file.

>>> from stdatamodels.jwst import datamodels
>>> model = datamodels.open("foo.fits")
>>> model.close()
>>> model.data
array([[0, 0,...]], dtype='>f4')

This is a side-effect of reading the entire contents into memory and has the negative consequence that the data (and other arrays and metadata) are not freed until the model is garbage collected even if the calling code explicitly closes the file.

Investigate disabling data access (and removing data references) on close to allow memory to be freed and test this new mode with jwst to make sure the pipeline does not rely on the above pattern.

Note that the same is not true for datamodels read from asdf files (which are used for some reference data but generally not used for pipeline data).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant