Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unified API for Micro-manager readers #222

Merged
merged 11 commits into from
Jul 2, 2024
Merged

Unified API for Micro-manager readers #222

merged 11 commits into from
Jul 2, 2024

Conversation

ieivanov
Copy link
Contributor

No description provided.

JoOkuma and others added 9 commits June 21, 2023 16:17
@JoOkuma @ziw-liu
Inheriting BaseFOV on ClearControlFOV (#148)
d40594e 
* inheriting basefov on ccfov

* add deprecation warning to ccfov.scale

* activating github pr workflow

* Update NDTIFF reader (#145)

use new ndtiff and stop sorting axes

* add ccfov scales tests

* Release timing requirement for I/O-heavy test (#147)

* release timing requirement for I/O-heavy test

* suppress data size check for arrays

---------

Co-authored-by: Ziwen Liu <[email protected]>
@ziw-liu
Merge branch 'main' into unified-api
8d6fb57
@ziw-liu
Merge branch 'main' into unified-api
d98bf34
@ziw-liu
Merge branch 'main' into unified-api
e4d418e
@ziw-liu @ieivanov
Universal API implementations for Micro-Manager OME-TIFF and NDTiff (#…
3ab89ba 
…185)

* wip: draft mmstack ome-tiff fov

* MM FOV base class

* tests

* bump tifffile

* comment

* fix indent after rebase

* use get default

* test pixel indexing

* set MM metadata

* style

* update dependencies

* add xarray

* move old readers to the `_deprecated` namespace

* uapi for ndtiff

* refactor test setup to parametrize by dataset
use globals instead of fixtures since parametrization happens before fixture evaluation

* convert mmstack

* fix and test chunking

* fix metadata conversion and test ndtiff

* update cli

* fix scaling

* test 1.4 and incomplete ome-tiffs

* move reader tests

* deprecate reader tests

* update deprecated tests

* update ngff tests

* isort

* update black target to 3.10

* lint

* fix download paths

* update docs references and theme

* untrack autogenerated file

* ignore execution time file

* add github icon

* update docstring

* update docstring

* show channel names and chunk size in info

* print plate chunk size if verbose

* fallback for pixel size

* remove log level setting

* do not filter logs and warnings in reader

* avoid root logger

* isort

* set default logging level to INFO

* format docstring

* improve conversion messages

* black

* fix ome-tiff channel name indexing

* fix ndtiff channel name indexing

* update converter test

* remove use of os.path in `reader`

* expand _check_ndtiff checks

* fix iteration

* fix python 3.10

using `Path.glob(*/)` to get subdirs was added in 3.11

* bump zarr version to include resizing fix
zarr-developers/zarr-python#1540

* fix cli default

* set log level with an environment variable

* fix unset

* catch non-existent page

* implement fallback for incomplete channel names
workaround for the Dragonfly microscope where the multi-camera setup only has one channel name written

* add debug logs

* handle virtual frames

* try reading pages from TiffFile directly

* filter error logs about ImageJ metadata being broken
this is a known MM limitation when writing OME-TIFFs

* fix regex

* remove use of os.path in `convert.py`

* better channel indexing in `_get_summary_metadata`

* style

* safer NoneType check

* private default axis names for NDTiff

* update documentation to reflect new entry point

* add repr to MM FOV and dataset types

* rename mm_meta and expose summary metadata

* add MicroManagerFOVMapping.root

* add MicroManagerFOVMapping.zyx_scale

* add warning log for failed position grid

* fix grid layout

* suppress hypothesis flakiness

* different health check suppression

---------

Co-authored-by: Ivan Ivanov <[email protected]>
@ieivanov
Fix bug determining number of rows and cols (#214)
0c6984e 
* fix bug determining number of rows and cols

* add another XY Stage variation

* add docs and fix style
@ieivanov
Fix bug reading dragonfly acquisitions (#215)
fac2c13 
* fix bug reading dragonfly acquisitions

* fix typo

* style

* bugfix
@ieivanov @ziw-liu
Sort keys in metadata .json files (#211)
420f052 
* wip: draft mmstack ome-tiff fov

* MM FOV base class

* tests

* bump tifffile

* comment

* fix indent after rebase

* use get default

* test pixel indexing

* set MM metadata

* style

* update dependencies

* add xarray

* move old readers to the `_deprecated` namespace

* uapi for ndtiff

* refactor test setup to parametrize by dataset
use globals instead of fixtures since parametrization happens before fixture evaluation

* convert mmstack

* fix and test chunking

* fix metadata conversion and test ndtiff

* update cli

* fix scaling

* test 1.4 and incomplete ome-tiffs

* move reader tests

* deprecate reader tests

* update deprecated tests

* update ngff tests

* isort

* update black target to 3.10

* lint

* fix download paths

* update docs references and theme

* untrack autogenerated file

* ignore execution time file

* add github icon

* update docstring

* update docstring

* show channel names and chunk size in info

* print plate chunk size if verbose

* fallback for pixel size

* remove log level setting

* do not filter logs and warnings in reader

* avoid root logger

* isort

* set default logging level to INFO

* format docstring

* improve conversion messages

* black

* fix ome-tiff channel name indexing

* fix ndtiff channel name indexing

* update converter test

* remove use of os.path in `reader`

* expand _check_ndtiff checks

* fix iteration

* fix python 3.10

using `Path.glob(*/)` to get subdirs was added in 3.11

* bump zarr version to include resizing fix
zarr-developers/zarr-python#1540

* fix cli default

* set log level with an environment variable

* fix unset

* catch non-existent page

* implement fallback for incomplete channel names
workaround for the Dragonfly microscope where the multi-camera setup only has one channel name written

* add debug logs

* handle virtual frames

* try reading pages from TiffFile directly

* filter error logs about ImageJ metadata being broken
this is a known MM limitation when writing OME-TIFFs

* fix regex

* remove use of os.path in `convert.py`

* better channel indexing in `_get_summary_metadata`

* style

* safer NoneType check

* private default axis names for NDTiff

* sort metadata keys

* Squashed commit of the following:

commit fac2c13
Author: Ivan Ivanov <[email protected]>
Date:   Tue Apr 9 11:25:36 2024 -0700

    Fix bug reading dragonfly acquisitions (#215)

    * fix bug reading dragonfly acquisitions

    * fix typo

    * style

    * bugfix

commit 0c6984e
Author: Ivan Ivanov <[email protected]>
Date:   Mon Mar 11 12:35:51 2024 -0700

    Fix bug determining number of rows and cols (#214)

    * fix bug determining number of rows and cols

    * add another XY Stage variation

    * add docs and fix style

commit 3ab89ba
Author: Ziwen Liu <[email protected]>
Date:   Mon Mar 4 11:02:49 2024 -0800

    Universal API implementations for Micro-Manager OME-TIFF and NDTiff (#185)

    * wip: draft mmstack ome-tiff fov

    * MM FOV base class

    * tests

    * bump tifffile

    * comment

    * fix indent after rebase

    * use get default

    * test pixel indexing

    * set MM metadata

    * style

    * update dependencies

    * add xarray

    * move old readers to the `_deprecated` namespace

    * uapi for ndtiff

    * refactor test setup to parametrize by dataset
    use globals instead of fixtures since parametrization happens before fixture evaluation

    * convert mmstack

    * fix and test chunking

    * fix metadata conversion and test ndtiff

    * update cli

    * fix scaling

    * test 1.4 and incomplete ome-tiffs

    * move reader tests

    * deprecate reader tests

    * update deprecated tests

    * update ngff tests

    * isort

    * update black target to 3.10

    * lint

    * fix download paths

    * update docs references and theme

    * untrack autogenerated file

    * ignore execution time file

    * add github icon

    * update docstring

    * update docstring

    * show channel names and chunk size in info

    * print plate chunk size if verbose

    * fallback for pixel size

    * remove log level setting

    * do not filter logs and warnings in reader

    * avoid root logger

    * isort

    * set default logging level to INFO

    * format docstring

    * improve conversion messages

    * black

    * fix ome-tiff channel name indexing

    * fix ndtiff channel name indexing

    * update converter test

    * remove use of os.path in `reader`

    * expand _check_ndtiff checks

    * fix iteration

    * fix python 3.10

    using `Path.glob(*/)` to get subdirs was added in 3.11

    * bump zarr version to include resizing fix
    zarr-developers/zarr-python#1540

    * fix cli default

    * set log level with an environment variable

    * fix unset

    * catch non-existent page

    * implement fallback for incomplete channel names
    workaround for the Dragonfly microscope where the multi-camera setup only has one channel name written

    * add debug logs

    * handle virtual frames

    * try reading pages from TiffFile directly

    * filter error logs about ImageJ metadata being broken
    this is a known MM limitation when writing OME-TIFFs

    * fix regex

    * remove use of os.path in `convert.py`

    * better channel indexing in `_get_summary_metadata`

    * style

    * safer NoneType check

    * private default axis names for NDTiff

    * update documentation to reflect new entry point

    * add repr to MM FOV and dataset types

    * rename mm_meta and expose summary metadata

    * add MicroManagerFOVMapping.root

    * add MicroManagerFOVMapping.zyx_scale

    * add warning log for failed position grid

    * fix grid layout

    * suppress hypothesis flakiness

    * different health check suppression

    ---------

    Co-authored-by: Ivan Ivanov <[email protected]>

* black

* bugfix

---------

Co-authored-by: Ziwen Liu <[email protected]>
@ieivanov @ziw-liu
Modify warning (#220)
a623faa 
* issue one warning per (t_idx, c_idx)

* update messaging and prettify logging

* fix ome-zarr-py breakage in round-trip test

---------

Co-authored-by: Ziwen Liu <[email protected]>
@ieivanov
Copy link
Contributor Author

@ziw-liu @JoOkuma I have been using the new readers and I think they work well. This summer we have plans for acquiring a lot of data and I would like to distribute the work of converting and reconstructing data by training a wider set of users (see draft of https://czbiohub.atlassian.net/wiki/x/GgBKw). It's important that we use this version of the code for data conversion to avoid millions of small files. It'll be helpful if we merge this branch into main now rather than waiting to convert the zarr file type to the universal API format as well. Do you see any setbacks?

@ziw-liu
Copy link
Collaborator

ziw-liu commented Jun 12, 2024

It'll be helpful if we merge this branch into main now rather than waiting to convert the zarr file type to the universal API format as well. Do you see any setbacks?

I don't see major issues in this plan. I will handle conflicts and open separate PRs to fix documentation before merging this.

@ziw-liu
Copy link
Collaborator

ziw-liu commented Jun 12, 2024

For history granularity reasons, I suggest avoid pushing to this branch directly and use a merge commit to main.

@ziw-liu ziw-liu added this to the 0.2.0 milestone Jun 12, 2024
@ziw-liu
Copy link
Collaborator

ziw-liu commented Jun 12, 2024

For history granularity reasons, I suggest avoid pushing to this branch directly and use a merge commit to main.

See #223

@ziw-liu ziw-liu self-requested a review June 12, 2024 22:17
This was linked to issues Jun 12, 2024
Improve TIFF I/O stack #53
Closed
Refactor _set_mm_meta in BaseReader and its children #85
Closed
The MicromanagerOmeTiffReader needs methods to retrieve the image plane metadata #151
Closed
The MicromanagerOmeTiffReader needs better definitions or documentation for position attributes #152
Closed
Single-page TIFF converter regression? #164
Closed
iohub info fails on MM datasets without xy_pixel_size #180
Closed
unpin zarr versions #207
Closed
@ziw-liu
Copy link
Collaborator

ziw-liu commented Jun 12, 2024

Linking issues slated for closing in #185.

@ziw-liu ziw-liu mentioned this pull request Jun 13, 2024
Closed
@ziw-liu
UAPI docs update (#223)
80b6839 
* format

* update README

* update type hint and docstring

* add version switcher item

* format json

* improve text on the documentation badge
@ziw-liu ziw-liu requested a review from JoOkuma July 1, 2024 21:25
@ziw-liu
Copy link
Collaborator

ziw-liu commented Jul 1, 2024 •
edited
Loading

@JoOkuma In my view this PR does not meet the v0.2 milestone. Is there a specific use case for an immediate release after merging?

We'll tag v0.2.0a0 after merging

@ziw-liu ziw-liu merged commit bcd6860 into main Jul 2, 2024
7 checks passed
@ziw-liu ziw-liu deleted the unified-api branch July 2, 2024 00:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ziw-liu ziw-liu ziw-liu approved these changes

@JoOkuma JoOkuma Awaiting requested review from JoOkuma

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.2.0
3 participants
@ieivanov @ziw-liu @JoOkuma