From 2281f3ea5dacd6b46c234bf0f779b5723865dfe7 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 21 May 2024 17:58:09 +1000 Subject: [PATCH 01/33] Add template for RFC-3 --- rfc/3/index.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 rfc/3/index.md diff --git a/rfc/3/index.md b/rfc/3/index.md new file mode 100644 index 00000000..5c2549b8 --- /dev/null +++ b/rfc/3/index.md @@ -0,0 +1,4 @@ +RFC-0 +===== + +Example From 687cd8fe1ef71fc9a690189a3ba061a3c74bcefa Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Wed, 22 May 2024 00:12:02 +1000 Subject: [PATCH 02/33] Complete draft RFC --- rfc/3/index.md | 155 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 152 insertions(+), 3 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 5c2549b8..0d76327d 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -1,4 +1,153 @@ -RFC-0 -===== +# RFC-3: more dimensions for thee -Example +Remove restrictions on the number of dimensions stored in OME-NGFF arrays. + +## Status + +Draft (D3). + +| Name | GitHub Handle | Institution | Date | Status | +|-----------|---------------|-------------|------------|----------------------------------------| +| Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Implementing][implementation] | +| Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | + + + +## Overview + +OME-NGFF version 0.4 places severe restrictions on the number, names, and types +of axes that are allowed in the axes metadata. This has had the effect of +limiting the datasets in proprietary formats that *can* be meaningfully +converted to NGFF. It has also prevented some novel datasets from being written +in NGFF format. This RFC only removes restrictions from the specification. An +important consequence is that all valid NGFF datasets would remain valid after +this change. + +## Background + +OME-NGFF [aims][nat methods paper] to provide a unified open format for +bioimaging data and metadata to make it findable, accessible, interoperable, and +reusable. The [paper describing NGFF][nat methods paper] notes that "the +diversity of [biological imaging's] applications have prevented the +establishment of a community-agreed standardized data format", but, +unfortunately, [version 0.4 of the NGFF specification][ngff 0.4] [imposes][ngff +0.4 multiscales metadata] severe restrictions on the applications: + +> The length of "axes" must be between 2 and 5 and MUST be equal to the +> dimensionality of the zarr arrays storing the image data (see +> "datasets:path"). The "axes" MUST contain 2 or 3 entries of "type:space" and +> MAY contain one additional entry of "type:time" and MAY contain one additional +> entry of "type:channel" or a null / custom type. The order of the entries MUST +> correspond to the order of dimensions of the zarr arrays. In addition, the +> entries MUST be ordered by "type" where the "time" axis must come first (if +> present), followed by the "channel" or custom axis (if present) and the axes +> of type "space". If there are three spatial axes where two correspond to the +> image plane ("yx") and images are stacked along the other (anisotropic) axis +> ("z"), the spatial axes SHOULD be ordered as "zyx". + +And: + +> Each "datasets" dictionary MUST have the same number of dimensions and MUST +> NOT have more than 5 dimensions. + +These restrictions are actively preventing users from converting existing +datasets to NGFF. For example, Zeiss .czi datasets [may contain][czi format +dimensions] dimensions such as H, I, and V to store different phases, +illumination directions, or views respectively. To say nothing of synthetic data +that may contain "artificial" dimensions such as principal components or axes of +other dimensionality reduction-techniques from many images. + +## Proposal + +This document proposes removing any restrictions on the number of dimensions +stored in NGFF arrays. Additionally, it removes restrictions on the names and +types of included dimensions. + +## Prior art and references + +All of the above removals are part of the draft proposed [transformations +specification][trafo spec], with one exception: the draft currently specifies +that a dataset may only have up to three spatial axes. However, this limitation +is [not set in stone][space dims comment] and could be removed, partly to +improve backwards compatibility. + +## Stakeholders + +Who has a stake in whether this RFC is accepted? + +* Facilitator: Josh Moore (OME) +* Reviewers: + - John Bogovic (HHMI Janelia Research Campus): lead author of draft + [transformations specification proposal][trafo spec] + - Will Moore (OME): maintainer of ome-zarr-py library + - Norman Rzepka (Scalable Minds): maintainer of zarrita + - ... Your name here? +* Consulted: + - ... +* Socialization: + - image.sc: https://forum.image.sc/t/ome-ngff-update-postponing-transforms-previously-v0-5/95617/2 + +## Implementation + +A draft implementation is [already in progress][implementation]. + +## Backwards Compatibility + +Since this proposal only removes restrictions, these changes are backwards +compatible at the file level: v0.4 files would transparently be v0.5 files if +this proposal is approved. + +Any readers or writers that proactively checked the dimension restrictions +(number of dimensions, dimension names, dimension types) would need to remove +those checks. However, this should be a small amount of work in most cases. + +## Forward Compatibility + +A draft proposal for [coordinate transformations][trafo spec] already includes +most of the changes proposed here, so we envision that this RFC is compatible +with future plans for the format. The proposal does currently limit the number +of dimensions of type "space" to at most 3, but that limit [could be +removed][space dims comment]. If this RFC is approved, the transformation +specification would need to be updated to reflect this. However, that is an easy +change and there seems to be sufficient support in the community for this idea. + +## Drawbacks, risks, alternatives, and unknowns + +The main reason specifications make restrictions on a file format is to limit +the space of possible implementations. This reduces the overall complexity of +supporting a file format and the burden on implementations. + +One particular concern that has been voiced in the past is that in general, +software dealing with these images knows what to do with axes called x, y, and +z, but might not know what to do with axes called foo, bar, and baz. However, +this concern is fully and properly addressed by the existence of the "type" key +in the "axes" metadata, and the special type called "space". + + + + + +## Performance + +This proposal has no performance implications. + +## Testing + +As part of the [proposed implementation][implementation], Davis Bennett has +created pydantic models that validate the proposed schema. These are actually +new additions to the NGFF specification, surfaced pre-existing errors in the +schema, and should prevent new errors from appearing. + +## License + +This RFC is placed in the public domain. + + +[nat methods paper]: https://www.nature.com/articles/s41592-021-01326-w +[ngff 0.4]: https://ngff.openmicroscopy.org/0.4/index.html +[ngff 0.4 multiscales metadata]: https://ngff.openmicroscopy.org/0.4/index.html#multiscale-md +[ngff 0.4 axes metadata]: https://ngff.openmicroscopy.org/0.4/index.html#axes-md +[czi format dimensions]: https://web.archive.org/web/20240521085825/https://zeiss.github.io/libczi/imagedocumentconcept.html#autotoc_md7 +[implementation]: https://github.com/ome/ngff/pull/235 +[trafo spec]: https://github.com/ome/ngff/pull/138 +[space dims comment]: https://github.com/ome/ngff/pull/138#issuecomment-1852891720 From 96bf663490bcf89d427b26b89b34e0e72a166c56 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Wed, 22 May 2024 00:44:06 +1000 Subject: [PATCH 03/33] Add Talley Lambert, Norman Rzepka, and Davis Bennett as endorsers --- rfc/3/index.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 0d76327d..b638b8b7 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -12,6 +12,9 @@ Draft (D3). | Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | +| Talley Lambert | [tlambert03](https://github.com/tlambert03) | Harvard Medical School | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) | --> +| Norman Rzepka | [normanrz](https://github.com/normanrz) | Scalable Minds | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +| Davis Bennett | [d-v-b](https://github.com/d-v-b) | | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> ## Overview From 35e10f2a8cd11d909059fea01bbe177db012aee5 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Wed, 22 May 2024 11:18:13 +1000 Subject: [PATCH 04/33] Add Doug Shepherd, John Bogovic, Eric Perlman, Lachlan Deakin as endorsers --- rfc/3/index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index b638b8b7..945a1e12 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -10,11 +10,13 @@ Draft (D3). |-----------|---------------|-------------|------------|----------------------------------------| | Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Implementing][implementation] | | Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | - - | Talley Lambert | [tlambert03](https://github.com/tlambert03) | Harvard Medical School | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) | --> | Norman Rzepka | [normanrz](https://github.com/normanrz) | Scalable Minds | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> | Davis Bennett | [d-v-b](https://github.com/d-v-b) | | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +| Doug Shepherd | [dpshepherd](https://github.com/dpshepherd) | Arizona State University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +| John Bogovic | [bogovicj](https://github.com/bogovicj) | HHMI Janelia Research Campus | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +| Eric Perlman | [perlman](https://github.com/perlman) | | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +| Lachlan Deakin | [LDeakin](https://github.com/LDeakin) | Australian National University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> ## Overview From be40c407f6d3331061fe6ed4464646f176106869 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Fri, 5 Jul 2024 16:01:10 +1000 Subject: [PATCH 05/33] Add Sebastian Rhode to endorsers --- rfc/3/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 945a1e12..5c3a8b68 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -17,6 +17,8 @@ Draft (D3). | John Bogovic | [bogovicj](https://github.com/bogovicj) | HHMI Janelia Research Campus | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> | Eric Perlman | [perlman](https://github.com/perlman) | | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> | Lachlan Deakin | [LDeakin](https://github.com/LDeakin) | Australian National University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +| Sebastian Rhode | [sebi06](https://github.com/sebi06) | Carl Zeiss Microscopy +GmbH| 2024-06-05 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> ## Overview From 4cf0457080161f64eafee9b9d35e340e64e6bb23 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:40:08 +0200 Subject: [PATCH 06/33] Clarify in summary that number, name, ordering and type restrictions are removed --- rfc/3/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 5c3a8b68..2f2d4c1f 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -1,6 +1,7 @@ # RFC-3: more dimensions for thee -Remove restrictions on the number of dimensions stored in OME-NGFF arrays. +Remove restrictions on the number, names, ordering, and type of dimensions +stored in OME-NGFF arrays. ## Status From e32423ae64c8319c746c528561b0137c7eac0d28 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:40:54 +0200 Subject: [PATCH 07/33] Clarify motivation and make less heated --- rfc/3/index.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 2f2d4c1f..1197c1a2 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -23,13 +23,15 @@ GmbH| 2024-06-05 | [Endorse](https://github.com/ome/ngff/pull/239#issue-23084364 ## Overview -OME-NGFF version 0.4 places severe restrictions on the number, names, and types -of axes that are allowed in the axes metadata. This has had the effect of -limiting the datasets in proprietary formats that *can* be meaningfully -converted to NGFF. It has also prevented some novel datasets from being written -in NGFF format. This RFC only removes restrictions from the specification. An -important consequence is that all valid NGFF datasets would remain valid after -this change. +OME-NGFF version 0.4 restricts the number, names, ordering, and type of axes +that are allowed in the axes metadata. These restrictions have limited +conversion of proprietary datasets, usage by microscope vendors[^1], and usage +by novel microscopy modalities[^2]. + +This RFC removes these restrictions, opening NGFF to many more users within its +target domain (and beyond). Because it *only* removes restrictions, existing +valid OME-NGFF datasets will remain valid after implementation of this +proposal. ## Background From dd15fd8ae01eefb53490dd069f255f1bc2654c08 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:41:15 +0200 Subject: [PATCH 08/33] Clarify historical context of OME model --- rfc/3/index.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 1197c1a2..0fc1a079 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -36,12 +36,13 @@ proposal. ## Background OME-NGFF [aims][nat methods paper] to provide a unified open format for -bioimaging data and metadata to make it findable, accessible, interoperable, and -reusable. The [paper describing NGFF][nat methods paper] notes that "the +bioimaging data and metadata to make it findable, accessible, interoperable, +and reusable. The [paper describing NGFF][nat methods paper] notes that "the diversity of [biological imaging's] applications have prevented the -establishment of a community-agreed standardized data format", but, -unfortunately, [version 0.4 of the NGFF specification][ngff 0.4] [imposes][ngff -0.4 multiscales metadata] severe restrictions on the applications: +establishment of a community-agreed standardized data format", but, [for +historical reasons][ome-model], [version 0.4 of the NGFF specification][ngff +0.4] [imposes][ngff 0.4 multiscales metadata] strict restrictions on the +applications: > The length of "axes" must be between 2 and 5 and MUST be equal to the > dimensionality of the zarr arrays storing the image data (see From d126b1c32901c877051e3a8db275c6de6284c7d4 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:41:50 +0200 Subject: [PATCH 09/33] Add SHOULD recommendation for 3 space and 1 time dim --- rfc/3/index.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 0fc1a079..4fae2c9e 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -74,6 +74,11 @@ This document proposes removing any restrictions on the number of dimensions stored in NGFF arrays. Additionally, it removes restrictions on the names and types of included dimensions. +To maximise compatibility with existing software, this proposal recommends that +images with 2-3 spatial dimensions SHOULD name them from the subset of "zyx" +and that they SHOULD have type "space". Similarly, if a dataset contains a +single time dimension, it SHOULD have name "t" and type "time". + ## Prior art and references All of the above removals are part of the draft proposed [transformations From f75105db8673483308b86da1966adbf836f7ed16 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:42:06 +0200 Subject: [PATCH 10/33] Add missing references --- rfc/3/index.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 4fae2c9e..bfdc98ae 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -160,6 +160,7 @@ This RFC is placed in the public domain. [nat methods paper]: https://www.nature.com/articles/s41592-021-01326-w +[ome-model]: https://github.com/ome/ngff/pull/239/files#r1609781780 [ngff 0.4]: https://ngff.openmicroscopy.org/0.4/index.html [ngff 0.4 multiscales metadata]: https://ngff.openmicroscopy.org/0.4/index.html#multiscale-md [ngff 0.4 axes metadata]: https://ngff.openmicroscopy.org/0.4/index.html#axes-md @@ -167,3 +168,8 @@ This RFC is placed in the public domain. [implementation]: https://github.com/ome/ngff/pull/235 [trafo spec]: https://github.com/ome/ngff/pull/138 [space dims comment]: https://github.com/ome/ngff/pull/138#issuecomment-1852891720 + + +[^1]: https://github.com/ome/ngff/pull/239#issuecomment-2122809286 +[^2]: https://github.com/ome/ngff/pull/239#issuecomment-2149119404 + From 91d4863e2ac6b5ce6a928455170fd5389c45bc26 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:44:37 +0200 Subject: [PATCH 11/33] are actively preventing -> prevent --- rfc/3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index bfdc98ae..155ec9f1 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -61,7 +61,7 @@ And: > Each "datasets" dictionary MUST have the same number of dimensions and MUST > NOT have more than 5 dimensions. -These restrictions are actively preventing users from converting existing +These restrictions prevent users from converting existing datasets to NGFF. For example, Zeiss .czi datasets [may contain][czi format dimensions] dimensions such as H, I, and V to store different phases, illumination directions, or views respectively. To say nothing of synthetic data From 28859f239a095b4b3a289ff2302e23a5f395c728 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:49:18 +0200 Subject: [PATCH 12/33] Update Davis's role --- rfc/3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 155ec9f1..79baa00a 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -9,7 +9,7 @@ Draft (D3). | Name | GitHub Handle | Institution | Date | Status | |-----------|---------------|-------------|------------|----------------------------------------| -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Implementing][implementation] | +| Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Spec updates][implementation] | | Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | | Talley Lambert | [tlambert03](https://github.com/tlambert03) | Harvard Medical School | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) | --> | Norman Rzepka | [normanrz](https://github.com/normanrz) | Scalable Minds | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> From a53a252deaf7dd73103dc4dd590a441cabf17c5e Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 22:49:30 +0200 Subject: [PATCH 13/33] Reviewers -> proposed reviewers --- rfc/3/index.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 79baa00a..877e468e 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -92,14 +92,13 @@ improve backwards compatibility. Who has a stake in whether this RFC is accepted? * Facilitator: Josh Moore (OME) -* Reviewers: +* Proposed reviewers: - John Bogovic (HHMI Janelia Research Campus): lead author of draft [transformations specification proposal][trafo spec] - Will Moore (OME): maintainer of ome-zarr-py library - Norman Rzepka (Scalable Minds): maintainer of zarrita - - ... Your name here? * Consulted: - - ... + - Every commenter [on this thread](https://github.com/ome/ngff/pull/239). * Socialization: - image.sc: https://forum.image.sc/t/ome-ngff-update-postponing-transforms-previously-v0-5/95617/2 From f0c12e64dc05e7367be94437fe21aa02f9f73ba6 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 23:05:07 +0200 Subject: [PATCH 14/33] Add section on partial implementations --- rfc/3/index.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 877e468e..6d537739 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -79,6 +79,16 @@ images with 2-3 spatial dimensions SHOULD name them from the subset of "zyx" and that they SHOULD have type "space". Similarly, if a dataset contains a single time dimension, it SHOULD have name "t" and type "time". +After this specification change, tools may encounter NGFF files that don't +match the earlier expectations of containing a subset of the TCZYX axes. This +proposal is agnostic as to what to do in those situations, and indeed the +appropriate action depends on the tool, but some suggestions include: +- fail with an informative error message. (i.e. *partial* implementations are + OK, especially if well-documented.) +- prompt the user about which axes to treat as spatial. +- arbitrarily choose which axes to treat as spatial. +- choose how to treat each axis based on heuristics such as size and position. + ## Prior art and references All of the above removals are part of the draft proposed [transformations From f241fd57fa0460b07db9b0989cecfd0e83db38b0 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Sep 2024 23:05:20 +0200 Subject: [PATCH 15/33] Update testing section --- rfc/3/index.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 6d537739..bfb781af 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -158,10 +158,8 @@ This proposal has no performance implications. ## Testing -As part of the [proposed implementation][implementation], Davis Bennett has -created pydantic models that validate the proposed schema. These are actually -new additions to the NGFF specification, surfaced pre-existing errors in the -schema, and should prevent new errors from appearing. +If the RFC is accepted, sample datasets matching the new spec will be +produced for implementations to test against. ## License From 8f28df6ec3e7989a830b379b5d2c3cbbe98a262a Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Wed, 11 Sep 2024 18:11:45 +1000 Subject: [PATCH 16/33] remove reference to implementation --- rfc/3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index bfb781af..6b071f5b 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -114,7 +114,7 @@ Who has a stake in whether this RFC is accepted? ## Implementation -A draft implementation is [already in progress][implementation]. +To Be Determined. ## Backwards Compatibility From 05eea39ee85b0567a57f54f0b5f478abae448b64 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 8 Oct 2024 10:41:18 +0200 Subject: [PATCH 17/33] Add navigation --- rfc/3/comments/index.md | 9 +++++++++ rfc/3/index.md | 17 ++++++++++++++++- rfc/3/responses/index.md | 9 +++++++++ rfc/3/reviews/index.md | 9 +++++++++ rfc/3/versions/index.md | 9 +++++++++ 5 files changed, 52 insertions(+), 1 deletion(-) create mode 100644 rfc/3/comments/index.md create mode 100644 rfc/3/responses/index.md create mode 100644 rfc/3/reviews/index.md create mode 100644 rfc/3/versions/index.md diff --git a/rfc/3/comments/index.md b/rfc/3/comments/index.md new file mode 100644 index 00000000..1ec0d893 --- /dev/null +++ b/rfc/3/comments/index.md @@ -0,0 +1,9 @@ +# Comments + +Additional comments of RFC-3: + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` diff --git a/rfc/3/index.md b/rfc/3/index.md index 945a1e12..a7ad06a9 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -1,10 +1,19 @@ # RFC-3: more dimensions for thee +```{toctree} +:hidden: +:maxdepth: 1 +reviews/index +comments/index +responses/index +versions/index +``` + Remove restrictions on the number of dimensions stored in OME-NGFF arrays. ## Status -Draft (D3). +This RFC is currently in RFC state `R1` (send for review). | Name | GitHub Handle | Institution | Date | Status | |-----------|---------------|-------------|------------|----------------------------------------| @@ -156,3 +165,9 @@ This RFC is placed in the public domain. [implementation]: https://github.com/ome/ngff/pull/235 [trafo spec]: https://github.com/ome/ngff/pull/138 [space dims comment]: https://github.com/ome/ngff/pull/138#issuecomment-1852891720 + +## Changelog + +| Date | Description | Link | +| ---------- | ---------------------------- | ---------------------------------------------------------------------------- | +| 2024-10-08 | RFC assigned and published | [https://github.com/ome/ngff/pull/239](https://github.com/ome/ngff/pull/239) | diff --git a/rfc/3/responses/index.md b/rfc/3/responses/index.md new file mode 100644 index 00000000..7b204b49 --- /dev/null +++ b/rfc/3/responses/index.md @@ -0,0 +1,9 @@ +# Responses + +Responses from the authors of RFC-3: + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` diff --git a/rfc/3/reviews/index.md b/rfc/3/reviews/index.md new file mode 100644 index 00000000..ab13b135 --- /dev/null +++ b/rfc/3/reviews/index.md @@ -0,0 +1,9 @@ +# Reviews + +Reviews of RFC-3: + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` diff --git a/rfc/3/versions/index.md b/rfc/3/versions/index.md new file mode 100644 index 00000000..6ebd2157 --- /dev/null +++ b/rfc/3/versions/index.md @@ -0,0 +1,9 @@ +# Versions + +Key versions of RFC-3 which have been sent for review, etc. + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` From d2437b9cd6cc211d52f8323a9424290ee221f13a Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 8 Oct 2024 15:18:40 +0200 Subject: [PATCH 18/33] Update status table --- rfc/3/index.md | 77 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 66 insertions(+), 11 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index a7ad06a9..f050c5bb 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -15,17 +15,72 @@ Remove restrictions on the number of dimensions stored in OME-NGFF arrays. This RFC is currently in RFC state `R1` (send for review). -| Name | GitHub Handle | Institution | Date | Status | -|-----------|---------------|-------------|------------|----------------------------------------| -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Implementing][implementation] | -| Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | -| Talley Lambert | [tlambert03](https://github.com/tlambert03) | Harvard Medical School | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) | --> -| Norman Rzepka | [normanrz](https://github.com/normanrz) | Scalable Minds | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Doug Shepherd | [dpshepherd](https://github.com/dpshepherd) | Arizona State University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| John Bogovic | [bogovicj](https://github.com/bogovicj) | HHMI Janelia Research Campus | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Eric Perlman | [perlman](https://github.com/perlman) | | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Lachlan Deakin | [LDeakin](https://github.com/LDeakin) | Australian National University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> +```{list-table} Record +:widths: 8, 20, 20, 20, 15, 10 +:header-rows: 1 +:stub-columns: 1 + +* - Role + - Name + - GitHub Handle + - Institution + - Date + - Status +* - Author + - Juan Nunez-Iglesias + - [jni](https://github.com/jni) + - Monash University + - 2024-05-21 + - +* - Implementor + - Davis Bennett + - [d-v-b](https://github.com/d-v-b) + - N/A + - 2024-05-01 + - [Implementing][implementation] +* - Endorser + - Talley Lambert + - [tlambert03](https://github.com/tlambert03) + - Harvard Medical School + - 2024-05-21 + - [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) +* - Endorser + - Norman Rzepka + - [normanrz](https://github.com/normanrz) + - Scalable Minds + - 2024-05-21 + - [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) +* - Endorser + - Davis Bennett + - [d-v-b](https://github.com/d-v-b) + - + - 2024-05-21 + - [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) +* - Endorser + - Doug Shepherd + - [dpshepherd](https://github.com/dpshepherd) + - Arizona State University + - 2024-05-22 + - [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) +* - Endorser + - John Bogovic + - [bogovicj](https://github.com/bogovicj) + - HHMI Janelia Research Campus + - 2024-05-22 + - [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) +* - Endorser + - Eric Perlman + - [perlman](https://github.com/perlman) + - + - 2024-05-22 + - [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) +* - Endorser + - Lachlan Deakin + - [LDeakin](https://github.com/LDeakin) + - Australian National University + - 2024-05-22 + - [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) +``` ## Overview From e152fb45582d5bbdef5d94257b9a06e9389559ec Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 8 Oct 2024 16:15:04 +0200 Subject: [PATCH 19/33] Fix codespell issue --- rfc/3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index f050c5bb..8368388e 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -32,7 +32,7 @@ This RFC is currently in RFC state `R1` (send for review). - Monash University - 2024-05-21 - -* - Implementor +* - Implementer - Davis Bennett - [d-v-b](https://github.com/d-v-b) - N/A From 33e5362ac806c5381dd85e2aa17e7e7bd9d4d370 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Thu, 31 Oct 2024 10:17:28 +0900 Subject: [PATCH 20/33] Remove git merge conflict markers :sweat-smile: --- rfc/3/index.md | 25 ------------------------- 1 file changed, 25 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 8e61e7dd..c6b9aa61 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -16,31 +16,6 @@ stored in OME-NGFF arrays. This RFC is currently in RFC state `R1` (send for review). -<<<<<<< HEAD -| Name | GitHub Handle | Institution | Date | Status | -|-----------|---------------|-------------|------------|----------------------------------------| -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Spec updates][implementation] | -| Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | -| Talley Lambert | [tlambert03](https://github.com/tlambert03) | Harvard Medical School | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) | --> -| Norman Rzepka | [normanrz](https://github.com/normanrz) | Scalable Minds | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Doug Shepherd | [dpshepherd](https://github.com/dpshepherd) | Arizona State University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| John Bogovic | [bogovicj](https://github.com/bogovicj) | HHMI Janelia Research Campus | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Eric Perlman | [perlman](https://github.com/perlman) | | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Lachlan Deakin | [LDeakin](https://github.com/LDeakin) | Australian National University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -||||||| 35e10f2 -| Name | GitHub Handle | Institution | Date | Status | -|-----------|---------------|-------------|------------|----------------------------------------| -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | N/A | 2024-05-01 | [Implementing][implementation] | -| Juan Nunez-Iglesias | [jni](https://github.com/jni) | Monash University | 2024-05-21 | Author | -| Talley Lambert | [tlambert03](https://github.com/tlambert03) | Harvard Medical School | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issuecomment-2122795327) | --> -| Norman Rzepka | [normanrz](https://github.com/normanrz) | Scalable Minds | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Davis Bennett | [d-v-b](https://github.com/d-v-b) | | 2024-05-21 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Doug Shepherd | [dpshepherd](https://github.com/dpshepherd) | Arizona State University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| John Bogovic | [bogovicj](https://github.com/bogovicj) | HHMI Janelia Research Campus | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Eric Perlman | [perlman](https://github.com/perlman) | | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -| Lachlan Deakin | [LDeakin](https://github.com/LDeakin) | Australian National University | 2024-05-22 | [Endorse](https://github.com/ome/ngff/pull/239#issue-2308436425) | --> -======= ```{list-table} Record :widths: 8, 20, 20, 20, 15, 10 :header-rows: 1 From 984a9f7f745e19b495a74af3a0630becea671fe8 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Mon, 2 Dec 2024 23:05:27 +1100 Subject: [PATCH 21/33] Remove d-v-b as implementer --- rfc/3/index.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index c6b9aa61..3a42e0b5 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -33,12 +33,6 @@ This RFC is currently in RFC state `R1` (send for review). - Monash University - 2024-05-21 - -* - Implementer - - Davis Bennett - - [d-v-b](https://github.com/d-v-b) - - N/A - - 2024-05-01 - - [Implementing][implementation] * - Endorser - Talley Lambert - [tlambert03](https://github.com/tlambert03) From 785f705c2fb46cd0a383124b55646bb9adcdc469 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Mon, 2 Dec 2024 23:26:06 +1100 Subject: [PATCH 22/33] Add two paragraphs about performance implications --- rfc/3/index.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 3a42e0b5..76c67968 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -216,7 +216,20 @@ in the "axes" metadata, and the special type called "space". ## Performance -This proposal has no performance implications. +The current OME-Zarr specification ensures arrays are stored in order TCZYX. +With C-order array data, this ensures efficient access for *some* but not *all* +access patterns. By removing restrictions on axis orderings, a new class of +"mistake" is possible, as someone could save an array in order XYTCZ, which +would combine poorly with C-order arrays to view XY planes. However, it is +arguable that Zarr chunking is in fact more important here — XYTCZ *could* be +a perfectly cromulent axis ordering for XY planes if the Zarr chunk size was +(1024, 1024, 1, 1, 1). + +Therefore, this proposal argues that any performance implications are better +addressed through good documentation and good defaults. Indeed, more flexible +dimension ordering could *improve* performance in some scenarios, such as +"pixel drilling", that is, extracting the value of a single x/y position over +time. ## Testing From 6f7408ad019ef29568c81d67c17334f135cce7ea Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Mon, 2 Dec 2024 23:26:22 +1100 Subject: [PATCH 23/33] Remove empty html comments --- rfc/3/index.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 76c67968..b4e8ee55 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -210,10 +210,6 @@ z, but might not know what to do with axes called foo, bar, and baz. However, this concern is fully and properly addressed by the existence of the "type" key in the "axes" metadata, and the special type called "space". - - - - ## Performance The current OME-Zarr specification ensures arrays are stored in order TCZYX. From b19bbbcd1929dc34bdb0cd686d37edd0109876d5 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Dec 2024 00:11:38 +1100 Subject: [PATCH 24/33] Summarise discussion from the PR --- rfc/3/index.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index b4e8ee55..09d8467e 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -204,12 +204,28 @@ The main reason specifications make restrictions on a file format is to limit the space of possible implementations. This reduces the overall complexity of supporting a file format and the burden on implementations. -One particular concern that has been voiced in the past is that in general, +Comments on the [pull request adding this proposal][this pr] and on the related +pull request [updating the specification text][implementation] have indeed +centered on this complexity. + +One particular concern that has been voiced is that in general, software dealing with these images knows what to do with axes called x, y, and z, but might not know what to do with axes called foo, bar, and baz. However, -this concern is fully and properly addressed by the existence of the "type" key +this concern is properly addressed by the existence of the "type" key in the "axes" metadata, and the special type called "space". +Further, this proposal recommends that, in the absence of other considerations, +spatial axes SHOULD be a subset of x, y, and z, to simplify implementations. It +also takes the position that partial implementations are OK: a software package +designed to view xyz volumetric, light microscopy data should feel free to +error when presented with axes foo, bar, and baz with type "arbitrary". This +mechanism allows maximum flexibility for the format while ensuring +domain-specific implementations do not need to grapple with its full +complexity. + +The addition of "SHOULD" recommendations for common microscopy data [seems to +have assuaged most implementation concerns][recap comment]. + ## Performance The current OME-Zarr specification ensures arrays are stored in order TCZYX. @@ -244,6 +260,8 @@ This RFC is placed in the public domain. [ngff 0.4 axes metadata]: https://ngff.openmicroscopy.org/0.4/index.html#axes-md [czi format dimensions]: https://web.archive.org/web/20240521085825/https://zeiss.github.io/libczi/imagedocumentconcept.html#autotoc_md7 [implementation]: https://github.com/ome/ngff/pull/235 +[this pr]: https://github.com/ome/ngff/pull/239 +[recap comment]: https://github.com/ome/ngff/pull/239#issuecomment-2327451719 [trafo spec]: https://github.com/ome/ngff/pull/138 [space dims comment]: https://github.com/ome/ngff/pull/138#issuecomment-1852891720 From 7b7dbf5a83dede0fd7811f4b9e132d9c0065be8e Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Dec 2024 00:43:42 +1100 Subject: [PATCH 25/33] Add dataset types as examples --- rfc/3/index.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 09d8467e..cdd5c3c1 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -130,6 +130,37 @@ illumination directions, or views respectively. To say nothing of synthetic data that may contain "artificial" dimensions such as principal components or axes of other dimensionality reduction-techniques from many images. +## Motivation + +In addition to the .czi datasets mentioned in the preceding paragraph, this +section describes six dataset types that are currently impossible to represent +in OME-Zarr: + +- in [electron backscatter diffraction (EBSD)][ebsd], a microscopy technique + common in materials science, a beam of electrons is scanned over a surface, + and for each (2D) position in the scan, a full 2D diffraction pattern is + recorded, resulting in a 4-dimensional data array. +- from the diffraction patterns, it is possible to obtain an *orientation map*, + containing a 3D angle at each 2D position of the material. +- the same principles apply to [diffusion tensor imaging][dti], where a + three-dimensional diffusion tensor is measured at each voxel. +- it is common to compute Fourier transforms of 3D images. The datasets have + three dimensions but they are measured in *frequency*, not space. +- when computing segmentations, one may use finer or coarser priors, resulting + in overlapping, equally valid segmentations, for example, of organelles at + one level, cells at another, and tissues at yet another. One common way to + store such a segmentation is to add a dimension for "coarseness". +- computed spaces may have arbitrary dimensions related to the computation. For + exmaple, in subtomogram averaging of [cryo electron tomography][CryoET], + single particles from a tomogram are picked and aligned, producing many + instances of the same 3-dimensional particle. One may wish to store all the + instances in a single 4-dimensional array (one dimension being the *instance + number*). Or, one may use dimension-reduction techniques such as PCA, then + browse average particles along each PCA axis. This creates a virtual 5D space + containing the three spatial dimensions, then a "component number" axis for + the PCA components and a "position" axis for the position along that + component. + ## Proposal This document proposes removing any restrictions on the number of dimensions @@ -264,6 +295,8 @@ This RFC is placed in the public domain. [recap comment]: https://github.com/ome/ngff/pull/239#issuecomment-2327451719 [trafo spec]: https://github.com/ome/ngff/pull/138 [space dims comment]: https://github.com/ome/ngff/pull/138#issuecomment-1852891720 +[ebsd]: https://en.wikipedia.org/wiki/Electron_backscatter_diffraction +[dti]: https://en.wikipedia.org/wiki/Diffusion-weighted_magnetic_resonance_imaging [^1]: https://github.com/ome/ngff/pull/239#issuecomment-2122809286 [^2]: https://github.com/ome/ngff/pull/239#issuecomment-2149119404 From 7e0e205e5538b1c5c30d3953ea15c07d6b8d5ba8 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Dec 2024 00:47:15 +1100 Subject: [PATCH 26/33] NGFF->OME-Zarr where appropriate (I think) --- rfc/3/index.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index cdd5c3c1..eabe51bf 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -10,7 +10,7 @@ versions/index ``` Remove restrictions on the number, names, ordering, and type of dimensions -stored in OME-NGFF arrays. +stored in OME-Zarr arrays. ## Status @@ -85,25 +85,26 @@ This RFC is currently in RFC state `R1` (send for review). ## Overview -OME-NGFF version 0.4 restricts the number, names, ordering, and type of axes +OME-Zarr version 0.4 restricts the number, names, ordering, and type of axes that are allowed in the axes metadata. These restrictions have limited conversion of proprietary datasets, usage by microscope vendors[^1], and usage by novel microscopy modalities[^2]. This RFC removes these restrictions, opening NGFF to many more users within its target domain (and beyond). Because it *only* removes restrictions, existing -valid OME-NGFF datasets will remain valid after implementation of this +valid OME-Zarr datasets will remain valid after implementation of this proposal. ## Background -OME-NGFF [aims][nat methods paper] to provide a unified open format for +OME-Zarr [aims][nat methods paper] to provide a unified open format for bioimaging data and metadata to make it findable, accessible, interoperable, -and reusable. The [paper describing NGFF][nat methods paper] notes that "the -diversity of [biological imaging's] applications have prevented the +and reusable. The [paper describing NGFF and OME-Zarr][nat methods paper] notes +that "the diversity of [biological imaging's] applications have prevented the establishment of a community-agreed standardized data format", but, [for -historical reasons][ome-model], [version 0.4 of the NGFF specification][ngff -0.4] [imposes][ngff 0.4 multiscales metadata] strict restrictions on the +historical reasons][ome-model], [version 0.4 of the OME-Zarr +specification][ngff 0.4] [imposes][ngff 0.4 multiscales metadata] strict +restrictions on the applications: > The length of "axes" must be between 2 and 5 and MUST be equal to the @@ -124,7 +125,7 @@ And: > NOT have more than 5 dimensions. These restrictions prevent users from converting existing -datasets to NGFF. For example, Zeiss .czi datasets [may contain][czi format +datasets to OME-Zarr. For example, Zeiss .czi datasets [may contain][czi format dimensions] dimensions such as H, I, and V to store different phases, illumination directions, or views respectively. To say nothing of synthetic data that may contain "artificial" dimensions such as principal components or axes of @@ -164,15 +165,15 @@ in OME-Zarr: ## Proposal This document proposes removing any restrictions on the number of dimensions -stored in NGFF arrays. Additionally, it removes restrictions on the names and -types of included dimensions. +stored in OME-Zarr arrays. Additionally, it removes restrictions on the names +and types of included dimensions. To maximise compatibility with existing software, this proposal recommends that images with 2-3 spatial dimensions SHOULD name them from the subset of "zyx" and that they SHOULD have type "space". Similarly, if a dataset contains a single time dimension, it SHOULD have name "t" and type "time". -After this specification change, tools may encounter NGFF files that don't +After this specification change, tools may encounter OME-Zarr files that don't match the earlier expectations of containing a subset of the TCZYX axes. This proposal is agnostic as to what to do in those situations, and indeed the appropriate action depends on the tool, but some suggestions include: From 21c8acf30019612219f9627cd36e8c6ca521b8c3 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Tue, 3 Dec 2024 00:50:04 +1100 Subject: [PATCH 27/33] Update spec update link alias --- rfc/3/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index eabe51bf..395e8a2f 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -237,7 +237,7 @@ the space of possible implementations. This reduces the overall complexity of supporting a file format and the burden on implementations. Comments on the [pull request adding this proposal][this pr] and on the related -pull request [updating the specification text][implementation] have indeed +pull request [updating the specification text][spec update] have indeed centered on this complexity. One particular concern that has been voiced is that in general, @@ -291,7 +291,7 @@ This RFC is placed in the public domain. [ngff 0.4 multiscales metadata]: https://ngff.openmicroscopy.org/0.4/index.html#multiscale-md [ngff 0.4 axes metadata]: https://ngff.openmicroscopy.org/0.4/index.html#axes-md [czi format dimensions]: https://web.archive.org/web/20240521085825/https://zeiss.github.io/libczi/imagedocumentconcept.html#autotoc_md7 -[implementation]: https://github.com/ome/ngff/pull/235 +[spec update]: https://github.com/ome/ngff/pull/235 [this pr]: https://github.com/ome/ngff/pull/239 [recap comment]: https://github.com/ome/ngff/pull/239#issuecomment-2327451719 [trafo spec]: https://github.com/ome/ngff/pull/138 From 11aa79cc72c1a39505de7208c17da0e7a8feee69 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 3 Dec 2024 16:07:45 +0100 Subject: [PATCH 28/33] Typo --- rfc/3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 395e8a2f..2a56ad21 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -152,7 +152,7 @@ in OME-Zarr: one level, cells at another, and tissues at yet another. One common way to store such a segmentation is to add a dimension for "coarseness". - computed spaces may have arbitrary dimensions related to the computation. For - exmaple, in subtomogram averaging of [cryo electron tomography][CryoET], + example, in subtomogram averaging of [cryo electron tomography][CryoET], single particles from a tomogram are picked and aligned, producing many instances of the same 3-dimensional particle. One may wish to store all the instances in a single 4-dimensional array (one dimension being the *instance From f7d06298a91bb9df3c5d1cc89cef68ba304a3994 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 3 Dec 2024 18:09:30 +0100 Subject: [PATCH 29/33] RFC-3: update listing --- rfc/listing.csv | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/listing.csv b/rfc/listing.csv index 24166b5d..1e2dc6a2 100644 --- a/rfc/listing.csv +++ b/rfc/listing.csv @@ -2,6 +2,6 @@ RFC,Description,Date,Status [0](0/index.md),Original consensus model for decision making,2021,N/A [1](1/index.md),RFC Process,2024,Adopted [2](2/index.md),Zarr V3 Support,2024,Accepted -3,Remove axis restrictions,2024, Clarifications +[3](3/index.md),Remove axis restrictions,2024,Under review [4](4/index.md),Axis Anatomical Orientation,2024,Under review [5](5/index.md),Coordinate systems and transformations,2024,Under review From 315c479c75f670578717be75318bc7c42019c338 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Tue, 3 Dec 2024 18:19:48 +0100 Subject: [PATCH 30/33] Make image.sc thread a link --- rfc/3/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 2a56ad21..096d0971 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -204,7 +204,7 @@ Who has a stake in whether this RFC is accepted? * Consulted: - Every commenter [on this thread](https://github.com/ome/ngff/pull/239). * Socialization: - - image.sc: https://forum.image.sc/t/ome-ngff-update-postponing-transforms-previously-v0-5/95617/2 + - image.sc: ## Implementation From c6e60094748ee3e00a43b0ae9816007abdc8b305 Mon Sep 17 00:00:00 2001 From: Juan Nunez-Iglesias Date: Wed, 4 Dec 2024 18:39:27 +1100 Subject: [PATCH 31/33] Add cryoet link --- rfc/3/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfc/3/index.md b/rfc/3/index.md index 096d0971..32d5360f 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -298,6 +298,7 @@ This RFC is placed in the public domain. [space dims comment]: https://github.com/ome/ngff/pull/138#issuecomment-1852891720 [ebsd]: https://en.wikipedia.org/wiki/Electron_backscatter_diffraction [dti]: https://en.wikipedia.org/wiki/Diffusion-weighted_magnetic_resonance_imaging +[CryoET]: https://en.wikipedia.org/wiki/Cryogenic_electron_tomography [^1]: https://github.com/ome/ngff/pull/239#issuecomment-2122809286 [^2]: https://github.com/ome/ngff/pull/239#issuecomment-2149119404 From 13f17989e5d7d5d9fa3c2c12a334fcddfb833841 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 4 Dec 2024 20:23:38 +0100 Subject: [PATCH 32/33] Drop 'impl: tbd' --- rfc/3/index.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 32d5360f..45369b64 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -206,10 +206,6 @@ Who has a stake in whether this RFC is accepted? * Socialization: - image.sc: -## Implementation - -To Be Determined. - ## Backwards Compatibility Since this proposal only removes restrictions, these changes are backwards From 6e558fa2698042f3e72c3d47103af5b3e8060a3a Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 4 Dec 2024 20:23:45 +0100 Subject: [PATCH 33/33] Drop use of version numbers --- rfc/3/index.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/rfc/3/index.md b/rfc/3/index.md index 45369b64..ca1fb781 100644 --- a/rfc/3/index.md +++ b/rfc/3/index.md @@ -209,12 +209,14 @@ Who has a stake in whether this RFC is accepted? ## Backwards Compatibility Since this proposal only removes restrictions, these changes are backwards -compatible at the file level: v0.4 files would transparently be v0.5 files if -this proposal is approved. +compatible at the file level and appropriate for a non-breaking release. Any readers or writers that proactively checked the dimension restrictions -(number of dimensions, dimension names, dimension types) would need to remove -those checks. However, this should be a small amount of work in most cases. +(number of dimensions, dimension names, dimension types) MAY remove +those checks or update the exception raised to specify that the dimensions +are not supported. + +This should be a small amount of work in most cases. ## Forward Compatibility