add 'ignore' case to create_layered_volume

[alTeska]

Add an ignore case to create_layered_volume based on an optional key in the metadata dictionary to allow literature-based barrel column names with queries present in the pipeline.

The regex cases ("queries": ["@.*1[ab]?$", "@.*[2-3][ab]?$", "@.*4[ab]?$", "@.*5[ab]?$", "@.*6[ab]?$"] ) struggled with the following pattern:

SSp-bfd:
    SSp-bfd-C1
        SSp-bfd-C1-1
        SSp-bfd-C1-2
        SSp-bfd-C1-3
...

Where the name of the barrel includes one of the layer numbers at the end (1, 2, 3, 4, 5). So for example column name ending with number 1, i.e. SSp-bfd-C1, would be included in the indices found for layer 1.

New suggested metadata has to follow this pattern (to be added into atlas-placement-hints):

{
    "region": {
        "name": "Isocortex",
        "query": "Isocortex",
        "attribute": "acronym",
        "with_descendants": true

    },
    "layers": {
        "names": ["layer_1", "layer_2", "layer_3", "layer_4", "layer_5", "layer_6"],
        "queries": ["@.*1[ab]?$", "@.*2[ab]?$", "@.*[^/]3[ab]?$", "@.*4[ab]?$", "@.*5[ab]?$", "@.*6[ab]?$"],
        "attribute": "acronym",
        "with_descendants": true
    },
    "ignore":{
        "name":"SSp-bfd",
        "query": "SSp-bfd-[^-]*$",
        "attribute": "acronym",
        "with_descendants": true
    }
}

Find a more detailed description of the issue in atlas-placement-hints: https://github.com/BlueBrain/atlas-placement-hints/issues/12 with visualizations.

The only alternative implementation I can think of would mimic the behaviour of assert_leaf_node and require loading a specific hierarchy dictionary to check if a given node has children.

@lecriste and @mgeplf please let me know what do you think, thanks

[codecov-commenter]

Codecov Report

All modified lines are covered by tests ✅

❗ No coverage uploaded for pull request base (main@31f827f). Click here to learn what that means.

Additional details and impacted files

@@           Coverage Diff           @@
##             main       #5   +/-   ##
=======================================
  Coverage        ?   76.57%           
=======================================
  Files           ?        4           
  Lines           ?      175           
  Branches        ?        0           
=======================================
  Hits            ?      134           
  Misses          ?       41           
  Partials        ?        0           

Flag	Coverage Δ
pytest	76.57% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lecriste]

I added a couple of comments here plus this.

[arnaudon]

hold on a little bit, I'm trying to find a regex that works, so it'll be easier that this (also to use same regex in other codes)

[alTeska]

Ok after some further investigation with Alexis, he found a new regex that would exclude the barrels:

As an example for layer one, we switch from "@.*1[ab]?$" to:

"@^(.(?!.*bfd-[A-Z][0-9]$))(.*1[ab]?$)*$"

What do you think?

Here is a little snippet we used for testing this:

import voxcell  
import numpy as np

hierarchy_path = "/gpfs/bbp.cscs.ch/project/proj100/atlas/mouse/atlas-release-mouse-barrels/hierarchy.json"
region_map = voxcell.RegionMap.load_json(hierarchy_path)


layer_ids = region_map.find(
"@^(.(?!.*bfd-[A-Z][0-9]$))(.*1[ab]?$)*$",
    attr="acronym",
    with_descendants=True,
)

region_ids = region_map.find(
    "Isocortex",
    attr="acronym",
    with_descendants=True,
)

results = list(layer_ids & region_ids)
               
C1index = region_map.find("SSp-bfd-C1", attr="acronym").pop()
N2index = region_map.find("SSp-n1", attr="acronym").pop()
C1L1index = region_map.find("SSp-bfd-C1-1", attr="acronym").pop()
sspll_L1index = region_map.find("SSp-ll1", attr="acronym").pop()
V1L1index = region_map.find("VISp1", attr="acronym").pop()

np.isin(C1index, results), np.isin(N2index, results), np.isin(C1L1index, results), np.isin(sspll_L1index, results), np.isin(V1L1index, results)

[lecriste]

Looks good to me. Do the placement hints produced with this new regex match the expected result?

[mgeplf]

Can it be simplified more to:
^(?!SSp-bfd-[A-Z][0-9]$).*1$

IIRC, only layers 2, 3 & 6 have ab suffixes.

Narrowing the A-Z and 0-9 character classes would be good too, imo, to what the maximum values are

[alTeska]

Yes that works too, I have not tested the full placement hints, but the test would just need a new metadata.json, I will create one

[arnaudon]

ah yes, the [ab] is only as in the original regex of course. Yes, narrowing I wasn't sure what were the bounds

[mgeplf]

It also removes the leading . (not sure what that's for), clarifies which region is affected (SSp-bfd-); and drops the capture around the trailing 1[ab] and its extra .*$

[mgeplf]

Ok after some further investigation with Alexis, he found a new regex that would exclude the barrels:

Doing it this way is great, btw, nice work @arnaudon / @alTeska.

[arnaudon]

yes, so I can use the same regex in synthesis later, without having to add this extra ignore thingy ;)

[mgeplf]

I have been away.
@alTeska, @lecriste, @arnaudon is this still needed now that the regex seem to work?