Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster compute average intensities #73

Merged
merged 9 commits into from
Mar 26, 2024
Merged

Conversation

mgeplf
Copy link
Collaborator

@mgeplf mgeplf commented Mar 1, 2024

Most of the speed up comes from not having to repeatedly create masks:

old: 1920s
new:
2min 26s
so about 13 times faster
10s, so about 192 times faster

Small change in computed densities:
(Pdb++) np.abs(result.to_numpy() - old.to_numpy()).max()
9.518734625513225e-08
Test code:

from atlas_densities.densities import fitting
from voxcell import VoxelData, RegionMap
from atlas_densities.densities import utils

annotation = VoxelData.load_nrrd('input-data/annotation_ccfv2_l23split_barrelsplit_validated.nrrd')

gene_voxeldata = {
    'gad67': VoxelData.load_nrrd('input-data/gene_gad67_correctednissl.nrrd'),
    'pv': VoxelData.load_nrrd('input-data/gene_pv_correctednissl.nrrd'),
    'sst': VoxelData.load_nrrd('input-data/gene_sst_correctednissl.nrrd'),
    'vip': VoxelData.load_nrrd('input-data/gene_vip_correctednissl.nrrd'),
}
gids = { 'gad67': '479', 'pv': '868', 'sst': '1001', 'vip': '77371835', }
slices = utils.load_json('input-data/realigned_slices.json')
gene_marker_volumes = {
    gene: {
        "intensity": gene_data.raw,
        "slices": slices[gids[gene]],  # list of integer slice indices
    }
    for (gene, gene_data) in gene_voxeldata.items()
}
region_map = RegionMap.load_json('input-data/hierarchy_ccfv2_l23split_barrelsplit.json')
hierarchy_info = utils.get_hierarchy_info(region_map, root='root')

res = fitting.compute_average_intensities(annotation.raw, gene_marker_volumes, hierarchy_info, region_map)

@mgeplf mgeplf force-pushed the faster-compute_average_intensities branch from e0d38b3 to 24f1946 Compare March 21, 2024 14:22
Base automatically changed from cleanup to main March 22, 2024 08:40
@mgeplf mgeplf force-pushed the faster-compute_average_intensities branch 4 times, most recently from 54c348e to 0b9adec Compare March 22, 2024 13:08
@mgeplf mgeplf requested a review from eleftherioszisis March 22, 2024 13:10
@codecov-commenter
Copy link

codecov-commenter commented Mar 22, 2024

Codecov Report

Attention: Patch coverage is 97.01493% with 2 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (main@5b08942). Click here to learn what that means.

Files Patch % Lines
atlas_densities/densities/fitting.py 97.01% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #73   +/-   ##
=======================================
  Coverage        ?   97.67%           
=======================================
  Files           ?       22           
  Lines           ?     1508           
  Branches        ?        0           
=======================================
  Hits            ?     1473           
  Misses          ?       35           
  Partials        ?        0           
Flag Coverage Δ
pytest 97.67% <97.01%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mgeplf mgeplf force-pushed the faster-compute_average_intensities branch 3 times, most recently from d33b2c1 to 4edbe62 Compare March 26, 2024 13:51
mgeplf and others added 8 commits March 26, 2024 14:57
Most of the speed up comes from being able to parallelize across cores:

old: 1920s
new: 2min 26s ± 199 ms

so about 13 times faster

Small change in computed densities:
    (Pdb++) np.abs(result.to_numpy() - old.to_numpy()).max()
        9.518734625513225e-08
Test code:
```
from atlas_densities.densities import fitting
from voxcell import VoxelData, RegionMap
from atlas_densities.densities import utils

annotation = VoxelData.load_nrrd('input-data/annotation_ccfv2_l23split_barrelsplit_validated.nrrd')

gene_voxeldata = {
    'gad67': VoxelData.load_nrrd('input-data/gene_gad67_correctednissl.nrrd'),
    'pv': VoxelData.load_nrrd('input-data/gene_pv_correctednissl.nrrd'),
    'sst': VoxelData.load_nrrd('input-data/gene_sst_correctednissl.nrrd'),
    'vip': VoxelData.load_nrrd('input-data/gene_vip_correctednissl.nrrd'),
}
gids = { 'gad67': '479', 'pv': '868', 'sst': '1001', 'vip': '77371835', }
slices = utils.load_json('input-data/realigned_slices.json')
gene_marker_volumes = {
    gene: {
        "intensity": gene_data.raw,
        "slices": slices[gids[gene]],  # list of integer slice indices
    }
    for (gene, gene_data) in gene_voxeldata.items()
}
region_map = RegionMap.load_json('input-data/hierarchy_ccfv2_l23split_barrelsplit.json')
hierarchy_info = utils.get_hierarchy_info(region_map, root='root')

res = fitting.compute_average_intensities(annotation.raw, gene_marker_volumes, hierarchy_info, region_map)
```
* Use voxel index to speed up the computation of densities
* Revert format changes
* Make lint happy
@mgeplf mgeplf force-pushed the faster-compute_average_intensities branch from 4edbe62 to 0e5ed80 Compare March 26, 2024 14:00
@mgeplf mgeplf force-pushed the faster-compute_average_intensities branch from 0e5ed80 to 651062d Compare March 26, 2024 14:03
@mgeplf mgeplf merged commit fba1e3f into main Mar 26, 2024
5 checks passed
@mgeplf mgeplf deleted the faster-compute_average_intensities branch March 26, 2024 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants