Skip to content

Commit

Permalink
Change where nd2 data gets copied.
Browse files Browse the repository at this point in the history
We are using the dask interface with nd2 to avoid loading all of a file
into memory.  When we read anything other then an entire frame, the
default is to copy the entire frame in memory and THEN apply whatever
index and step is being used.  The nd2 library docs caution that if copy
isn't done, there can be a segfault.  Instead, use our own lock, get the
frame data, subset it with our index and step and THEN copy it; since
this is in a singular lock, it won't have any greater risk that the nd2
library.  Since the copy only applies to the actual data we are using,
much less memory is copied.

In a simple example with a file with large tiles, this saves close to 2
seconds per tile.
  • Loading branch information
manthey committed May 4, 2022
1 parent 32dbb06 commit 05f0c9a
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions sources/nd2/large_image_source_nd2/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import math
import os
import threading

import numpy

Expand Down Expand Up @@ -139,7 +140,7 @@ def __init__(self, path, **kwargs):
if not os.path.isfile(self._largeImagePath):
raise TileSourceFileNotFoundError(self._largeImagePath) from None
raise TileSourceError('File cannot be opened via nd2reader.')
self._nd2array = self._nd2.to_dask()
self._nd2array = self._nd2.to_dask(copy=False)
arrayOrder = list(self._nd2.sizes)
# Reorder this so that it is XY (P), T, Z, C, Y, X, S (or at least end
# in Y, X[, S]).
Expand Down Expand Up @@ -171,6 +172,7 @@ def __init__(self, path, **kwargs):
self._bandnames = {
chan.channel.name.lower(): idx for idx, chan in enumerate(self._nd2.metadata.channels)}
self._channels = [chan.channel.name for chan in self._nd2.metadata.channels]
self._tileLock = threading.RLock()

def getNativeMagnification(self):
"""
Expand Down Expand Up @@ -262,8 +264,8 @@ def getTile(self, x, y, z, pilImageAllowed=False, numpyAllowed=False, **kwargs):
fc //= self._nd2.sizes[axis]
tileframe = tileframe[fp // fc]
fp = fp % fc

tile = numpy.array(tileframe[y0:y1:step, x0:x1:step])
with self._tileLock:
tile = tileframe[y0:y1:step, x0:x1:step].compute().copy()
return self._outputTile(tile, TILE_FORMAT_NUMPY, x, y, z,
pilImageAllowed, numpyAllowed, **kwargs)

Expand Down

0 comments on commit 05f0c9a

Please sign in to comment.