Convenience method for pyramidal TIFF segmentation to JSON conversion? #977
Comments
|
We don't actually support pyramidal annotations as such -- we expect the shaped annotations all to be in the base image coordinates (image annotations can have scaling applied). When an annotation has a lot of polygons, we load their centroids to give a quick idea of where they are and then load a limited number of the largest polygons that are in view, updating on pan / zoom. This really could use better optimization. We do have plans to load complex annotations via vector tiles at some point. This would, ideally, load reduced resolution versions at reduced image resolutions. Since we are using mongo as the back-end database, this will require more internal bookkeeping than if we were using something like PostGIS. When we get to it, some of the under-the-hood management of annotations might change, but the exposed json schema will remain the same. Explicitly having predefined reduced resolution annotations isn't in our current plan, since that requires more work from the creator of the annotations, though given good reasons or PRs we would adapt. |
|
Basically, we don't really care much that the annotation image is pyramidal. It is convenient for rendering purposes. But for JSON conversion I just use a single plane. I came quite far today, but seems like only a single nuclei in a full tile is drawn and rendered. Probably need to check that Im providing the correct input to the Its quite silly, and my solution is not really that optimized, but I believe it should work. Will make a thorough attempt tomorrow, but now there is beer time. Will keep you updated on the manner and post the solution for others when ready, if of interest :] |
|
It works! Or at least kind of. I got an issue when attempting to save >140k cell nuclei from a gigapixel WSI. I have a pretrained deep learning model that performs segmentation of nuclei and I want to store and render the boundaries, similar to what is done in the NucleiDetection example in HistomicsTK. In the NucleiDetection example you store all predictions in a huge list and then it seems like you write them to disk all at once. Perhaps I'm reaching a limit of how many objects I can write? I guess that is what the error message is hinting at. What happens is that the program succeeds after this, but the annotation file is not available after. Nanny closes right after these "warnings". Might be due to it being unresponsive for too long - or maybe this is just done after each process, not sure. Is there a better way in HistomicsTK to write thousands/millions of objects as annotations on disk? It would be ideal if all of these were stored in the same JSON annotation file. |
|
So that may be a mongo issue.. with that.many nuclei you may want to create
several documents..
…On Sat, Feb 25, 2023, 4:39 PM André Pedersen ***@***.***> wrote:
It works! Or at least kind of. I got an issue when attempting to save
>140k cell nuclei from a gigapixel WSI. I have a pretrained deep learning
model that performs segmentation of nuclei and I want to store and render
the boundaries, similar to what is done in the NucleiDetection example
<https://github.com/DigitalSlideArchive/HistomicsTK/blob/master/histomicstk/cli/NucleiDetection/NucleiDetection.py#L276>
in HistomicsTK.
>> Done iterating tiles. Total number of objects were: 140644
>> Writing annotation file ...
2023-02-25 21:26:24,701 - distributed.core - INFO - Event loop was unresponsive in Nanny for 3.00s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 3.00s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:25,782 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 4.08s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 4.08s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:25,845 - distributed.core - INFO - Event loop was unresponsive in Nanny for 4.14s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 4.14s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:25,854 - distributed.core - INFO - Event loop was unresponsive in Nanny for 4.15s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 4.15s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:27,345 - distributed.core - INFO - Event loop was unresponsive in Nanny for 5.49s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 5.49s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:27,367 - distributed.core - INFO - Event loop was unresponsive in Nanny for 4.76s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 4.76s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:27,370 - distributed.core - INFO - Event loop was unresponsive in Nanny for 4.76s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 4.76s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:28,505 - distributed.core - INFO - Event loop was unresponsive in Nanny for 4.82s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 4.82s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:29,234 - distributed.core - INFO - Event loop was unresponsive in Nanny for 4.53s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 4.53s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:29,661 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 3.88s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 3.88s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:29,665 - distributed.core - INFO - Event loop was unresponsive in Nanny for 3.82s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 3.82s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-02-25 21:26:29,665 - distributed.core - INFO - Event loop was unresponsive in Nanny for 3.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 3.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
>> Does JSON file exist: True
In the NucleiDetection example
<https://github.com/DigitalSlideArchive/HistomicsTK/blob/master/histomicstk/cli/NucleiDetection/NucleiDetection.py#L276>
you store all predictions in a huge list and then it seems like you write
them to disk all at once. Perhaps I'm reaching a limit of how many objects
I can write? I guess that is what the error message is hinting at.
What happens is that the program succeeds after this, but the annotation
file is not available after. Nanny closes right after these "warnings".
Might be due to it being unresponsive for too long - or maybe this is just
done after each process, not sure.
Is there a better way in HistomicsTK to write thousands/millions of
objects as annotations on disk? It would be ideal if all of these were
stored in the same JSON annotation file.
—
Reply to this email directly, view it on GitHub
<#977 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFODTU3EXDPEIHZCQWS6JLWZJ3ZZANCNFSM6AAAAAAVFTSKWU>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Hehe, yup. Do you mean that I've reached a limit of how large a single annotation can be in the mongo database? That is, a single JSON file cannot be this large? Does that mean that I would have to split nuclei object boundaries in two or more separate annotation JSON files? Isn't that kind of annoying? Is there no other way? If I understand it correctly, it is not a memory issue or JSON issue, but rather a mongo issue, right? |
|
So I think you may have reached a practical limit based on what we have
tested for performance...
We have similar annotation docs that are just as large...
It's mostly an issue of whether we have the proper indexes or time outs..
if you have a slow connection and it takes 6 minutes to upload your giant
json blob.. and our default timeout it five mins for a connection, etc etc..
So the system can handle it in theory and in practice, there may just wind
up being some indexes or parametes in the default mongo container params.
Basically tuning things may be helpful...
…On Sat, Feb 25, 2023, 4:47 PM André Pedersen ***@***.***> wrote:
So that may be a mongo issue.. with that.many nuclei you may want to
create several documents..
Hehe, yup. Do you mean that I've reached a limit of how large a single
annotation can be in the mongo database? That is, a single JSON file cannot
be this large?
Does that mean that I would have to split nuclei object boundaries in two
or more separate annotation JSON files? Isn't that kind of annoying? Is
there no other way? If I understand it correctly, it is not a memory issue
or JSON issue, but rather a mongo issue, right?
—
Reply to this email directly, view it on GitHub
<#977 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFODTXUN3M6NWGGVLXH2K3WZJ4XJANCNFSM6AAAAAAVFTSKWU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
OK, great. Then I will wait until you have tuned some values before I make a new attempt. When you have made an attempt, I could also try to run nuclei segmentation on 40x instead of 10x, which should produce far more objects. For now I can work on a different task that generates less objects. I also wanted to add patch-wise classifier support, which likely means that I should store annotations as rectangles, similar to what QuPath does. Its basically a heatmap/mosaic I would then like to render.
Aaah, I see. Just note that writing the annotation file I have now I believe actually is quite fast, even on my HDD, but I might be wrong. I mean it is not going to take a millisecond, but it sure won't take minutes. But as you say, adjusting when timeouts occur should resolve this. |
|
As I have removed Dask (for now) for my custom plugin, I no longer get the I figured it was smart to read up a little on what DSA actually does in the backend. From going down the rabbit hole I learned that MongoDB, which DSA depends on, has a document size limit of 16 MB. I checked the file that was generated by my plugin and it was closer to 32 MB. Hence, there should be an issue trying to save these. However, I also see that using the GridFS API you can get around this limitation. This is exactly what Girder seems to do (see here), which seems to handle most of the backend logic in DSA. Hence, it makes sense that the issue is related to some timeouts, but that would also mean that there is something I would need to set in Girder then to resolve this issue, no? I did not see that many options to change this behaviour, but I might be wrong... Sadly, as I have removed Dask, it seems like everything went just fine. That is, there was no warnings or errors shown. It succeeded to write the file to JSON, but the annotation is not available after processing. Likely due to it failing to insert it into the DB as a document (timeout issue), which I believe is outside the scope of what i can set in the plugin itself at least. So yeah, I learned some new stuff, but haven't seen exactly what to change to fix this. Perhaps there are these timeouts params you would need to tune: |
|
Last thing. In the source code I came across this comment: From this, I guess one should be able to change these by modifying the URI here: I made an attempt and restarted the docker container, but annotations are still not showing - no file is produced, or at least not available in HistomicsUI. I tried setting the params to a stupidly high value, like so: Perhaps I wrote something wrong? |
|
Oh, or maybe, in the NucleiDetection example, you are only providing a regular MongoDB document, when it needs to be prepared differently to work with GridFS? See here. I noticed this from the source code here. Anyways kind of annoying that I'm not getting any verbose when or why this fails to write/load in DSA/HistomicsUI. |
|
We should have no problem with millions of annotation elementss. Internally, each polygon (annotation element) is stored as a separate mongo document and there is a parent mongo document for the annotation (in a different collection). If we emit a huge json file, it does get parsed in memory and validated -- the girder logs will show this, then committed to database. The mongo connection timeout shouldn't be the factor, either. In the Nuclei Detection example, the results are sent back as a file, uploaded into the system, then read, parsed, validated, and committed. The commit time usually isn't much per document. The logs will report progress "routinely" so if you are looking at the girder logs you should be able to see where it is in the process. |
|
I've now managed to convert both patch-wise segmentation and classification results to the appropriate format, and both render well. However, if I run inference on the full WSI with the segmentation model (which generates quite a lot of objects), the entire job is successful (so there are no indications from the website's job logs that something went wrong), but the result in the end is not added as part of available annotations nor is it rendered. I could take another look at the girder logs in the terminal; that is where I launched DSA, however, I could not see anything indicating that something failed or what the problem actually was. I will take another look soon and pay close attention to the logs after the job is successful. EDIT: I also check the file size of the generated JSON file after saving, and it does indeed save something, quite a lot so (~64 MB for one WSI I tested on). Hence, I believe the plugin itself is fine. If you want to take a look, you can find the CLI code of my plugin here. It is a work in progress, but at least you can use that for further debugging, in necessary. |
|
The problem was due to an unintended limitation when reading the annotation. This has now been addressed in girder/large_image#1075. See discussion here. Hence, we are able to save millions of annotations which were originally stored in pyramidal TIFF but converted to the expected JSON format. As this is now working as intended, and there is not really a need for a built in method for pyramidal TIFF -> JSON converter, I'm closing this issue. A PR could be made in the future, if others request such a conversion method. |
I'm developing my own custom plugin partly based on HistomicsTK (to be used in DSA), where I have used a different backend (FAST) to run some algorithm that generates a tiled, pyramidal TIFF segmentation image.
The TIFF is quite standard stuff. It is a pyramidal, tiled TIFF where each uint is assigned to a specific class. It can be generated on full resolution and thus the segmentation image coordinates is one-to-one to the original WSI. Hence, what we do in another software, FastPathology is simply to render the segmentation on top of the WSI as overlay.
Then to be able to add the annotations and render these, I need to convert it into a JSON format, which you talk about here.
However, I had the impression that there was support for pyramidal TIFF annotations as well in HistomicsTK, and thus I was wondering if there is a convenience method for this conversion?
Otherwise, I guess the only things I need to build the correct JSON file, is to get the girder client parameters and ground truth codec info, and then write the border coordinates of every single segmentation object inside the pyramidal TIFF. However, from the documentations here it seemed more complicated than it needed to be if we already have the pyramidal TIFF segmentation image.
Any suggestions?
The text was updated successfully, but these errors were encountered: