GUI waiting for segmentation and trainer without error log

[GeraultTr]

Hi, first of all thank you for this amazing work. I am a PhD student trying to use RootPainter on large soil rhizotron images. I am also a model developper so I don't mind going into details.

I am facing an issue on a machine with NVIDIA Quadro P620, cuda toolkit 12.3 installed and Pytorch properly detecting the GPU on trainer launch. I am trying to run both trainer and painter locally on the same machine using a common sync folder.

The trainer creates the expected file structure, and seems to consume instructions properly. The problem is that when I start the painter, the release or the main from your repo, I get a pending "Network not training", "Starting training..." or "Loading Segmentation..." preventing me from annotating more than the first image. As the segmentation doesn't appear and my GPU ressources are not used, I think that the training is not performed but no particular error is logged by the trainer.

For further details, I get a batch size of 0 uppon trainer startup. Also I didn't used your pip install as the torch version was not detecting the GPU (AssertionError: Torch not compiled with CUDA enabled). So I either tried a manual installation from your requirements and pytorch.org, or an install with pip, uninstalling torch and then reinstall torch, torchvision and torchaudio (both for cuda 11.8 and 12.1). Also tried to install cuda with conda instead of the toolkit but it didn't change the issue.

Does this situation remind any common issue to you?

Thank you very much in advance for your help, I am looking forward to discuss with you and I remain available for further details.

Sincerely

[Abe404]

Dear Tristan,

For further details, I get a batch size of 0 uppon trainer startup.

There's not enough memory on your GPU to create a batch of 1. (I recommend 4+).

I noticed the NVIDIA Quadro P620 only has 2GB of memory. This is really small. You will need to reduce the default patch size to get RootPainter to work on a GPU this small. There is a command line argument for this when you run the trainer. But I recommend you use a more powerful GPU. RootPainter typically requires 12GB (or more) for the GPU memory. I recommend 16GB or more.

As your GPU has such a restricted memory, You may have more success running RootPainter via the colab notebook:
https://colab.research.google.com/drive/104narYAvTBt-X4QEDrBSOZm_DRaAKHtA?usp=sharing

Which will assign you a 'free' GPU from google.

Kind regards,
Abraham

[Abe404]

In the colab notebook there is a cell which shows how to run the trainer with a reduced patch size:

You can do something like this:

python main.py --patchsize 188

That said, I do not have much experience with using these smaller patch sizes with RootPainter so I am not really sure how it will effect your results, but it's something you could try if you want to get RootPainter running with your current GPU.

[GeraultTr]

Dear Abraham,

First of all thank you for your fast reply time. Yes I had no issues getting the collab setup to work, only issue being the limitations of the free collab GPU sessions and the fact that each of my rhizotron images weight approximately 500 Mb, with an overall dataset reaching easily sevral hundreds of Gb for 1 experimentations, so I will also quickly face the limitations of Drive's storage, as well as data transfer speed. This is why I attempted a local setup in the first place.

That said, I tried to lower the patchsize down to 92, but it didn't help building a batch, indeed the Quadro P620 isn't suited for that kind of use. I could have access to a 8Gb GeForce GTX 1080, but I fear I might end up in the same situation as you mention before.

So my questions are, what did you use during your PhD? What would you advise for such amount of data? Buying a capable enough graphic card or a subscription to an online GPU service?

Thank you in advance for your answer,

Sincerely,

Tristan

[Abe404]

Those are really big images. How big are they if you convert to JPEG with 95% quality compression? I agree it's better to set things up on your own hardware if you have the funds and know-how.

In the RootPainter study we used two Nvidia RTX 2080 Ti GPUs connected with NVLink.

These days I would suggest a single NVIDIA 4090 GPU as I believe it is the most cost effective option. It doesn't fit in any computer though so please be aware of space, power-supply and motherboard compatibility etc.

For further guidance regarding which GPU to buy I suggest Tim dettmers. see:
https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

The 8Gb GeForce GTX 1080 might be 'good enough' for some tasks. It might give you a batch size of 2. I believe it can be possible to train an OK model with a batch size of 2, but it's lower than what I recommend.

A subscription to an online service might make sense if you are going to be using the GPU infrequently or buying a GPU is not easy for you.

I have had some success with paperspace: https://www.paperspace.com
There are of course cloud GPUs available from google, amazon, Microsoft etc.

Personally, I have a strong preference for having my own computer with a GPU as I find this more straightforward for research work and I don't have to upload all data to the cloud.

Kind regards,
Abraham

[GeraultTr]

Dear Abraham,

Thank you for your answer. Those image are already jpeg compressed. In fact, they result from an assembly of more than 60 macro images intended at recomposing a high resolution image of the whole rhizotron (90 * 45 cm). This is why we need to keep such high resolution after stiching in order to keep root hair information and a clear contouring of the roots.

Just for supporting information on the minimum specs locally, I finally managed to train a model on the individual images prior stiching, using a laptop Quadro P600, restricting to batch size 1 and 3 workers due to the small 4 Gb Ram, and keeping the default patch size. That enabled a convenient local testing to convince my supervisors of the interest of purshasing a new hardware.

Then, I was just wondering (I can start a new discussion if you want), How would you segment the larger 1 Gb rhizotron images based on a model trained on individual tiles prior stiching (20 Mb)? Is there a way to increase the limit image size for inference?

Thank you in advance for your answer,

Kind regards,

Tristan

[Abe404]

Dear Tristan,

I'm glad you were able to find an interim solution and convince your supervisors to get better hardware with the Quadro P600. Nice work.

Just to clarify, working with smaller images in a training dataset and then segmenting larger images is part of the standard workflow for using RootPainter. I suggest reading into the 'create training dataset' functionality described in the paper in the 'Creating a dataset' part of the methods section https://nph.onlinelibrary.wiley.com/doi/full/10.1111/nph.18387

The mini-guide also refers briefly to this process. I discuss creating a training dataset to train the model and then also processing the original images i.e full size/resolution. https://github.com/Abe404/root_painter/blob/master/docs/mini_guide.md

Is there a way to increase the limit image size for inference?

I'm not yet sure which limit you are referring to. Perhaps my answer regarding creating a training dataset helps clarify. Training a RootPainter model using smaller images and then segmenting much bigger ones is standard practice. The large image will be automatically split up into patches (internally) to allow it to be segmented. The segmentations for each of the smaller patches will then be joined together to give you the single segmentation file for the large image. All of this happens automatically when you process your folder of large images with the 'segment folder' command in RootPainter (available from the network menu).

I hope that helps.

Kind regards,
Abraham

[GeraultTr]

Dear Abraham,

Thank you for your quick and detailed answer. Yes I was well aware of this workflow in RootPainter and I tried it following your tutorials and paper. However I chose to train the model with the small tiles I already had prior stiching the larger image as the resolution is very similar to what I would get from the 'Creating a dataset' process.

The issue I am facing occurs when trying to segment the larger images with the "segment folder" command in RootPainter. To be more precise, what I get is the following :

Exception loading C:\Users\....\R12_24_02_12.jpg Image size (977457730 pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.

And so on for all the images to segment. So this is the 178956970 threshold I was refering to, do you think this is hardcoded in RootPainter and could be changed or should I split these images and assemble them in parallel of the 'segment folder' command?

Thanks again for your insights,
Tristan

[GeraultTr]

Sorry, I answered without a basic search being in a rush these days. I found out that the error was pretty common with pillow, and increasing the threshold was enough :

Import PIL.Image
PIL.Image.MAX_IMAGE_PIXELS = 1e10

Thank you again for you reactive help, the results are amazing! I will try to further train the model on root hair features in the next days, would you have some documentation advice for the multiclass strategy? Or would you rather consider training a second model on this specific feature?

Kind regards,
Tristan

[Abe404]

Thanks Tristan,

I've made this change in the source code now too:
225eeb5
It's on the master branch (so you will only get it if you pull /clone the source again) but it should make it into the next release.
Please let me know if you have any comments or suggestions regarding my change.

I'm happy you are getting amazing results. I'm curious, do you have any images you could share? (with the segmentation shown overlaid perhaps).

I will try to further train the model on root hair features in the next days, would you have some documentation advice for the multiclass strategy? Or would you rather consider training a second model on this specific feature?

I have built a version of RootPainter that supports multi class, but to be honest I do not feel it is totally ready yet in terms of reliable training performance. For now I suggest to train a separate model, which with the right size images and annotation approach is usually pretty quick anyway (depending also on the difficulty, variation and noise in your dataset).

Kind regards,
Abraham

[GeraultTr]

Hello Abraham,

Thank you for the commit, it's achieving the desired behavior indeed.

Yes I will send you an overlay soon, the experiment is stil ongoing and the final shot of the rhizotron should demonstrate an interesting segmentation on a complex root system.

Ok thank you for your answer regarding multiclass segmentation, on short term I will then train feature specific models.

Simple curiosity, will you be at ISRR 2024?

Kind regards,
Tristan

[Abe404]

HI Tristan,

Nice to hear. I do not currently have plans to go to ISRR 2024. I hope you have a good time there!

Kind regards,
Abraham

[GeraultTr]

Hi Abraham,

After a long delay, here is an example segmentation time-series from tests with root-painter as promised : https://filesender.renater.fr/?s=download&token=c7257c6c-71de-4e2b-b91f-3185ad7b336b

The segmentation performance being very promising, would you have in mind an automatic annotation tool that would enable separating each root axis individually? I have in mind root-system-tracker (https://www.youtube.com/watch?v=SWEqnnOhIOU&pp=ygUTcm9vdCBzeXN0ZW0gdHJhY2tlcg%3D%3D), but it doesn't fit for soil rhizobox situations when we have interruptions on the data like what I've sent you, which would require integration capabilities using the derivative between two successive images.

Thank you in advance for any tip!

Sincerely,

Tristan

[Abe404]

Hi Tristan,

Sorry, somehow I missed this notification for your response! I think that's a really cool idea but I'm not yet sure how I would do that. I also missed your file download. How are things going with your project? Any updates? I'm also interested to discuss via zoom call or similar if you like.

Kind regards,
Abraham