GUI gets stuck at "Starting training" while the chunk on Co-lab shows "Training 0 of max 60 epochs"

[TheaKristensen]

Hi,
I often experience that the Rootpainter GUI gets stuck at "Starting training" while the chunk on Co-lab shows "Training 0 of max 60 epochs".
Sometimes after being stuck for at while it changes, but often it stays there. Does this mean, that Rootpainter is not training? Or can it be running in the background without me being able to see it.

I hope my question makes sense.

KR Thea

[TheaKristensen]

I will add that often it doesn't just get stuck there, but actually looks like this when I open the program and doesn't change.

[Abe404]

Hi Thea,

Thanks for trying RootPainter and for reaching out about the problem!

"I often experience that the Rootpainter GUI gets stuck at "Starting training" while the chunk on Co-lab shows "Training 0 of max 60 epochs".

It sounds like it might be a sync issue. If it is taking some time for the message from the server to get to the client. Perhaps there could be other files syncing in your local google drive? For example, if you had just added a dataset. Or an intermittent internet connection?

It might just be that you are surprised at how slow things are. Unfortunately the client-server communication is limited by the sync speed of google drive. I've been looking into alternatives but haven't found anything free and convenient yet.

" can it be running in the background without me being able to see it."

Yes, if you can see RootPainter training in colab then it is training. It sounds like the client doesn't tell you this because the message has not yet been syned from the server to the client. The client-server IO in RootPainter happens via google drive and google drive desktop client. Sometimes sync can be a little slow due to this.

I just did a quick test myself using the biopores data with the colab notebook available at:
https://colab.research.google.com/drive/104narYAvTBt-X4QEDrBSOZm_DRaAKHtA?usp=sharing

And it seemed to work OK with the message of 'Training 0 of max 60 epochs without progress' showing up on the client (after it showed up in the colab notebook).

If you think something might be wrong with RootPainter itself (and not just sync speed slow) then the best thing is to write step by step instructions for me to reproduce the bug.

Basically tell me exactly which steps to follow, which behaviour you expected and which behaviour you experienced. Including the version of the software used would also be helpful.

Kind regards,
Abraham

[Abe404]

HI Thea,

I'm happy to hear you have found the software easy and accessible! That was the aim.

I have added an issue to document the problems with sync speed, in which I have referenced this discussion: #61

I think the colab tutorial works OK but I'd really like to make it faster when I get time, including the sync performance.

Kind regards,
Abraham

[TheaKristensen]

Regarding the tutorial: I think it is clear and well written, making it very easy to get started.

From my point of view it would be nice with some more information/tutorial on what to do once the training has finished.
There is some information in the article, but it would be nice with a similar step-by-step guide on how to go about the next part. (Segmenting, checking/validating/testing, possibly expanding training set, extracting results (?)).

I want to add that the above is meant as a suggestion, not a critique:-)

[Abe404]

Thanks for the suggestion. I totally agree. For now the best example of how to do that is probably this recent YouTube video demo (linked to the time where I start to explain extracting segmentations and results):
https://youtu.be/73u73tBvRO4?t=6809

I've updated the colab tutorial to link to this part of the video at the end and also to mention the other video where these steps are covered.

See inserted image showing updates:

I would like to eventually both create a better video and also include step by step instructions with screenshots, as you suggest.

[Abe404]

I created an issue to keep track of this #62. I agree the colab tutorial doesn't cover everything and a follow up guide including the next steps could be helpful.

[TheaKristensen]

Thank you again Abraham. Great video, I’ll hopefully be able to test it in one of the coming days :-)