Steps for Multi-label 2D Segmentation in Endoscopy App

[bilalcodehub]

Hi,

I want to customise the built-in endoscopy app for a multi-labeling 2D segmentation in 'monailabel'. I want to capture everything I needed or did for this customisation with the hope that someone (newbie like me) might find it useful, plus to get community support which is key to keep me get going with this task.

The first step was to define the data source containing images from surgical videos. I was able to achieve this by mapping the host folder with monailabel docker using the following command:

docker run -it --rm --gpus all --ipc=host --net=host -v /home/big-deal/Documents/mlworks/datasets:/opt/monai/datasets projectmonai/monailabel:latest bash

The -v switch maps the host folder /home/big-deal/Documents/mlworks/datasets with docker volume/opt/monai/datasets to keep the work persistent when the docker is closed and make life much easier. Then I accessed host folder from the monailabel server using the following command:

monailabel start_server --app apps/endoscopy --studies datasets/images_short --conf models tooltracking --conf epistemic_enabled true --conf epistemic_top_k 10 --conf cvat_segment_size 10^C

It worked like a charm.

The next step is to define custom labels for fourteen surgical instruments. I added the line:self.lables = {'tool-1':1,'tool-b':2, 'tool-c':3,...}, to apps/endoscopy/lib/configs/tooltracking.py file and started the server using the following command:

monailabel start_server --app apps/endoscopy --studies datasets/images --conf models tooltracking --conf epistemic_enabled true --conf epistemic_top_k 20 --conf cvat_segment_size 10

The server runs and starts pushing tasks to the CVAT docker. However, I still can't see my custom labels. Check the screenshot below:

CVAT is still showing the Tool, InBody, and OutBody for the task. Is that the right place to make the changes? Do I have to add labels to CVAT to make it happen?

[SachidanandAlle]

-c cvat_labels option is there.. but missed the documentation
https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/endoscopy/main.py#L106

https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/datastore/cvat.py#L52
you can pass json string some thing like -c cvat_labels '[{"name":"Tool","attributes":[],"color":"#66ff66"},{"name":"InBody","attributes":[],"color":"#ff0000"},{"name":"OutBody","attributes":[],"color":"#0000ff"}]'

[bilalcodehub]

@SachidanandAlle thanks, it worked.

monailabel start_server --app apps/endoscopy \
--studies datasets/images \
--conf models tooltracking \
--conf epistemic_enabled true \
--conf epistemic_top_k 10 \
--conf cvat_segment_size 10 \
--conf cvat_labels '[{"name":"hand","attributes":[],"color":"#66ff66"},{"name":"suture","attributes":[],"color":"#ff0000"},{"name":"needle-driver","attributes":[],"color":"#0000ff"}]'

The CVAT IDE has picked up the labels. See the images below:

A couple of questions:

1. Can we specify a file name for the labels like --conf cvat_labels <labels-file-path.txt> to keep the monailabel start_server simple when a task has many labels?
2. Will the model for tool tracking will pick up the same labels automatically during the training or do we need to specify it in one of the .py files inside the configs folder?
3. How to start training the tooltracking model from scratch with a few manually annotated images? I searched for the use_pretrained_model option like in the radiology segmentation app but I can't see it in the tooltracking.py file inside the endoscopy app.

[SachidanandAlle]

you can put labels in file and possible use --conf cvat_labels cat labels-file.txt or you can support loading it from file by modifying your app code..

tooltracking.. u should be train from scratch.. i didn't get where you are currently stuck. you should try using directly main.py from endoscopy to train your model from scratch.. which shall be more easy.

Since it's a monaibundle.. you can use the bundle directly to train a model.

[bilalcodehub]

Thanks @SachidanandAlle.

I tried --conf cvat_labels cat datasets/tool-labels.txt, but it didn't work. Not sure how to structure the label file. Shall the file contain a full string in the form '[{"name":"hand","attributes":[],"color":"#66ff66"},{"name":"suture","attributes":[],"color":"#ff0000"},{"name":"needle-driver","attributes":[],"color":"#0000ff"}]' or comma-separated strings with three columns name, attributes and color for each tool on new lines?

I also tried changing MyApp by passing the labels argument

super().__init__(
            app_dir=app_dir,
            studies=studies,
            conf=conf,
            name=f"MONAILabel - Endoscopy ({monailabel.__version__})",
            description="DeepLearning models for endoscopy",
            version=monailabel.__version__,
            labels={'hand':1,"suture":2,'needle-driver':3},
        )

and also tried in the init file:

self.conf["cvat_labels"]='[{"name":"hand","attributes":[],"color":"#66ff66”}, {“name":"suture","attributes”:[],”color":"#ff0000”},{“name":"needle-driver","attributes":[],"color":"#0000ff"}]'

both didn't work.

Regarding the training of tooltracking model from scratch, I can try running the training directly using main.py as you suggested. Thanks.

But when I label images in the first iteration produced by the monailabel and marked the task as finished in CVAT, I expect monailabel to start fine-tuning based on the process listed in the GitHub doc (see the screenshot below) but this is not happening. Any idea what might be wrong?

[SachidanandAlle]

Will reply to your questions shortly with example/solution.. meanwhile can you please let the MONAI team know the usecase that you are trying. Some background/context. This will help to prioritize and support things better.

[bilalcodehub]

Thanks that will be much appreciated. I'm trying to annotate surgical videos for all the tools and tissues present in the image to train a multi-label segmentation model. I hope this helps. Let me know if you need more information.

[SachidanandAlle]

Little bit more.. like how big is the usecase.. is it for institute or hospital etc.. how many annotator involved.. end objective etc.. from product perspective..

[bilalcodehub]

This work is part of one of my university research projects in partnership with NHS Bristol Heart Institute to co-create surgical data science tools. We have an access to a massive dataset of over a thousand procedures with an average video time of two hours, entailing 200K images each. There will be up to three annotators but it depends on the time our collaborating surgeons can find to spend on the annotation task. I wish to configure monailabel to make this a better experience for the clinical team and get the best out of their engagement. Also active learning seems to be the best strategy which comes off-the-shelf with monailabel. Thanks for your great work.

[SachidanandAlle]

For the label file.. I suggest you change things here.. you can define your own --conf saying cvat_labels_file

And then use the same load your json file and pass it to cvat datastore.
https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/endoscopy/main.py#L98-L111

You can read the file.json and pass labels when creating CVATDatastore object.

   filename = self.conf.get("cvat_labels_file")
   with open(filename, "r") as fp:
     labels = json.load(fp)

    def init_datastore(self) -> Datastore:
        if settings.MONAI_LABEL_DATASTORE_URL and settings.MONAI_LABEL_DATASTORE.lower() == "cvat":
            logger.info(f"Using CVAT: {self.studies}")
            return CVATDatastore(labels=labels, ...,)

        return super().init_datastore()

[bilalcodehub]

Thanks @SachidanandAlle for the support. It worked like a charm.

I made a few changes to the code to make it read from the file. The main.py looks like the following:

def create_label_list(self,file_path):
        import random
        with open(file_path, 'r') as f:
            labels = [line.strip() for line in f]
        color_codes = ['#%06x' % random.randint(0, 0xFFFFFF) for _ in range(len(labels))]
        label_list = []
        for i, label in enumerate(labels):
            label_dict = {"name": label, "attributes": [], "color": color_codes[i]}
            label_list.append(label_dict)
        return label_list

    def init_datastore(self) -> Datastore:
        if settings.MONAI_LABEL_DATASTORE_URL and settings.MONAI_LABEL_DATASTORE.lower() == "cvat":
            logger.info(f"Using CVAT: {self.studies}")
            label_file_path = self.conf.get("cvat_labels_file")
            label_list = self.create_label_list(label_file_path)
            return CVATDatastore(
                datastore_path=self.studies,
                api_url=settings.MONAI_LABEL_DATASTORE_URL,
                username=settings.MONAI_LABEL_DATASTORE_USERNAME,
                password=settings.MONAI_LABEL_DATASTORE_PASSWORD,
                project=self.conf.get("cvat_project", "MONAILabel"),
                task_prefix=self.conf.get("cvat_task_prefix", "ActiveLearning_Iteration"),
                image_quality=int(self.conf.get("cvat_image_quality", "70")),
                labels=label_list,
                normalize_label=strtobool(self.conf.get("cvat_normalize_label", "true")),
                segment_size=int(self.conf.get("cvat_segment_size", "1")),
                extensions=settings.MONAI_LABEL_DATASTORE_FILE_EXT,
                auto_reload=settings.MONAI_LABEL_DATASTORE_AUTO_RELOAD,
            )

        return super().init_datastore()

The monailabel server is started with the following command:

monailabel start_server --app apps/endoscopy --studies datasets/images --conf models tooltracking --conf epistemic_enabled true --conf epistemic_top_k 10 --conf cvat_segment_size 10 --conf multi_gpu true --conf cvat_labels_file datasets/tool-labels.txt

And CVAT started to pick all the labels. See below:

A quick question: Do these labels be picked by the monailabel training routines automatically or I still need to add the same code somewhere else?

[bilalcodehub]

Please ignore the question regarding monailabel using the same labels. The main() is calling the MyApp class so will certainly pick these labels.

[bilalcodehub]

@SachidanandAlle: While monailabel is up and running, now I can start the manual annotation of surgical videos.

Just wondering if there is a Grow from Seed feature available for the endoscopy app as it is available in the radiology app 3D slicer plugin?

It is not easy to manually segment all tools even for a few examples.

[SachidanandAlle]

Train a simple deep grow model..

[bilalcodehub]

Thanks @SachidanandAlle.

1. Is there anything relevant in the docs I can read more about training a deep grow model?
2. I've started training the tooltracking model manually (python apps/endoscopy/main.py) since it won't begin after annotations in CVAT get finished and marked completed. I don't know why Monailabel is not starting the training itself. Any idea?
3. I keep getting the CUDA out-of-memory error. I have 4 GPUs on the system with 8 gigs of memory. I know it is not much but should be enough to train a segmentation model on 30 images in the dataset. How to allocate a different GPU to the training process? I have reduced the batch size to two but nothing seems working.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 7.80 GiB total capacity; 6.50 GiB already allocated; 12.56 MiB free; 6.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[SachidanandAlle]

@tangy5 can help you a bit here..

[tangy5]

HI @bilalcodehub ,

1. For the deepgrow model, you could refer to this tutorial, you can setup a separate workflow for a deepgrow model. It should be working on 2D and 3D images. There is no deepgrow model for CVAT+monailabel yet, there is only deepEdit, but you could prototype a deepgrow model with your annotated labels.
2. I feel there might be some issues of the communication between CVAT and MONAI Label, I can't tell what's wrong here, if you could provide more details, we can dig into it. But we now have a tutorial: https://github.com/Project-MONAI/tutorials/blob/main/monailabel/monailabel_endoscopy_cvat_tooltracking.ipynb. Not sure these steps are helpful, but you could double check on each step.
3. Typically, we can modify visible torch device by: such as CUDA_VISIBLE_DEVICES=1,2,3 if you don't wan tot use 0.

[bilalcodehub]

Hi @tangy5,

Thanks for such a detailed response. Much appreciated.

1. You referred to a tutorial for deepgrow training but forget to add the link. Mind sharing it? Thanks.
2. I followed most steps in the tutorial except the one to edit the train.json file as I couldn't locate it then. But with a bit of familiarity with MONAILabel, it is located in the apps/endoscopy/model/endoscopic_tool_segmentation/cofigs folder. I was expecting the training function will do splitting itself for the folder provided through the studies switch. Also, MONALabel can communicate with CVAT as it is currently pushing the first iteration. Additionally, when CVAT job is marked complete, MONAILabel picks up the completion and creates masks in the labels/final folder. I can dig more and share logs. Any specific info you need to troubleshoot the issue?
3. I tried running export CUDA_VISIBLE_DEVICES=1,2,3 before manually running the training script, but it still got the following CUDA error. I am not sure why the training over 30 images with a batch size of two (2) won't fit in the GPU.

root@turing:/opt/monai# python apps/endoscopy/main.py 
Main called
studies location is: datasets/images
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.inbody.InBody'>
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.tooltracking.ToolTracking'>
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (__main__:81) - +++ Adding Model: tooltracking => lib.configs.tooltracking.ToolTracking
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (lib.configs.tooltracking:51) - EPISTEMIC Enabled: True; Samples: 3
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (__main__:85) - +++ Using Models: ['tooltracking']
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: datasets/images
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (monailabel.datastore.local:129) - Auto Reload: False; Extensions: ['*.png', '*.jpg', '*.jpeg', '.xml']
[2023-02-28 09:36:49] [983] [MainThread] [INFO] (monailabel.datastore.local:576) - Invalidate count: 0
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:138) - +++ Adding Inferer:: tooltracking => <lib.infers.tooltracking.ToolTracking object at 0x7f66f6814f10>
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:152) - +++ Adding Trainer:: tooltracking => <lib.trainers.tooltracking.ToolTracking object at 0x7f66f6814fd0>
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:170) - +++ Adding Strategy:: tooltracking_epistemic => <monailabel.tasks.activelearning.epistemic.Epistemic object at 0x7f66eceec310>
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:173) - Active Learning Strategies:: ['random', 'tooltracking_epistemic']
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:187) - +++ Adding Scoring Method:: tooltracking_epistemic => <lib.scoring.cvat.CVATEpistemicScoring object at 0x7f66eceec370>
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:190) - Active Learning Scoring Methods:: ['tooltracking_epistemic']
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (__main__:279) - {'total': 30, 'completed': 30, 'label_tags': {'final': 30}}
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:194) - Train Request: {'model': 'tooltracking', 'max_epochs': 10, 'dataset': 'Dataset', 'train_batch_size': 2, 'val_batch_size': 2, 'multi_gpu': False, 'val_split': 0.1}
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:164) - Total Records in Dataset: 30; Validation Split: 0.1
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:174) - Total Records for Training: 27
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:175) - Total Records for Validation: 3
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:206) - Using Multi GPU: False; GPUS: [0, 1]
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:207) - CUDA_VISIBLE_DEVICES: 1,2,3
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:210) - Using device: cuda; Type: <class 'str'>
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:221) - (Experiment Management) Tracking: mlflow
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:222) - (Experiment Management) Tracking URI: 
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:223) - (Experiment Management) Experiment Name: tooltracking
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:224) - (Experiment Management) Run Name: None
[2023-02-28 09:36:50] [983] [MainThread] [INFO] (monailabel.tasks.train.bundle:181) - Add Checkpoint Loader for Path: apps/endoscopy/model/endoscopic_tool_segmentation/models/model.pt
2023-02-28 09:36:50,686 - INFO - --- input summary of monai.bundle.scripts.run ---
2023-02-28 09:36:50,687 - INFO - > runner_id: 'training'
2023-02-28 09:36:50,687 - INFO - > meta_file: 'apps/endoscopy/model/endoscopic_tool_segmentation/configs/metadata.json'
2023-02-28 09:36:50,687 - INFO - > config_file: 'apps/endoscopy/model/endoscopic_tool_segmentation/configs/train.json'
2023-02-28 09:36:50,687 - INFO - > tracking: 'mlflow'
2023-02-28 09:36:50,687 - INFO - > bundle_root: 'apps/endoscopy/model/endoscopic_tool_segmentation'
2023-02-28 09:36:50,687 - INFO - > train#trainer#max_epochs: 10
2023-02-28 09:36:50,688 - INFO - > train#dataset#data: [{'image': '/opt/monai/datasets/images/000011.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000011.png'},
 {'image': '/opt/monai/datasets/images/000012.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000012.png'},
 {'image': '/opt/monai/datasets/images/000013.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000013.png'},
 {'image': '/opt/monai/datasets/images/000014.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000014.png'},
 {'image': '/opt/monai/datasets/images/000015.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000015.png'},
 {'image': '/opt/monai/datasets/images/000016.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000016.png'},
 {'image': '/opt/monai/datasets/images/000017.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000017.png'},
 {'image': '/opt/monai/datasets/images/000018.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000018.png'},
 {'image': '/opt/monai/datasets/images/000019.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000019.png'},
 {'image': '/opt/monai/datasets/images/000020.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000020.png'},
 {'image': '/opt/monai/datasets/images/000021.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000021.png'},
 {'image': '/opt/monai/datasets/images/000022.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000022.png'},
 {'image': '/opt/monai/datasets/images/000023.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000023.png'},
 {'image': '/opt/monai/datasets/images/000024.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000024.png'},
 {'image': '/opt/monai/datasets/images/000025.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000025.png'},
 {'image': '/opt/monai/datasets/images/000026.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000026.png'},
 {'image': '/opt/monai/datasets/images/000027.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000027.png'},
 {'image': '/opt/monai/datasets/images/000028.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000028.png'},
 {'image': '/opt/monai/datasets/images/000029.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000029.png'},
 {'image': '/opt/monai/datasets/images/000030.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000030.png'},
 {'image': '/opt/monai/datasets/images/000031.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000031.png'},
 {'image': '/opt/monai/datasets/images/000032.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000032.png'},
 {'image': '/opt/monai/datasets/images/000033.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000033.png'},
 {'image': '/opt/monai/datasets/images/000034.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000034.png'},
 {'image': '/opt/monai/datasets/images/000035.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000035.png'},
 {'image': '/opt/monai/datasets/images/000036.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000036.png'},
 {'image': '/opt/monai/datasets/images/000037.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000037.png'}]
2023-02-28 09:36:50,688 - INFO - > device: 'cuda'
2023-02-28 09:36:50,688 - INFO - > train#handlers: [{'_target_': 'CheckpointLoader',
  'load_dict': {'model': '$@network'},
  'load_path': 'apps/endoscopy/model/endoscopic_tool_segmentation/models/model.pt',
  'strict': False},
 {'_target_': 'ValidationHandler',
  'epoch_level': True,
  'interval': '@val_interval',
  'validator': '@validate#evaluator'},
 {'_target_': 'StatsHandler',
  'output_transform': "$monai.handlers.from_engine(['loss'], first=True)",
  'tag_name': 'train_loss'},
 {'_target_': 'TensorBoardStatsHandler',
  'log_dir': '@output_dir',
  'output_transform': "$monai.handlers.from_engine(['loss'], first=True)",
  'tag_name': 'train_loss'},
 {'_target_': 'LrScheduleHandler',
  'lr_scheduler': '@lr_scheduler',
  'print_lr': True}]
2023-02-28 09:36:50,688 - INFO - > experiment_name: 'tooltracking'
2023-02-28 09:36:50,688 - INFO - > validate#dataset#data: [{'image': '/opt/monai/datasets/images/000038.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000038.png'},
 {'image': '/opt/monai/datasets/images/000039.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000039.png'},
 {'image': '/opt/monai/datasets/images/000040.jpg',
  'label': '/opt/monai/datasets/images/labels/final/000040.png'}]
2023-02-28 09:36:50,688 - INFO - ---


<class 'monai.transforms.utility.array.AsChannelFirst'>: Class `AsChannelFirst` has been deprecated since version 0.8. please use MetaTensor data type and monai.transforms.EnsureChannelFirst instead.
<class 'monai.transforms.utility.array.AddChannel'>: Class `AddChannel` has been deprecated since version 0.8. please use MetaTensor data type and monai.transforms.EnsureChannelFirst instead.
Loading dataset: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  4.17it/s]
[2023-02-28 09:36:52] [983] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 10 epochs
[2023-02-28 09:36:52] [983] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:138) - Restored all variables from apps/endoscopy/model/endoscopic_tool_segmentation/models/model.pt
[2023-02-28 09:36:54] [983] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:1086) - Current run is terminating due to exception: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-02-28 09:36:54] [983] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:180) - Exception: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 1068, in _run_once_on_dataset_as_gen
    self.state.output = self._process_function(self, self.state.batch)
  File "/opt/monai/monai/engines/trainer.py", line 234, in _iteration
    _compute_pred_loss()
  File "/opt/monai/monai/engines/trainer.py", line 218, in _compute_pred_loss
    engine.state.output[Keys.PRED] = engine.inferer(inputs, engine.network, *args, **kwargs)
  File "/opt/monai/monai/inferers/inferer.py", line 85, in __call__
    return network(inputs, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/flexible_unet.py", line 336, in forward
    enc_out = self.encoder(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/efficientnet.py", line 631, in forward
    x = block(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/efficientnet.py", line 191, in forward
    x = self._bn0(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 170, in forward
    return F.batch_norm(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2431, in batch_norm
    return handle_torch_function(
  File "/opt/conda/lib/python3.8/site-packages/torch/overrides.py", line 1530, in handle_torch_function
    result = torch_func_method(public_api, types, args, kwargs)
  File "/opt/monai/monai/data/meta_tensor.py", line 268, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 1265, in __torch_function__
    ret = func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2446, in batch_norm
    return torch.batch_norm(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-02-28 09:36:54] [983] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:992) - Engine run is terminating due to exception: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-02-28 09:36:54] [983] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:180) - Exception: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 959, in _internal_run_as_gen
    epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 1087, in _run_once_on_dataset_as_gen
    self._handle_exception(e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 636, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/opt/monai/monai/handlers/stats_handler.py", line 181, in exception_raised
    raise e
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 1068, in _run_once_on_dataset_as_gen
    self.state.output = self._process_function(self, self.state.batch)
  File "/opt/monai/monai/engines/trainer.py", line 234, in _iteration
    _compute_pred_loss()
  File "/opt/monai/monai/engines/trainer.py", line 218, in _compute_pred_loss
    engine.state.output[Keys.PRED] = engine.inferer(inputs, engine.network, *args, **kwargs)
  File "/opt/monai/monai/inferers/inferer.py", line 85, in __call__
    return network(inputs, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/flexible_unet.py", line 336, in forward
    enc_out = self.encoder(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/efficientnet.py", line 631, in forward
    x = block(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/efficientnet.py", line 191, in forward
    x = self._bn0(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 170, in forward
    return F.batch_norm(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2431, in batch_norm
    return handle_torch_function(
  File "/opt/conda/lib/python3.8/site-packages/torch/overrides.py", line 1530, in handle_torch_function
    result = torch_func_method(public_api, types, args, kwargs)
  File "/opt/monai/monai/data/meta_tensor.py", line 268, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 1265, in __torch_function__
    ret = func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2446, in batch_norm
    return torch.batch_norm(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
  File "apps/endoscopy/main.py", line 455, in <module>
    main()
  File "apps/endoscopy/main.py", line 281, in main
    train_tooltracking(app)
  File "apps/endoscopy/main.py", line 379, in train_tooltracking
    res = app.train(
  File "/opt/conda/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 429, in train
    result = task(request, self.datastore())
  File "/opt/conda/lib/python3.8/site-packages/monailabel/tasks/train/bundle.py", line 296, in __call__
    self.run_single_gpu(request, overrides)
  File "/opt/conda/lib/python3.8/site-packages/monailabel/tasks/train/bundle.py", line 302, in run_single_gpu
    monai.bundle.run(
  File "/opt/monai/monai/bundle/scripts.py", line 742, in run
    return [parser.get_parsed_content(i, lazy=True, eval_expr=True, instantiate=True) for i in ensure_tuple(runner_id_)]
  File "/opt/monai/monai/bundle/scripts.py", line 742, in <listcomp>
    return [parser.get_parsed_content(i, lazy=True, eval_expr=True, instantiate=True) for i in ensure_tuple(runner_id_)]
  File "/opt/monai/monai/bundle/config_parser.py", line 271, in get_parsed_content
    return self.ref_resolver.get_resolved_content(id=id, **kwargs)
  File "/opt/monai/monai/bundle/reference_resolver.py", line 185, in get_resolved_content
    return self._resolve_one_item(id=id, **kwargs)
  File "/opt/monai/monai/bundle/reference_resolver.py", line 155, in _resolve_one_item
    self._resolve_one_item(id=d, waiting_list=waiting_list, **kwargs)
  File "/opt/monai/monai/bundle/reference_resolver.py", line 167, in _resolve_one_item
    item.evaluate(globals={f"{self._vars}": self.resolved_content}) if run_eval else item
  File "/opt/monai/monai/bundle/config_item.py", line 366, in evaluate
    return eval(value[len(self.prefix) :], globals_, locals)
  File "<string>", line 1, in <module>
  File "/opt/monai/monai/engines/trainer.py", line 53, in run
    super().run()
  File "/opt/monai/monai/engines/workflow.py", line 281, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 892, in run
    return self._internal_run()
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 935, in _internal_run
    return next(self._internal_run_generator)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 993, in _internal_run_as_gen
    self._handle_exception(e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 636, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/opt/monai/monai/handlers/stats_handler.py", line 181, in exception_raised
    raise e
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 959, in _internal_run_as_gen
    epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 1087, in _run_once_on_dataset_as_gen
    self._handle_exception(e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 636, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/opt/monai/monai/handlers/stats_handler.py", line 181, in exception_raised
    raise e
  File "/opt/conda/lib/python3.8/site-packages/ignite/engine/engine.py", line 1068, in _run_once_on_dataset_as_gen
    self.state.output = self._process_function(self, self.state.batch)
  File "/opt/monai/monai/engines/trainer.py", line 234, in _iteration
    _compute_pred_loss()
  File "/opt/monai/monai/engines/trainer.py", line 218, in _compute_pred_loss
    engine.state.output[Keys.PRED] = engine.inferer(inputs, engine.network, *args, **kwargs)
  File "/opt/monai/monai/inferers/inferer.py", line 85, in __call__
    return network(inputs, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/flexible_unet.py", line 336, in forward
    enc_out = self.encoder(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/efficientnet.py", line 631, in forward
    x = block(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/monai/monai/networks/nets/efficientnet.py", line 191, in forward
    x = self._bn0(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 170, in forward
    return F.batch_norm(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2431, in batch_norm
    return handle_torch_function(
  File "/opt/conda/lib/python3.8/site-packages/torch/overrides.py", line 1530, in handle_torch_function
    result = torch_func_method(public_api, types, args, kwargs)
  File "/opt/monai/monai/data/meta_tensor.py", line 268, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 1265, in __torch_function__
    ret = func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2446, in batch_norm
    return torch.batch_norm(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.80 GiB total capacity; 6.79 GiB already allocated; 18.56 MiB free; 6.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
root@turing:/opt/monai#