Suggestions to improve the model? #1026

royak · 2022-09-28T16:00:40Z

royak
Sep 28, 2022

Hi All,
I am using the Monai label segmentation model on CT Lymph node data from https://wiki.cancerimagingarchive.net/display/Public/CT+Lymph+Nodes
I converted 90 of them, did a little bit of preprocessing, and saved them in a monai format folder:
TCIA_lymphnode_monailabel
I am working on GCP and connecting it with a tunnel to a local 3d slicer on MacBook
I switched to the new version of monailabel and tweaked scale intensity and learning rate and some other small changes which are visible in the code which I pasted under this issue, every time I run I get a lot of errors between epochs as shown here:

My dice score does not improve either.

Any suggestion to improve this model is greatly appreciated.

In this image
Ground truth: green
monailabel prediction : red

royak · 2022-09-28T20:04:22Z

royak
Sep 28, 2022
Author

https://github.com/LymphNodeQuantification/Monailabel-LNQ/blob/main/apps/radiology/lib/trainers/segmentation.py

This is the segmentation part of the code:

import torch
from lib.transforms.transforms import NormalizeLabelsInDatasetd
from monai.handlers import TensorBoardImageHandler, from_engine
from monai.inferers import SlidingWindowInferer
from monai.losses import DiceCELoss
from monai.transforms import (
    Activationsd,
    AsDiscreted,
    CropForegroundd,
    EnsureChannelFirstd,
    EnsureTyped,
    LoadImaged,
    RandCropByPosNegLabeld,
    RandFlipd,
    RandRotate90d,
    RandShiftIntensityd,
    ScaleIntensityRanged,
    SelectItemsd,
    Spacingd,
    SpatialPadd,
)

from monailabel.tasks.train.basic_train import BasicTrainTask, Context
from monailabel.tasks.train.utils import region_wise_metrics
from monai.optimizers import Novograd

logger = logging.getLogger(__name__)
class Segmentation(BasicTrainTask):
    def  __init__(
        self,
        model_dir,
        network,
        roi_size=(64, 64, 64),
        target_spacing=(1.0, 1.0, 1.0),
        num_samples=4,
        description="Train Segmentation model",
        **kwargs,```
        self._network = network
        self.roi_size = roi_size
        self.target_spacing = target_spacing
        self.num_samples = num_samples
        super().__init__(model_dir, description, **kwargs)

    def network(self, context: Context):
        return self._network

    def optimizer(self, context: Context):
     
        return torch.optim.AdamW(context.network.parameters(), lr=1e-5, weight_decay=1e-5)

    def loss_function(self, context: Context):
        return DiceCELoss(to_onehot_y=True, softmax=True)

    def lr_scheduler_handler(self, context: Context):
        return None

    def train_data_loader(self, context, num_workers=0, shuffle=False):
        return super().train_data_loader(context, num_workers, True)
def train_pre_transforms(self, context: Context):
        return [
            LoadImaged(keys=("image", "label"), reader="ITKReader"),
            EnsureChannelFirstd(keys=("image", "label")),
            NormalizeLabelsInDatasetd(keys="label", label_names=self._labels),  # Specially for missing labels
            Spacingd(keys=("image", "label"), pixdim=self.target_spacing, mode=("bilinear", "nearest")),
            CropForegroundd(keys=("image", "label"), source_key="image"),
            SpatialPadd(keys=("image", "label"), spatial_size=self.roi_size),
            ScaleIntensityRanged(keys="image", a_min=-400, a_max=1000, b_min=0.0, b_max=1.0, clip=True),
            RandCropByPosNegLabeld(
                keys=("image", "label"),
                label_key="label",
                spatial_size=self.roi_size,
                pos=1,
                neg=1,
                num_samples=self.num_samples,
                image_key="image",
                image_threshold=0,
            ),
            EnsureTyped(keys=("image", "label"), device=context.device),
            RandFlipd(keys=("image", "label"), spatial_axis=[0], prob=0.10),
            RandFlipd(keys=("image", "label"), spatial_axis=[1], prob=0.10),
            RandFlipd(keys=("image", "label"), spatial_axis=[2], prob=0.10),
            RandRotate90d(keys=("image", "label"), prob=0.10, max_k=3),
            RandShiftIntensityd(keys="image", offsets=0.1, prob=0.5),
            SelectItemsd(keys=("image", "label")),
        ]
def train_post_transforms(self, context: Context):
        return [
            EnsureTyped(keys="pred", device=context.device),
            Activationsd(keys="pred", softmax=True),
            AsDiscreted(
                keys=("pred", "label"),
                argmax=(True, False),
                to_onehot=len(self._labels) + 1,
            ),
        ]

    def val_pre_transforms(self, context: Context):
        return [
            LoadImaged(keys=("image", "label"), reader="ITKReader"),
            EnsureChannelFirstd(keys=("image", "label")),
            NormalizeLabelsInDatasetd(keys="label", label_names=self._labels),  # Specially for missing labels
            Spacingd(keys=("image", "label"), pixdim=self.target_spacing, mode=("bilinear", "nearest")),
            ScaleIntensityRanged(keys="image", a_min=-400, a_max=1000, b_min=0.0, b_max=1.0, clip=True),
            EnsureTyped(keys=("image", "label")),
            SelectItemsd(keys=("image", "label")),
        ]

    def val_inferer(self, context: Context):
        return SlidingWindowInferer(roi_size=self.roi_size, sw_batch_size=8)
def norm_labels(self):
        # This should be applied along with NormalizeLabelsInDatasetd transform
        new_label_nums = {}
        for idx, (key_label, _) in enumerate(self._labels.items(), start=1):
            if key_label != "background":
                new_label_nums[key_label] = idx
            if key_label == "background":
                new_label_nums["background"] = 0
        return new_label_nums

    def train_key_metric(self, context: Context):
        return region_wise_metrics(self.norm_labels(), self.TRAIN_KEY_METRIC, "train")

    def val_key_metric(self, context: Context):
        return region_wise_metrics(self.norm_labels(), self.VAL_KEY_METRIC, "val")

    def train_handlers(self, context: Context):
        handlers = super().train_handlers(context)
        if context.local_rank == 0:
            handlers.append(
                TensorBoardImageHandler(
                    log_dir=context.events_dir,
                    batch_transform=from_engine(["image", "label"]),
                    output_transform=from_engine(["pred"]),
                    interval=20,
                    epoch_level=True,
                )
            )
        return handlers

0 replies

tangy5 · 2022-09-28T23:27:38Z

tangy5
Sep 28, 2022
Maintainer

HI @royak, thanks for posting the question.
Looking at the logs, it seems there is something wrong with the pre-transformation RandCropByPosNegLabeld, the error probably means an index issue. I'm not sure, because the error is not interrupting training? You could try set number of samples to 1.
Regarding the visualization of not aligned labels, if this is a training sample, I doubt it's caused by CropForegroundd, the predicted labels are cropped. If so, you could try padding at post processing, or just try remove CropForegroundd for training transformations.

Regarding Dice metric, it probably related to these issues. Is just validation Dice not improving? Or both training and validation Dice are not improving?

If above suggestions do not help, feel free to show some training logs (e.g., dice, loss, ...), it would help to dig into more.
Thanks.

1 reply

royak Sep 29, 2022
Author

Hi Tangy,
Thank you for putting the time to have a look at this,
I tried to run with your suggestions but I got the following error or there was no improvement.

    raise ValueError("the data to aggregate must be PyTorch Tensor.")
ValueError: the data to aggregate must be PyTorch Tensor.
[2022-09-29 17:25:37,008] [22611] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:77) - Return code: 1```

royak · 2022-09-29T21:26:11Z

royak
Sep 29, 2022
Author

Do you mind having a look at this part of the log from the original code that I pasted above,
It might help us dig more into what my problem is as the error is not interrupting training.

    data = apply_transform(_transform, data)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/transform.py", line 118, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x7f79cf307cd0>
[2022-09-29 17:51:59,148] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 25/68 -- train_loss: 0.6035
[2022-09-29 17:51:59,375] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 26/68 -- train_loss: 0.4399
[2022-09-29 17:51:59,512] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 27/68 -- train_loss: 0.5228
Exception in thread Thread-271:
Traceback (most recent call last):
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/transform.py", line 91, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/transform.py", line 55, in _apply_transform
    return transform(parameters)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/croppad/dictionary.py", line 861, in __call__
    self.randomize(label, fg_indices, bg_indices, image)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/croppad/dictionary.py", line 852, in randomize
    self.cropper.randomize(label=label, fg_indices=fg_indices, bg_indices=bg_indices, image=image)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/croppad/array.py", line 1060, in randomize
    self.centers = generate_pos_neg_label_crop_centers(
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/utils.py", line 516, in generate_pos_neg_label_crop_centers
    random_int = rand_state.randint(len(indices_to_use))
  File "mtrand.pyx", line 748, in numpy.random.mtrand.RandomState.randint
  File "_bounded_integers.pyx", line 1247, in numpy.random._bounded_integers._rand_int64
ValueError: high <= 0
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/conda/envs/monailabel/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/conda/envs/monailabel/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/data/thread_buffer.py", line 48, in enqueue_values
    for src_val in self.src:
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/data/dataset.py", line 105, in __getitem__
    return self._transform(index)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/data/dataset.py", line 878, in _transform
    data = apply_transform(_transform, data)
  File "/opt/conda/envs/monailabel/lib/python3.9/site-packages/monai/transforms/transform.py", line 118, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x7f79cf307cd0>
[2022-09-29 17:51:59,582] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 28/68 -- train_loss: 0.5060
[2022-09-29 17:51:59,819] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 29/68 -- train_loss: 0.3136
[2022-09-29 17:51:59,959] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 30/68 -- train_loss: 0.4781
[2022-09-29 17:52:00,108] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 31/68 -- train_loss: 0.3439
[2022-09-29 17:52:00,239] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 32/68 -- train_loss: 0.4332
[2022-09-29 17:52:00,363] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 33/68 -- train_loss: 0.4240
[2022-09-29 17:52:00,487] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 34/68 -- train_loss: 0.4061
[2022-09-29 17:52:00,623] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 35/68 -- train_loss: 0.5092
[2022-09-29 17:52:00,766] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 36/68 -- train_loss: 0.4445
[2022-09-29 17:52:00,901] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 37/68 -- train_loss: 0.2634
[2022-09-29 17:52:01,027] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 38/68 -- train_loss: 0.4738
[2022-09-29 17:52:01,161] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 39/68 -- train_loss: 0.3784
[2022-09-29 17:52:01,248] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 40/68 -- train_loss: 0.4484
[2022-09-29 17:52:01,413] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:269) - Epoch: 50/50, Iter: 41/68 -- train_loss: 0.4487
[2022-09-29 17:52:01,431] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:254) - Got new best metric of train_dice: 0.499088317155838
[2022-09-29 17:52:01,432] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:199) - Epoch[50] Metrics -- train_dice: 0.4991 train_lymph node_dice: 0.4991
[2022-09-29 17:52:01,432] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:209) - Key metric: train_dice best value: 0.499088317155838 at epoch: 50
[2022-09-29 17:52:01,433] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 49 until 50 epochs
[2022-09-29 17:52:04,351] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:199) - Epoch[50] Metrics -- val_lymph node_dice: 0.4471 val_mean_dice: 0.4471
[2022-09-29 17:52:04,351] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:209) - Key metric: val_mean_dice best value: 0.4591790735721588 at epoch: 37
[2022-09-29 17:52:04,352] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:972) - Epoch[50] Complete. Time taken: 00:00:02.919
[2022-09-29 17:52:04,352] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:988) - Engine run complete. Time taken: 00:00:02.919
[2022-09-29 17:52:04,391] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:972) - Epoch[50] Complete. Time taken: 00:00:12.620
[2022-09-29 17:52:04,427] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:267) - Train completed, saved final checkpoint: /home/rkhajavibajestani_bwh_harvard_ed/monailabel/radiology/model/segmentation/train_01/model_final_iteration=3101.pt
[2022-09-29 17:52:04,427] [2780] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:988) - Engine run complete. Time taken: 00:09:10.965
[2022-09-29 17:52:04,440] [2780] [MainThread] [INFO] (monailabel.tasks.train.basic_train:504) - Running cleanup...
[2022-09-29 17:52:04,440] [2780] [MainThread] [INFO] (__main__:62) - Result: {"rank": 0, "current_epoch": 50, "current_iteration": 3101, "total_epochs": 50, "total_iterations": 68, "epoch": 50, "start_ts": 1664472885, "total_time": "0:17:18", "best_metric": 0.4591790735721588, "train": {"metrics": {"train_dice": 0.499088317155838, "train_lymph node_dice": 0.499088317155838}, "key_metric_name": "train_dice", "best_metric": 0.499088317155838, "best_metric_epoch": 50}, "eval": {"metrics": {"val_mean_dice": 0.4471223056316376, "val_lymph node_dice": 0.4471223056316376}, "key_metric_name": "val_mean_dice", "best_metric": 0.4591790735721588, "best_metric_epoch": 37}}
[2022-09-29 17:52:05,609] [32342] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:77) - Return code: 0

4 replies

tangy5 Sep 29, 2022
Maintainer

Thanks for the follow-ups. at least we can see the training and validation dice scores. I see the best training dice is got at Epoch 50, and validation best dice achieved at Epoch 37. The numbers seems making sense, I'm not sure about the task background, but the model is learning. I believe this is not an optimal performance as you mentioned above (not improving), can you set the training epochs longer, such as 200 or 300 instead of 50. And see whether the model can learn better.

Regarding the errors by RandCropByPosNegLabeld, seems the cropping center indices has an issue, but I can't tell details about where is wrong from the logs, I suggest to save a sample image after the transformation, and visualize it and check if it's making sense because it's not interrupting training.

Thanks you

royak Sep 30, 2022
Author

Thank you so much for the fast reply. I'll have a look at that too.

diazandr3s Sep 30, 2022
Maintainer

Hi @royak,

It seems there is a volume without label - which I guess is normal for this task.
That might be the reason the RandCropByPosNegLabeld is failing. My suggestion is to use RandSpatialCropd instead. Something like this: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/localization_spine.py#L83
BTW, lymph node segmentation isn't an easy task. My understanding is that they can be in any part of the CT. So once you have the model training well, consider using more images and tailoring the pre-transforms. Maybe random flipping isn't needed here. These arguments on the scale intensity transform were tailored for abdominal organ segmentation, they might not be the best for the lymph node segmentation, etc

Hope this helps,

royak Sep 30, 2022
Author

Hi @diazandr3s
Thank you so much. I have already changed the ScaleIntensityRanged in the code, but thank you for the tip, I will use the other suggestions like RandSpatialCropd and dig more into it.

pieper · 2022-09-30T23:02:47Z

pieper
Sep 30, 2022

Thanks so much for your help @diazandr3s and @tangy5 👍

@royak and I put together a repository to track development of this model https://github.com/LymphNodeQuantification/Monailabel-LNQ

It has everything you need to get the data and try training the model. We really appreciate any and all suggestions!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suggestions to improve the model? #1026

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Suggestions to improve the model? #1026

royak Sep 28, 2022

Replies: 4 comments · 5 replies

royak Sep 28, 2022 Author

tangy5 Sep 28, 2022 Maintainer

royak Sep 29, 2022 Author

royak Sep 29, 2022 Author

tangy5 Sep 29, 2022 Maintainer

royak Sep 30, 2022 Author

diazandr3s Sep 30, 2022 Maintainer

royak Sep 30, 2022 Author

pieper Sep 30, 2022

royak
Sep 28, 2022

Replies: 4 comments 5 replies

royak
Sep 28, 2022
Author

tangy5
Sep 28, 2022
Maintainer

royak Sep 29, 2022
Author

royak
Sep 29, 2022
Author

tangy5 Sep 29, 2022
Maintainer

royak Sep 30, 2022
Author

diazandr3s Sep 30, 2022
Maintainer

royak Sep 30, 2022
Author

pieper
Sep 30, 2022