numpy.linalg.LinAlgError and RuntimeError when trying to train the model from scratch #956

hourglassnam · 2022-08-30T12:33:17Z

hourglassnam
Aug 30, 2022

Dear all,
I am trying to train a model from scratch, but keep having trouble loading the segmentation images.
I made 10 segmented samples, Seg01 to Seg10, to try out the training process and placed the volumes(.nrrd) in the imagesTr folder and the segmentation(.nrrd) in the imagesTr/labels/final folder.
According to the log, the NibabelReader, PILReader, and NumpyReader cannot load the segmentation file, and I get numpy.linalg.LinAlgError and RuntimeError.
The volumes do not show up when I click the 'Next Sample' button. Instead, it gives me a warning message, "Unabled Samples/Images Not Found at server. Instead you can load your own image."
I could not figure this out by myself.
Could you guys help me out?

Thank you in advance, and below is my log.

monailabel start_server --app C:/Users/njy95/MONAILabel/sample-apps/histology --studies C:/Users/njy95/datasets/Task11_BR/imagesTr --conf models deepedit --conf preload true --port 8000 --host 127.0.0.1
# The histology is same thing as radiology app I just copied and renamed it so I can change it around

Using PYTHONPATH=C:\Users\njy95\MONAILabel;
[2022-08-30 16:05:00,758] [57148] [MainThread] [INFO] (__main__:269) - USING:: app = C:\Users\njy95\MONAILabel\sample-apps\histology
[2022-08-30 16:05:00,758] [57148] [MainThread] [INFO] (__main__:269) - USING:: studies = C:\Users\njy95\datasets\Task11_BR\imagesTr
[2022-08-30 16:05:00,762] [57148] [MainThread] [INFO] (__main__:269) - USING:: verbose = INFO
[2022-08-30 16:05:00,762] [57148] [MainThread] [INFO] (__main__:269) - USING:: conf = [['models', 'deepedit'], ['preload', 'true']]
[2022-08-30 16:05:00,777] [57148] [MainThread] [INFO] (__main__:269) - USING:: host = 127.0.0.1
[2022-08-30 16:05:00,803] [57148] [MainThread] [INFO] (__main__:269) - USING:: port = 8000
[2022-08-30 16:05:00,803] [57148] [MainThread] [INFO] (__main__:269) - USING:: uvicorn_app = monailabel.app:app
[2022-08-30 16:05:00,804] [57148] [MainThread] [INFO] (__main__:269) - USING:: ssl_keyfile = None
[2022-08-30 16:05:00,804] [57148] [MainThread] [INFO] (__main__:269) - USING:: ssl_certfile = None
[2022-08-30 16:05:00,804] [57148] [MainThread] [INFO] (__main__:269) - USING:: ssl_keyfile_password = None
[2022-08-30 16:05:00,809] [57148] [MainThread] [INFO] (__main__:269) - USING:: ssl_ca_certs = None
[2022-08-30 16:05:00,826] [57148] [MainThread] [INFO] (__main__:269) - USING:: workers = None
[2022-08-30 16:05:00,843] [57148] [MainThread] [INFO] (__main__:269) - USING:: limit_concurrency = None
[2022-08-30 16:05:00,845] [57148] [MainThread] [INFO] (__main__:269) - USING:: access_log = False
[2022-08-30 16:05:00,847] [57148] [MainThread] [INFO] (__main__:269) - USING:: log_config = None
[2022-08-30 16:05:00,849] [57148] [MainThread] [INFO] (__main__:269) - USING:: dryrun = False
[2022-08-30 16:05:00,849] [57148] [MainThread] [INFO] (__main__:269) - USING:: action = start_server
[2022-08-30 16:05:00,850] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_API_STR =
[2022-08-30 16:05:00,850] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
[2022-08-30 16:05:00,851] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
[2022-08-30 16:05:00,851] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_STUDIES =
[2022-08-30 16:05:00,852] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
[2022-08-30 16:05:00,857] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_AUTH_DB =
[2022-08-30 16:05:00,871] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
[2022-08-30 16:05:00,888] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
[2022-08-30 16:05:00,892] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
[2022-08-30 16:05:00,907] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
[2022-08-30 16:05:00,908] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
[2022-08-30 16:05:00,908] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
[2022-08-30 16:05:00,909] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
[2022-08-30 16:05:00,909] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
[2022-08-30 16:05:00,910] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX =
[2022-08-30 16:05:00,910] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX =
[2022-08-30 16:05:00,911] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX =
[2022-08-30 16:05:00,912] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
[2022-08-30 16:05:00,913] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
[2022-08-30 16:05:00,913] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '["*.nii.gz", "*.nii", "*.nrrd", "*.jpg", "*.png", "*.tif", "*.svs", "*.xml"]'
[2022-08-30 16:05:00,914] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
[2022-08-30 16:05:00,920] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
[2022-08-30 16:05:00,933] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
[2022-08-30 16:05:00,951] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
[2022-08-30 16:05:00,969] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
[2022-08-30 16:05:00,969] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
[2022-08-30 16:05:00,970] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
[2022-08-30 16:05:00,970] [57148] [MainThread] [INFO] (__main__:273) - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
[2022-08-30 16:05:00,971] [57148] [MainThread] [INFO] (__main__:274) -
[2022-08-30 16:05:01,590] [57148] [MainThread] [INFO] (uvicorn.error:75) - Started server process [57148]
[2022-08-30 16:05:01,590] [57148] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2022-08-30 16:05:01,594] [57148] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: C:\Users\njy95\MONAILabel\sample-apps\histology; studies: C:\Users\njy95\datasets\Task11_BR\imagesTr; conf: {'models': 'deepedit', 'preload': 'true'}
[2022-08-30 16:05:01,649] [57148] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-08-30 16:05:01,667] [57148] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-08-30 16:05:01,670] [57148] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-08-30 16:05:01,671] [57148] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-08-30 16:05:01,671] [57148] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-08-30 16:05:01,672] [57148] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2022-08-30 16:05:01,673] [57148] [MainThread] [INFO] (main:83) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2022-08-30 16:05:02,183] [57148] [MainThread] [INFO] (lib.configs.deepedit:142) - EPISTEMIC Enabled: 0; Samples: 5
[2022-08-30 16:05:02,184] [57148] [MainThread] [INFO] (lib.configs.deepedit:146) - TTA Enabled: 0; Samples: 5
[2022-08-30 16:05:02,204] [57148] [MainThread] [INFO] (main:87) - +++ Using Models: ['deepedit']
[2022-08-30 16:05:02,208] [57148] [MainThread] [INFO] (monailabel.interfaces.app:126) - Init Datastore for: C:\Users\njy95\datasets\Task11_BR\imagesTr
[2022-08-30 16:05:02,227] [57148] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-08-30 16:05:02,234] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg01 => Seg01.nrrd
[2022-08-30 16:05:02,248] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg02 => Seg02.nrrd
[2022-08-30 16:05:02,251] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg03 => Seg03.nrrd
[2022-08-30 16:05:02,261] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg04 => Seg04.nrrd
[2022-08-30 16:05:02,262] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg05 => Seg05.nrrd
[2022-08-30 16:05:02,266] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg06 => Seg06.nrrd
[2022-08-30 16:05:02,275] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg07 => Seg07.nrrd
[2022-08-30 16:05:02,276] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg08 => Seg08.nrrd
[2022-08-30 16:05:02,277] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg09 => Seg09.nrrd
[2022-08-30 16:05:02,283] [57148] [MainThread] [INFO] (monailabel.datastore.local:557) - Adding New Image: Seg10 => Seg10.nrrd
[2022-08-30 16:05:02,294] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg01 => Seg01.nrrd
[2022-08-30 16:05:02,300] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg02 => Seg02.nrrd
[2022-08-30 16:05:02,311] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg03 => Seg03.nrrd
[2022-08-30 16:05:02,317] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg04 => Seg04.nrrd
[2022-08-30 16:05:02,326] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg05 => Seg05.nrrd
[2022-08-30 16:05:02,331] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg06 => Seg06.nrrd
[2022-08-30 16:05:02,340] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg07 => Seg07.nrrd
[2022-08-30 16:05:02,345] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg08 => Seg08.nrrd
[2022-08-30 16:05:02,358] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg09 => Seg09.nrrd
[2022-08-30 16:05:02,362] [57148] [MainThread] [INFO] (monailabel.datastore.local:587) - Adding New Label: final => Seg10 => Seg10.nrrd
[2022-08-30 16:05:02,377] [57148] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 20
[2022-08-30 16:05:02,380] [57148] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-08-30 16:05:02,391] [57148] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:120) - Preload Network for device: cuda:0
[2022-08-30 16:05:02,392] [57148] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:352) - Infer model path: None
[2022-08-30 16:05:03,630] [57148] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:120) - Preload Network for device: cuda:0
[2022-08-30 16:05:03,631] [57148] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:352) - Infer model path: None
[2022-08-30 16:05:03,711] [57148] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x000001B6E80831F0>
[2022-08-30 16:05:03,712] [57148] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x000001B6E81F8190>
[2022-08-30 16:05:03,723] [57148] [MainThread] [INFO] (main:161) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x000001B6E8275400>
[2022-08-30 16:05:03,724] [57148] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\njy95\.cache\monailabel\sessions
[2022-08-30 16:05:03,735] [57148] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-08-30 16:05:03,736] [57148] [MainThread] [INFO] (monailabel.interfaces.app:431) - App Init - completed
[2022-08-30 16:05:03,737] [timeloop] [INFO] Starting Timeloop..
[2022-08-30 16:05:03,737] [57148] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-08-30 16:05:03,738] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x000001B6E826D040>
[2022-08-30 16:05:03,738] [57148] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x000001B6E826D040>
[2022-08-30 16:05:03,739] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-08-30 16:05:03,739] [57148] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-08-30 16:05:03,752] [57148] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2022-08-30 16:05:03,752] [57148] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)


[2022-08-30 16:06:49,761] [57148] [MainThread] [INFO] (monailabel.endpoints.activelearning:43) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2022-08-30 16:08:45,869] [57148] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Train request: {'model': 'deepedit', 'name': 'train_01', 'pretrained': 0, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'CacheDataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz'}
[2022-08-30 16:08:45,871] [57148] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:58) - COMMAND:: C:\ProgramData\Anaconda3\python.exe -m monailabel.interfaces.utils.app -m train -r {"model":"deepedit","name":"train_01","pretrained":0,"device":"cuda","max_epochs":50,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"CacheDataset","dataloader":"ThreadDataLoader","client_id":"user-xyz"}
[2022-08-30 16:08:46,136] [19348] [MainThread] [INFO] (__main__:38) - Initializing App from: C:\Users\njy95\MONAILabel\sample-apps\histology; studies: C:\Users\njy95\datasets\Task11_BR\imagesTr; conf: {'models': 'deepedit', 'preload': 'true'}
[2022-08-30 16:08:48,136] [19348] [MainThread] [INFO] (numexpr.utils:147) - Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
[2022-08-30 16:08:48,136] [19348] [MainThread] [INFO] (numexpr.utils:159) - NumExpr defaulting to 8 threads.
[2022-08-30 16:08:50,802] [19348] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-08-30 16:08:50,822] [19348] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-08-30 16:08:50,823] [19348] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-08-30 16:08:50,824] [19348] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-08-30 16:08:50,824] [19348] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-08-30 16:08:50,825] [19348] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2022-08-30 16:08:50,825] [19348] [MainThread] [INFO] (main:83) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2022-08-30 16:08:51,449] [19348] [MainThread] [INFO] (lib.configs.deepedit:142) - EPISTEMIC Enabled: 0; Samples: 5
[2022-08-30 16:08:51,449] [19348] [MainThread] [INFO] (lib.configs.deepedit:146) - TTA Enabled: 0; Samples: 5
[2022-08-30 16:08:51,449] [19348] [MainThread] [INFO] (main:87) - +++ Using Models: ['deepedit']
[2022-08-30 16:08:51,449] [19348] [MainThread] [INFO] (monailabel.interfaces.app:126) - Init Datastore for: C:\Users\njy95\datasets\Task11_BR\imagesTr
[2022-08-30 16:08:51,450] [19348] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-08-30 16:08:51,468] [19348] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2022-08-30 16:08:51,468] [19348] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:120) - Preload Network for device: cuda:0
[2022-08-30 16:08:51,468] [19348] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:352) - Infer model path: None
[2022-08-30 16:08:52,847] [19348] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:120) - Preload Network for device: cuda:0
[2022-08-30 16:08:52,847] [19348] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:352) - Infer model path: None
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x00000220763AE280>
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x00000220764DA250>
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (main:161) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x000002207FF856A0>
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\njy95\.cache\monailabel\sessions
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:353) - Train Request (input): {'model': 'deepedit', 'name': 'train_01', 'pretrained': 0, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'CacheDataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz', 'local_rank': 0}
[2022-08-30 16:08:52,924] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:363) - CUDA_VISIBLE_DEVICES: None
[2022-08-30 16:08:52,927] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:368) - Distributed/Multi GPU is limited
[2022-08-30 16:08:52,927] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:381) - Distributed Training = FALSE
[2022-08-30 16:08:52,927] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:408) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': 0, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'CacheDataset', 'dataloader': 'ThreadDataLoader', 'model': 'deepedit', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20220830_1608'}
[2022-08-30 16:08:52,927] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:504) - 0 - Using Device: cuda; IDX: None
[2022-08-30 16:08:52,927] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:331) - Total Records for Training: 8
[2022-08-30 16:08:52,927] [19348] [MainThread] [INFO] (monailabel.tasks.train.basic_train:332) - Total Records for Validation: 2
Loading dataset:   0%|          | 0/2 [00:00<?, ?it/s][2022-08-30 16:09:24,661] [19348] [Thread-4] [INFO] (LoadImage:221) - ITKReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg09.nrrd',).
[2022-08-30 16:09:24,662] [19348] [Thread-4] [INFO] (LoadImage:221) - NibabelReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg09.nrrd',).
[2022-08-30 16:09:24,734] [19348] [Thread-4] [INFO] (LoadImage:221) - PILReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg09.nrrd',).
[2022-08-30 16:09:24,744] [19348] [Thread-4] [INFO] (LoadImage:221) - NumpyReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg09.nrrd',).
Loading dataset:   0%|          | 0/2 [00:35<?, ?it/s]
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\transforms\transform.py", line 89, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\transforms\transform.py", line 53, in _apply_transform
    return transform(parameters)
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\transforms\spatial\dictionary.py", line 662, in __call__
    d[key], old_affine, new_affine = self.ornt_transform(d[key], affine=meta_data["affine"])
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\transforms\spatial\array.py", line 543, in __call__
    src = nib.io_orientation(affine_)
  File "C:\ProgramData\Anaconda3\lib\site-packages\nibabel\orientations.py", line 65, in io_orientation
    P, S, Qs = npl.svd(RS, full_matrices=False)
  File "<__array_function__ internals>", line 180, in svd
  File "C:\Users\njy95\AppData\Roaming\Python\Python39\site-packages\numpy\linalg\linalg.py", line 1648, in svd
    u, s, vh = gufunc(a, signature=signature, extobj=extobj)
  File "C:\Users\njy95\AppData\Roaming\Python\Python39\site-packages\numpy\linalg\linalg.py", line 97, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\njy95\MONAILabel\monailabel\interfaces\utils\app.py", line 132, in <module>
    run_main()
  File "C:\Users\njy95\MONAILabel\monailabel\interfaces\utils\app.py", line 117, in run_main
    result = a.train(request)
  File "C:\Users\njy95\MONAILabel\monailabel\interfaces\app.py", line 382, in train
    result = task(request, self.datastore())
  File "C:\Users\njy95\MONAILabel\monailabel\tasks\train\basic_train.py", line 382, in __call__
    res = self.train(0, world_size, req, datalist)
  File "C:\Users\njy95\MONAILabel\monailabel\tasks\train\basic_train.py", line 427, in train
    context.evaluator = self._create_evaluator(context)
  File "C:\Users\njy95\MONAILabel\monailabel\tasks\train\basic_train.py", line 540, in _create_evaluator
    val_data_loader=self.val_data_loader(context),
  File "C:\Users\njy95\MONAILabel\monailabel\tasks\train\basic_train.py", line 274, in val_data_loader
    dataset, datalist = self._dataset(context, context.val_datalist)
  File "C:\Users\njy95\MONAILabel\monailabel\tasks\train\basic_train.py", line 199, in _dataset
    CacheDataset(datalist, transforms)
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\data\dataset.py", line 723, in __init__
    self.set_data(data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\data\dataset.py", line 748, in set_data
    self._cache = _compute_cache()
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\data\dataset.py", line 737, in _compute_cache
    return self._fill_cache()
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\data\dataset.py", line 757, in _fill_cache
    return list(
  File "C:\ProgramData\Anaconda3\lib\site-packages\tqdm\std.py", line 1195, in __iter__
    for obj in iterable:
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 870, in next
[2022-08-30 16:09:30,241] [19348] [Thread-4] [INFO] (LoadImage:221) - ITKReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg10.nrrd',).
[2022-08-30 16:10:23,569] [19348] [Thread-4] [INFO] (LoadImage:221) - NibabelReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg10.nrrd',).
    raise value
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 125, in worker
[2022-08-30 16:10:23,570] [19348] [Thread-4] [INFO] (LoadImage:221) - PILReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg10.nrrd',).
    result = (True, func(*args, **kwds))
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\data\dataset.py", line 777, in _load_cache_item
[2022-08-30 16:10:23,570] [19348] [Thread-4] [INFO] (LoadImage:221) - NumpyReader: unable to load ('C:\\Users\\njy95\\datasets\\Task11_BR\\imagesTr\\labels\\final\\Seg10.nrrd',).
    item = apply_transform(_xform, item)
  File "C:\ProgramData\Anaconda3\lib\site-packages\monai\transforms\transform.py", line 113, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.spatial.dictionary.Orientationd object at 0x00000220C22CBF40>
[2022-08-30 16:10:25,253] [57148] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:76) - Return code: 1

diazandr3s · 2022-08-30T14:30:02Z

diazandr3s
Aug 30, 2022
Maintainer

Hi @hourglassnam,

Thanks for opening this discussion.

It seems there are inf or nans values in the segmentations you've created. Can you please make sure that's not the case?

https://stackoverflow.com/questions/21827594/raise-linalgerrorsvd-did-not-converge-linalgerror-svd-did-not-converge-in-m

Hope this helps,

9 replies

diazandr3s Sep 12, 2022
Maintainer

Hi @hourglassnam,

I'm glad to hear you solved this issue.
An 8GB GPU is small for the default image size of the DeepEdit (128x128x128). If you can't access a bigger GPU, I'd suggest you reduce the training/inference image size here: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/configs/deepedit.py#L80

Perhaps 128x128x64 can fit the GPU you're using.

I hope this helps,

hourglassnam Sep 15, 2022
Author

Hello @diazandr3s,

Thank you always for your tips!

Sadly, I tried (128, 128, 64) and (96,96,96) but could not fix the error.
So I was thinking of using the Google Cloud Platform for running the model with a larger GPU.
I created a window instance with Tesla T4(16GiB) and am downloading the Cuda, PyTorch, etc.
Also, I am desperately trying to make the monailabel server on the GCP VM connected to the Monailabel module on my local computer.
Is there any to make this happen?
Is this even possible?
I am just giving it a shot but worried that this might not work.
I would be grateful to hear how other people approach this.

I truly appreciate all your advice and hope you have a great day.

diazandr3s Sep 15, 2022
Maintainer

Hi @hourglassnam,

Yes, you should be able to install MONAI Label on a GCP instance and connect to it from your local PC.
Once MONAi Label is installed, check what's the public IP of that machine and make sure port 8000 is open.

Hope that helps,

SachidanandAlle Sep 15, 2022
Maintainer

You can run infer on 8GB box.. by disabling few transform in pre transforms section.. Ensurexxx ToTensor etc.. also disable cache transform

hourglassnam Sep 18, 2022
Author

Hello @diazandr3s, thank you for your reply! I am on my way to setting up the GPU environment.
I could not see my GPU on the vnc of the window VM I created, so instead, I made a Ubuntu VM.
Would it be okay to use the server on Ubuntu and run the program on the Window?
Would it be a big problem?
Do they have to match each other?

pieper · 2022-09-15T17:06:55Z

pieper
Sep 15, 2022

As a follow up to @diazandr3s 's advice, you could also tunnel port 8000 via ssh so you don't expose your MONAI Label server to the internet. You can use a command like this:

gcloud compute ssh ${VMNAME} -- -L 8000:localhost:8000

1 reply

hourglassnam Sep 18, 2022
Author

Thank you for saving me:)
I was actually about to expose my server naively!
Using ssh seems so much safer, and I am glad to know about it!

SachidanandAlle · 2022-09-16T19:09:59Z

SachidanandAlle
Sep 16, 2022
Maintainer

I have no issues running on simple image (392,392,210) deepedit infer on my windows laptop (with 6 GB GPU)
(This is with no changes.. whatever is there on main branch)

1 reply

hourglassnam Sep 18, 2022
Author

Hello, @SachidanandAlle!
Thank you for letting me know! From what you have shown, turning off the transforms saves so much memory!
I will definitely check out and see if the accuracy still works for me!
Thank you for your suggestion!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

numpy.linalg.LinAlgError and RuntimeError when trying to train the model from scratch #956

{{title}}

Replies: 3 comments 11 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

numpy.linalg.LinAlgError and RuntimeError when trying to train the model from scratch #956

hourglassnam Aug 30, 2022

Replies: 3 comments · 11 replies

diazandr3s Aug 30, 2022 Maintainer

diazandr3s Sep 12, 2022 Maintainer

hourglassnam Sep 15, 2022 Author

diazandr3s Sep 15, 2022 Maintainer

SachidanandAlle Sep 15, 2022 Maintainer

hourglassnam Sep 18, 2022 Author

pieper Sep 15, 2022

hourglassnam Sep 18, 2022 Author

SachidanandAlle Sep 16, 2022 Maintainer

hourglassnam Sep 18, 2022 Author

hourglassnam
Aug 30, 2022

Replies: 3 comments 11 replies

diazandr3s
Aug 30, 2022
Maintainer

diazandr3s Sep 12, 2022
Maintainer

hourglassnam Sep 15, 2022
Author

diazandr3s Sep 15, 2022
Maintainer

SachidanandAlle Sep 15, 2022
Maintainer

hourglassnam Sep 18, 2022
Author

pieper
Sep 15, 2022

hourglassnam Sep 18, 2022
Author

SachidanandAlle
Sep 16, 2022
Maintainer

hourglassnam Sep 18, 2022
Author