GPU memory requirement difference depending on which environment I am on #1029

hourglassnam · 2022-09-30T04:08:50Z

hourglassnam
Sep 30, 2022

Hello,
I have a question!
I use to get OOM error on my Window10 computer with RTX3070(8 GB), i7-11800H so I created GCP Ubuntu(20.04) with Tesla T4(16GB) to run my monailabel server on.
I thought 16GB be enough since the OOM message used to say allocate 1.20GiB

CUDA out of memory. Tried to allocate 1.20 GiB (GPU 0; 8.00 GiB total capacity; 5.28 GiB already allocated; 735.12 MiB free; 5.33 GiB reserved in total by PyTorch)

However, I still get OOM message and now it tells me to allocate 20.04 GiB!

RuntimeError: CUDA out of memory. Tried to allocate 20.04 GiB (GPU 0; 14.76 GiB total capacity; 6.42 GiB already allocated; 7.07 GiB free; 6.44 GiB reserved in total by PyTorch)"

I guess I can solve this by adding additional GPU on my instance but I just want to understand this situation.
I am using same sample and model so I do not see what can cause this requirement difference.
Can anyone please tell me why this is happening?

Answered by SachidanandAlle

Sep 30, 2022

You are using very large image image: (553, 449, 677)
POST - Transform (Activationsd) fails as it tries to do it in GPU (for faster performance/user experience)

You can disable to run it on GPU by commenting EnsureTyped in your post transform. This will be slow but you will get the result..

Or you can try to use smaller image first and see all is ok e2e with the workflow (which should be)

View full answer

SachidanandAlle · 2022-09-30T04:12:31Z

SachidanandAlle
Sep 30, 2022
Maintainer

Which operation you are trying? inference? training?
Which example model? (also load single model --conf models segmentation_spleen instead of all.. and possibly you can set --conf preload false)
https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/radiology#segmentation-spleen

If you have too big image, try something small.. you can also download sample dataset and try some of them (Ex Spleen)

I can run inference for segmentation_spleen model on my windows (6GB GPU)
May be try with smaller ones to make sure all is good.. and depending on the input/image and model you can go for higher ones..

2 replies

SachidanandAlle Sep 30, 2022
Maintainer

And it shall be more helpful if you share the logs (including the command which you are using to start the server)

hourglassnam Sep 30, 2022
Author

Thank you so much for your fast reply!
I am trying deepedit Auto Segmentation.

Here is how I started my server

hourglassnam@instance-name:~$ monailabel start_server --app /home/hourglassnam/.local/monailabel/sample-apps/histology_morph --studies /home/hourglassnam/Documents/Task14_Unsep/imagesTr --conf models deepedit --port 8000
Using PYTHONPATH=/home/hourglassnam:
2022-09-30 03:55:21,369 - USING:: version = False
2022-09-30 03:55:21,369 - USING:: app = /home/hourglassnam/.local/monailabel/sample-apps/histology_morph
2022-09-30 03:55:21,369 - USING:: studies = /home/hourglassnam/Documents/Task14_Unsep/imagesTr
2022-09-30 03:55:21,369 - USING:: verbose = INFO
2022-09-30 03:55:21,369 - USING:: conf = [['models', 'deepedit']]
2022-09-30 03:55:21,369 - USING:: host = 0.0.0.0
2022-09-30 03:55:21,369 - USING:: port = 8000
2022-09-30 03:55:21,369 - USING:: uvicorn_app = monailabel.app:app
2022-09-30 03:55:21,369 - USING:: ssl_keyfile = None
2022-09-30 03:55:21,370 - USING:: ssl_certfile = None
2022-09-30 03:55:21,370 - USING:: ssl_keyfile_password = None
2022-09-30 03:55:21,370 - USING:: ssl_ca_certs = None
2022-09-30 03:55:21,370 - USING:: workers = None
2022-09-30 03:55:21,370 - USING:: limit_concurrency = None
2022-09-30 03:55:21,370 - USING:: access_log = False
2022-09-30 03:55:21,370 - USING:: log_config = None
2022-09-30 03:55:21,370 - USING:: dryrun = False
2022-09-30 03:55:21,370 - USING:: action = start_server
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_API_STR = 
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_APP_DIR = 
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_STUDIES = 
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB = 
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2022-09-30 03:55:21,370 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = 
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2022-09-30 03:55:21,371 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '["*.nii.gz", "*.nii", "*.nrrd", "*.jpg", "*.png", "*.tif", "*.svs", "*.xml"]'
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH = 
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2022-09-30 03:55:21,372 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2022-09-30 03:55:21,372 - 
Allow Origins: ['*']
[2022-09-30 03:55:21,781] [147356] [MainThread] [INFO] (uvicorn.error:75) - Started server process [147356]
[2022-09-30 03:55:21,782] [147356] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2022-09-30 03:55:21,782] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /home/hourglassnam/.local/monailabel/sample-apps/histology_morph; studies: /home/hourglassnam/Documents/Task14_Unsep/imagesTr; conf: {'models': 'deepedit'}
[2022-09-30 03:55:21,855] [147356] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-09-30 03:55:21,862] [147356] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-09-30 03:55:21,862] [147356] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-09-30 03:55:21,863] [147356] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-09-30 03:55:21,863] [147356] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-09-30 03:55:21,863] [147356] [MainThread] [INFO] (main:86) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2022-09-30 03:55:23,029] [147356] [MainThread] [INFO] (lib.configs.deepedit:141) - EPISTEMIC Enabled: 0; Samples: 5
[2022-09-30 03:55:23,029] [147356] [MainThread] [INFO] (lib.configs.deepedit:145) - TTA Enabled: 0; Samples: 5
[2022-09-30 03:55:23,030] [147356] [MainThread] [INFO] (main:90) - +++ Using Models: ['deepedit']
[2022-09-30 03:55:23,030] [147356] [MainThread] [INFO] (monailabel.interfaces.app:128) - Init Datastore for: /home/hourglassnam/Documents/Task14_Unsep/imagesTr
[2022-09-30 03:55:23,030] [147356] [MainThread] [INFO] (monailabel.datastore.local:126) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-09-30 03:55:23,054] [147356] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2022-09-30 03:55:23,055] [147356] [MainThread] [INFO] (monailabel.datastore.local:146) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-09-30 03:55:23,056] [147356] [MainThread] [INFO] (main:116) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x7f23740b4eb0>
[2022-09-30 03:55:23,056] [147356] [MainThread] [INFO] (main:116) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x7f237c01f4c0>
[2022-09-30 03:55:23,056] [147356] [MainThread] [INFO] (main:172) - {'deepedit': <lib.infers.deepedit.DeepEdit object at 0x7f23740b4eb0>, 'deepedit_seg': <lib.infers.deepedit.DeepEdit object at 0x7f237c01f4c0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7f2352c25910>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7f2352c259a0>}
[2022-09-30 03:55:23,056] [147356] [MainThread] [INFO] (main:185) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x7f2352c25a00>
[2022-09-30 03:55:23,056] [147356] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/hourglassnam/.cache/monailabel/sessions
[2022-09-30 03:55:23,057] [147356] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-09-30 03:55:23,057] [147356] [MainThread] [INFO] (monailabel.interfaces.app:465) - App Init - completed
[2022-09-30 03:55:23,057] [timeloop] [INFO] Starting Timeloop..
[2022-09-30 03:55:23,057] [147356] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-09-30 03:55:23,057] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7f23740f0dc0>
[2022-09-30 03:55:23,057] [147356] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7f23740f0dc0>
[2022-09-30 03:55:23,057] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-09-30 03:55:23,057] [147356] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-09-30 03:55:23,057] [147356] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2022-09-30 03:55:23,058] [147356] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

I tried --conf preload false as well, but the result was the same.
And Here is my error log

[2022-09-30 03:57:57,805] [147356] [MainThread] [INFO] (monailabel.endpoints.session:68) - Uncompress: False; Expiry: 0
[2022-09-30 03:57:57,805] [147356] [MainThread] [INFO] (monailabel.endpoints.session:69) - Request Files: [<starlette.datastructures.UploadFile object at 0x7f2350b2d160>]
[2022-09-30 03:57:57,857] [147356] [MainThread] [INFO] (monailabel.endpoints.session:83) - 1 => <starlette.datastructures.UploadFile object at 0x7f2350b2d160> => /tmp/tmpnzas3_y7/tmpc7v6mkci.nii.gz
[2022-09-30 03:57:57,858] [147356] [MainThread] [INFO] (monailabel.utils.sessions:154) - ++ Time consumed to add session 11df3ca2-4074-11ed-af75-bb7884a64484: 0.0005438327789306641
[2022-09-30 03:57:57,858] [147356] [MainThread] [INFO] (monailabel.endpoints.session:95) - Session ID: 11df3ca2-4074-11ed-af75-bb7884a64484; Info: {"name": "11df3ca2-4074-11ed-af75-bb7884a64484", "path": "/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484", "image": "/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz", "meta": {}, "create_ts": 1664510277, "last_access_ts": 1664510277, "expiry": 3600}
[2022-09-30 03:57:58,706] [147356] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'deepedit_seg', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'session': {'name': '11df3ca2-4074-11ed-af75-bb7884a64484', 'path': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'meta': {}, 'create_ts': 1664510277, 'last_access_ts': 1664510278, 'expiry': 3600}}
[2022-09-30 03:57:58,706] [147356] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:279) - Infer Request (final): {'model': 'deepedit_seg', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'session': {'name': '11df3ca2-4074-11ed-af75-bb7884a64484', 'path': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'meta': {}, 'create_ts': 1664510277, 'last_access_ts': 1664510278, 'expiry': 3600}, 'description': 'A DeepEdit model for volumetric (3D) segmentation over 3D Images', 'save_label': False, 'device': 'cuda'}
[2022-09-30 03:57:58,708] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Run Transform(s)
[2022-09-30 03:57:58,708] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - PRE - Input Keys: ['model', 'image', 'result_extension', 'result_dtype', 'client_id', 'session', 'description', 'save_label', 'device', 'image_path']
[2022-09-30 03:58:02,131] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (LoadImaged): Time: 3.4217; image: (553, 449, 677)(torch.float32)
[2022-09-30 03:58:02,132] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureChannelFirstd): Time: 0.0006; image: (1, 553, 449, 677)(torch.float32)
[2022-09-30 03:58:04,496] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Orientationd): Time: 2.3639; image: (1, 553, 449, 677)(torch.float32)
[2022-09-30 03:58:05,430] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (ScaleIntensityRanged): Time: 0.9342; image: (1, 553, 449, 677)(torch.float32)
[2022-09-30 03:58:05,913] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Resized): Time: 0.482; image: (1, 128, 128, 128)(torch.float32)
[2022-09-30 03:58:05,953] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (DiscardAddGuidanced): Time: 0.0394; image: (5, 128, 128, 128)(torch.float32)
[2022-09-30 03:58:10,372] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureTyped): Time: 4.419; image: (5, 128, 128, 128)(torch.float32)
[2022-09-30 03:58:10,372] [147356] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:447) - Inferer:: cuda => SimpleInferer => {}
[2022-09-30 03:58:10,372] [147356] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:396) - Infer model path: /home/hourglassnam/.local/monailabel/sample-apps/histology_morph/model/deepedit_dynunet.pt
[2022-09-30 03:58:11,274] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - INV - Run Transform(s)
[2022-09-30 03:58:11,274] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - INV - Input Keys: ['model', 'image', 'result_extension', 'result_dtype', 'client_id', 'session', 'description', 'save_label', 'device', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2022-09-30 03:58:11,281] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - INV - Transform (Resized): Time: 0.006; image: (4, 553, 449, 677)(torch.float32); pred: (4, 128, 128, 128)(torch.float32)
[2022-09-30 03:58:11,383] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - INV - Transform (Orientationd): Time: 0.1015; image: (4, 553, 449, 677)(torch.float32); pred: (4, 128, 128, 128)(torch.float32)
[2022-09-30 03:58:11,384] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - POST - Run Transform(s)
[2022-09-30 03:58:11,384] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - POST - Input Keys: ['model', 'image', 'result_extension', 'result_dtype', 'client_id', 'session', 'description', 'save_label', 'device', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2022-09-30 03:58:11,384] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (EnsureTyped): Time: 0.0001; image: (5, 128, 128, 128)(torch.float32); pred: (4, 553, 449, 677)(torch.float32)
[2022-09-30 03:58:11,385] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Activationsd): Time: 0.0004; image: (5, 128, 128, 128)(torch.float32); pred: (4, 553, 449, 677)(torch.float32)
[2022-09-30 03:58:11,464] [147356] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 179, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 161, in run_inference
    result = instance.infer(request)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 294, in infer
    result_file_name, result_json = task(request)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 300, in __call__
    data = self.run_post_transforms(data, self.post_transforms(data))
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 389, in run_post_transforms
    return run_transforms(data, transforms, log_prefix="POST")
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/utils/transform.py", line 93, in run_transforms
    data = t(data)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 189, in __call__
    d[key] = self.converter(d[key], argmax, to_onehot, threshold, rounding)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monai/transforms/post/array.py", line 203, in __call__
    img_t = torch.argmax(img_t, dim=0, keepdim=True)
RuntimeError: CUDA out of memory. Tried to allocate 20.04 GiB (GPU 0; 14.76 GiB total capacity; 6.42 GiB already allocated; 7.07 GiB free; 6.44 GiB reserved in total by PyTorch)
[2022-09-30 04:16:19,141] [147356] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'deepedit_seg', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'session': {'name': '11df3ca2-4074-11ed-af75-bb7884a64484', 'path': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'meta': {}, 'create_ts': 1664510277, 'last_access_ts': 1664511379, 'expiry': 3600}}
[2022-09-30 04:16:19,142] [147356] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:279) - Infer Request (final): {'model': 'deepedit_seg', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'session': {'name': '11df3ca2-4074-11ed-af75-bb7884a64484', 'path': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484', 'image': '/home/hourglassnam/.cache/monailabel/sessions/11df3ca2-4074-11ed-af75-bb7884a64484/tmpc7v6mkci.nii.gz', 'meta': {}, 'create_ts': 1664510277, 'last_access_ts': 1664511379, 'expiry': 3600}, 'description': 'A DeepEdit model for volumetric (3D) segmentation over 3D Images', 'save_label': False, 'device': 'cuda'}
[2022-09-30 04:16:19,143] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Run Transform(s)
[2022-09-30 04:16:19,143] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - PRE - Input Keys: ['model', 'image', 'result_extension', 'result_dtype', 'client_id', 'session', 'description', 'save_label', 'device', 'image_path']
[2022-09-30 04:16:21,041] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (LoadImaged): Time: 1.8974; image: (553, 449, 677)(torch.float32)
[2022-09-30 04:16:21,041] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureChannelFirstd): Time: 0.0002; image: (1, 553, 449, 677)(torch.float32)
[2022-09-30 04:16:23,417] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Orientationd): Time: 2.3752; image: (1, 553, 449, 677)(torch.float32)
[2022-09-30 04:16:24,340] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (ScaleIntensityRanged): Time: 0.9223; image: (1, 553, 449, 677)(torch.float32)
[2022-09-30 04:16:24,821] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Resized): Time: 0.4807; image: (1, 128, 128, 128)(torch.float32)
[2022-09-30 04:16:24,860] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (DiscardAddGuidanced): Time: 0.0389; image: (5, 128, 128, 128)(torch.float32)
[2022-09-30 04:16:24,871] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureTyped): Time: 0.0105; image: (5, 128, 128, 128)(torch.float32)
[2022-09-30 04:16:24,871] [147356] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:447) - Inferer:: cuda => SimpleInferer => {}
[2022-09-30 04:16:24,871] [147356] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:396) - Infer model path: /home/hourglassnam/.local/monailabel/sample-apps/histology_morph/model/deepedit_dynunet.pt
[2022-09-30 04:16:25,286] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - INV - Run Transform(s)
[2022-09-30 04:16:25,287] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - INV - Input Keys: ['model', 'image', 'result_extension', 'result_dtype', 'client_id', 'session', 'description', 'save_label', 'device', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2022-09-30 04:16:25,293] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - INV - Transform (Resized): Time: 0.0062; image: (4, 553, 449, 677)(torch.float32); pred: (4, 128, 128, 128)(torch.float32)
[2022-09-30 04:16:25,403] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - INV - Transform (Orientationd): Time: 0.109; image: (4, 553, 449, 677)(torch.float32); pred: (4, 128, 128, 128)(torch.float32)
[2022-09-30 04:16:25,403] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - POST - Run Transform(s)
[2022-09-30 04:16:25,403] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - POST - Input Keys: ['model', 'image', 'result_extension', 'result_dtype', 'client_id', 'session', 'description', 'save_label', 'device', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2022-09-30 04:16:25,404] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (EnsureTyped): Time: 0.0001; image: (5, 128, 128, 128)(torch.float32); pred: (4, 553, 449, 677)(torch.float32)
[2022-09-30 04:16:25,404] [147356] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Activationsd): Time: 0.0003; image: (5, 128, 128, 128)(torch.float32); pred: (4, 553, 449, 677)(torch.float32)
[2022-09-30 04:16:25,479] [147356] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 179, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 161, in run_inference
    result = instance.infer(request)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 294, in infer
    result_file_name, result_json = task(request)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 300, in __call__
    data = self.run_post_transforms(data, self.post_transforms(data))
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 389, in run_post_transforms
    return run_transforms(data, transforms, log_prefix="POST")
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monailabel/interfaces/utils/transform.py", line 93, in run_transforms
    data = t(data)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 189, in __call__
    d[key] = self.converter(d[key], argmax, to_onehot, threshold, rounding)
  File "/home/hourglassnam/.local/lib/python3.8/site-packages/monai/transforms/post/array.py", line 203, in __call__
    img_t = torch.argmax(img_t, dim=0, keepdim=True)
RuntimeError: CUDA out of memory. Tried to allocate 20.04 GiB (GPU 0; 14.76 GiB total capacity; 11.47 GiB already allocated; 821.75 MiB free; 12.71 GiB reserved in total by PyTorch)

SachidanandAlle · 2022-09-30T04:51:04Z

SachidanandAlle
Sep 30, 2022
Maintainer

You are using very large image image: (553, 449, 677)
POST - Transform (Activationsd) fails as it tries to do it in GPU (for faster performance/user experience)

You can disable to run it on GPU by commenting EnsureTyped in your post transform. This will be slow but you will get the result..

Or you can try to use smaller image first and see all is ok e2e with the workflow (which should be)

1 reply

hourglassnam Oct 3, 2022
Author

Dear @SachidanandAlle
Thank you so much for pointing out the image dimension size!
Finally, I was able to run the model by resampling the dimension size to (553, 449, 338)!
I am so grateful for all of the help from this community!!

I am still curious about why the GPU requirement differs depending on the machine(Ubuntu, Window10).
Does it have to do with the GPU I have?

tangy5 · 2022-09-30T05:12:50Z

tangy5
Sep 30, 2022
Maintainer

In addition, as another option or try. If you using SlidingWindowInference, you could set sw_device to CPU to avoid saving large volume data or large number of output channels prediction in GPU memory (If CPU mem allows).

1 reply

diazandr3s Sep 30, 2022
Maintainer

DeepEdit model uses the whole volume to do training and inference. It's not trained on patches so it doesn't use SlidingWindowInference

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPU memory requirement difference depending on which environment I am on #1029

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

GPU memory requirement difference depending on which environment I am on #1029

hourglassnam Sep 30, 2022

Replies: 3 comments · 4 replies

SachidanandAlle Sep 30, 2022 Maintainer

SachidanandAlle Sep 30, 2022 Maintainer

hourglassnam Sep 30, 2022 Author

SachidanandAlle Sep 30, 2022 Maintainer

hourglassnam Oct 3, 2022 Author

tangy5 Sep 30, 2022 Maintainer

diazandr3s Sep 30, 2022 Maintainer

hourglassnam
Sep 30, 2022

Replies: 3 comments 4 replies

SachidanandAlle
Sep 30, 2022
Maintainer

SachidanandAlle Sep 30, 2022
Maintainer

hourglassnam Sep 30, 2022
Author

SachidanandAlle
Sep 30, 2022
Maintainer

hourglassnam Oct 3, 2022
Author

tangy5
Sep 30, 2022
Maintainer

diazandr3s Sep 30, 2022
Maintainer