CUDA out of memory

[kukumallou]

First of all, thanks for the contribution. Very nice project.
I came across CUDA out of memory when running dense reconstruction (run_neuralangelo-colmap_dense.sh)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.05 GiB (GPU 0; 23.64 GiB total capacity; 19.51 GiB already allocated; 911.19 MiB free; 20.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have tried to smaller the number of samples per ray from 1024. to 512 and 256 as suggested in the FAQ. But the error message is the same. BTW I have succeeded in running sparse reconstruction script and got correct results. Any idea to fix this problem? Thanks a lot

[hugoycj]

Would you mind testing the latest version, and replace python launch.py --config configs/neuralangelo-colmap_dense-SH.yaml --gpu 0 --train dataset.root_dir=$INPUT_DIR with python launch.py --config configs/neuralangelo-colmap_dense.yaml --gpu 0 --train dataset.root_dir=$INPUT_DIR in run_neuralangelo-colmap_dense.sh script.

[kukumallou]

I tried the latest version (Tue Nov 2), but the error still exists. The card I got is a 4090 with 24G memory.

[hugoycj]

Sorry to bother you. Would you mind to share in which step the out of memory happens and what's the resolution of your images

[kukumallou]

There're 140 images with resolution 1920x1440. And below is the output log of the script.

---sfm---
Sparse map datasets/cake exist. Aborting
---model_converter---
---colmap2mvsnet---
Image pair datasets/cake/dense/pair.txt exist. Aborting
Number of model parameters: 1162696
load third_party/Vis-MVSNet/pretrained_model/vis/-1
(1, 1, 528, 960): 100%|█████| 140/140 [02:39<00:00, 1.14s/it]
---mvsnet_fusion---
load data: 100%|███| 140/140 [00:01<00:00, 137.01it/s]
prob filter: 100%|███ 140/140 [00:00<00:00, 203.46it/s]
vis filter and med fusion: 100%|████| 140/140 [00:05<00:00, 27.54it/s]
vis filter and ave fusion: 100%|████| 140/140 [00:04<00:00, 31.20it/s]
vis filter: 100%|███| 140/140 [00:04<00:00, 30.62it/s]
back proj: 100%|████| 140/140 [00:00<00:00, 293.64it/s]
Construct combined PCD
Estimate normal
---angelo_recon---
Global seed set to 42
Using 16bit native Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Trainer(limit_train_batches=1.0) was configured so 100% of the batches per epoch will be used..
Global seed set to 42
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1

distributed_backend=nccl
All distributed processes registered. Starting with 1 processes

Loading dense prior from datasets/cake/dense/fused.ply
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params

0 | model | NeuSModel | 28.0 M

28.0 M Trainable params
0 Non-trainable params
28.0 M Total params
55.914 Total estimated model params size (MB)
Epoch 0: : 0it [00:00, ?it/s]Update finite_difference_eps to 0.06801176275750971
Traceback (most recent call last):
File "launch.py", line 125, in
main()
File "launch.py", line 114, in main
trainer.fit(system, datamodule=dm)
File "/home/****/anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
self._call_and_handle_interrupt(
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 648, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, kwargs)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
return function(*args, kwargs)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
results = self._run_stage()
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
return self._run_train()
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train
self.fit_loop.run()
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
self.advance(*args, kwargs)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 271, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
self.advance(*args, kwargs)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 194, in advance
response = self.trainer._call_lightning_module_hook("on_train_batch_start", batch, batch_idx)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1550, in _call_lightning_module_hook
output = fn(*args, kwargs)
File "/home//Dev/instant-angelo/systems/base.py", line 57, in on_train_batch_start
update_module_step(self.model, self.current_epoch, self.global_step)
File "/home//Dev/instant-angelo/systems/utils.py", line 351, in update_module_step
m.update_step(epoch, global_step)
File "/home//Dev/instant-angelo/models/neus.py", line 111, in update_step
self.occupancy_grid_bg.every_n_step(step=global_step, occ_eval_fn=occ_eval_fn_bg, occ_thre=self.config.get('grid_prune_occ_thre_bg', 0.01))
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, kwargs)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/nerfacc/grid.py", line 271, in every_n_step
self._update(
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, kwargs)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/nerfacc/grid.py", line 229, in _update
occ = occ_eval_fn(x).squeeze(-1)
File "/home//Dev/instant-angelo/models/neus.py", line 104, in occ_eval_fn_bg
density, _ = self.geometry_bg(x)
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, kwargs)
File "/home//Dev/instant-angelo/models/geometry.py", line 125, in forward
out = self.encoding_with_network(points.view(-1, self.n_input_dims)).view(*points.shape[:-1], self.n_output_dims).float()
File "/home//anaconda3/envs/objmodel/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, kwargs)
File "/home//Dev/instant-angelo/models/network_utils.py", line 193, in forward
return self.network(self.encoding(x))
File "/home/*/anaconda3/envs/objmodel/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, kwargs)
File "/home//Dev/instant-angelo/models/network_utils.py", line 76, in forward
return self.encoding(x, *args) if not self.include_xyz else torch.cat([x * self.xyz_scale + self.xyz_offset, self.encoding(x, *args)], dim=-1)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.15 GiB (GPU 0; 23.64 GiB total capacity; 19.97 GiB already allocated; 970.44 MiB free; 20.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Epoch 0: : 0it [00:07, ?it/s]
start time: 2023-11-03 08:46:23
sfm time: 2023-11-03 08:46:23
model_converter finished: 2023-11-03 08:46:24
colmap2mvsnet finished: 2023-11-03 08:46:25
mvsnet_inference finished: 2023-11-03 08:49:06
mvsnet_fusion finished: 2023-11-03 08:49:33
angelo_recon finished: 2023-11-03 08:50:11

[lyupei]

Hi, I have decreased the model.num_samples_per_ray from 1024 to 128, but still encountered vram OOM issues. I'm using a 2070 with 8G vram, can I run this project by adjusting other parameters?

[jianghr-shanghaitech]

@kukumallou @lyupei Wondering if you guys have solved the problem, it seems that OOM will happen if the VRAM is below 11G. Like 3080 with 10 VRAM.