CUDA_ERROR_OUT_OF_MEMORY

[Hopperpop]

Hi,

Thanks for sharing this great work. I'm trying to run the samples on smaller gpu: GTX1060 6Gb.
The "Einstein" examples runs fine, but when I run the fox example I get:

13:49:05 SUCCESS  Loaded 50 images of size 1080x1920 after 1s
13:49:05 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
13:49:05 INFO     Loading network config from: configs\nerf\base.json
13:49:05 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
13:49:05 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
13:49:05 INFO     Color model:   3--[SphericalHarmonics]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
13:49:05 INFO       total_encoding_params=13074912 total_network_params=9728
13:49:06 ERROR    Uncaught exception: ***\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:531 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: ***\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:452 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE

Is it still possible to run this example with some modified parameters for gpu's with lower memory, or should I give up?

Small note:
atomicAdd(__half2) is also not supported on my architecture (=61). I needed to disable it in "common_device.cuh".

[Tom94]

Hi there,

you might be able to further squeeze down the memory usage by reducing the resolution --width 1280 --height 720, but I'm unsure this will be enough.

Regarding atomicAdd(__half2): I'm surprised actually. How does this error manifest? I'd like to make this codebase work on as wide a range of GPUs as possible and both the CUDA documentation and CI suggest it should work on compute capability 61.

[Hopperpop]

When building I get:

D:***\include\neural-graphics-primitives/common_device.cuh(127): error : no instance of overloaded function "atomicAdd" matches the argument list [D:\***\build\ngp.vcxproj]
              argument types are: (__half2 *, {...})
            detected during instantiation of "void ngp::deposit_image_gradient(const Eigen::Matrix<float, N_DIMS, 1, <expression>, N_DIMS, 1> &, T *, T *, const Eigen::Vector2i &, const
   Eigen::Vector2f &) [with N_DIMS=2U, T=float]"
  D:\***\src\testbed_nerf.cu(1512): here

D:\***\include\neural-graphics-primitives/common_device.cuh(128): error : no instance of overloaded function "atomicAdd" matches the argument list [D:***\build\ngp.vcxproj]
              argument types are: (__half2 *, {...})
            detected during instantiation of "void ngp::deposit_image_gradient(const Eigen::Matrix<float, N_DIMS, 1, <expression>, N_DIMS, 1> &, T *, T *, const Eigen::Vector2i &, const
   Eigen::Vector2f &) [with N_DIMS=2U, T=float]"
  D:\***\src\testbed_nerf.cu(1512): here

Maybe it's a wrong dependency problem, instead of the hardware not supporting it:
Win10
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0
cmake version 3.23.0-rc1
Python 3.8.10

Running following command still gives me the same error:
./build/testbed.exe --scene data/nerf/fox --width 10 --height 10

But reducing the amount of photo's to 20, makes it possible to run it.

[Tom94]

Thanks for elaborating on the atomicAdd(__half2) issue. Could you pass the --verbose argument to the cmake build command and paste the nvcc CLI? I'm curious whether this issue is happening on your end (i.e. CMake is passing a too low compute architecture to your compiler).

[Hopperpop]

It's indeed the same issue. I'm seeing -gencode=arch=compute_52,code=\"sm_52,compute_52\" while it's still detects -DTCNN_MIN_GPU_ARCH=61. Adding the extra flags resolves it.

[smtabatabaie]

Can anyone elaborate on this? I couldn't get what has been done to fix the issue. Would really appreciate it.

[thiloschenkelberg]

Can anyone elaborate on this? I couldn't get what has been done to fix the issue. Would really appreciate it.

#204 (comment)

[Krulknul]

I just got the same error, running on a GTX 1080.
The fox does work for me, but when I try to run a dataset I prepared myself it gives this error. That was a very large dataset though, so I just tried shrinking it down and it still gives the same error. (smaller than the fox in MB's at this point.)

edit: I forgot to add, that adding --width 10 --height 10 also does nothing for me.

[3a1b2c3]

1050_ti runs out of memory as well

[JustASquid]

I'm also getting the same error regarding atomicAdd (Running on a GTX 1080 TI)

[chris-aeviator]

What are the VRAM requirements for the provided examples after all?

[Tom94]

The VRAM requirements vary with architecture, older GPUs unfortunately requiring more RAM due to needing fp32 for efficiency and not being able to run fully fused neural networks.

In general, it seems that 8 GB are enough to run fox in all cases -- so only a little push would be needed to make it fit into OP's 6 GB card.

[chris-aeviator]

Thanks for clarification.

[Krulknul]

In general, it seems that 8 GB are enough to run fox in all cases -- so only a little push would be needed to make it fit into OP's 6 GB card.

As I've said I've prepared my own dataset and I get this error no matter how far I optimize it, but the fox works every time. Unless I'm not really understanding how the VRAM requirements vary per dataset, it should be working, shouldn't it?
Fox has 50 jpg images, totaling 17.5 MB
My own set I have now shrinked down to 33 jpg images totaling to 2 MB, but it will still say there's not enough memory.

[Tom94]

There are multiple aspects to this:

1. The size of the actual jpg files does not directly matter. What does matter is the resolution of the images, because instant-ngp loads the images into memory in uncompressed form. To figure out how much memory your images need, calculate n_bytes = n_images * width * height * 4 * 2. The number 4 is the number color channels internal to instant-ngp, and the number 2 refers to the fact that 2 bytes (fp16) are used to represent each channel.
2. The aabb_scale setting of the dataset incurs additional memory overhead. The higher the aabb_scale, the more memory is used for cascaded occupancy grids. The fox dataset uses "aabb_scale": 4. See the NeRF dataset tips for additional details about this parameter.

Cheers!

[Krulknul]

Aha! That was absolutely it! Changing the aabb_scale from 16 to 4 and rerunning the preparation makes my set fit inside the card 😁

Amazing work. Thank you! I am a university student in my first year and amazed at what you guys are doing. May I please one day create these kinds of cool things too.😆

[khilller]

Thank you for explaining that aspect @Tom94 . It worked on my 1060!

[rgherta]

Same problem here

roman@DESKTOP-5B9D0K1 MINGW64 ~/Documents/instant-ngp (master)
$ ./build/testbed.exe --scene data/nerf/fox --width 10 --height 10
18:05:03 INFO     Loading NeRF dataset from
18:05:03 INFO       data\nerf\fox\transforms.json
18:05:03 SUCCESS  Loaded 12 images of size 1080x1920 after 0s
18:05:03 INFO       cam_aabb=[min=[1.47019,-1.33309,0.171354], max=[2.38974,-0.344117,0.314917]]
18:05:03 INFO     Loading network config from: configs\nerf\base.json
18:05:03 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 52. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 52. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 52. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
18:05:03 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
18:05:03 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
18:05:03 INFO       total_encoding_params=13074912 total_network_params=9728
18:05:05 ERROR    Uncaught exception: C:\Users\roman\Documents\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:558 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: C:\Users\roman\Documents\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:462 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE

$ nvidia-smi
Sun Apr  3 18:06:21 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 511.65       Driver Version: 511.65       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro M2200       WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   37C    P8    N/A /  N/A |      0MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

[aaronbshoemaker]

Running a 3090 I've found the limit for different resolutions and developed a short cut for knowing how many images I need. Take the total pixel count of the image and multiply by 11.77. Divide your vram by that number and you will get a good estimate for how many photos you can handle. Your card has 4GB of Vram I believe. You have to subtract what your system uses to operate. At 1920 x 1080 you only have room for 13 photos at 4GB. There are 2 things you can do. Decrease the size of the photos until the process runs, or run your monitor from the video out of your motherboard if you have one. Hopefully you can free up enough vram to make the fox run at current resolution.

[DeveloperNo579212]

Same error here CUDA_ERROR_OUT_OF_MEMORY, Could not free memory: CUDA_ERROR_INVALID_VALUE.

I tried to remove all but two images to create a minimal setup, still fails.

Thu Apr 7 15:57:53 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 511.23 Driver Version: 511.23 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P1000 WDDM | 00000000:01:00.0 Off | N/A |
| N/A 53C P8 N/A / N/A | 0MiB / 4096MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |

[YuhsiHu]

Hey, thank you for your great work!
My GPU is 3060 (12G) and I am trying to reconstruct self-collected scenes. All the images are 2976*3968 and transformed by colmap2nerf.py. Each scene has about 100 images and imgs in json file varies from 20 to 50. Some scenes can be reconstructed and others cannot.
14:36:06 INFO Loading NeRF dataset from 14:36:06 INFO /media/hyx/code/instant-ngp/data/xgrids/goact/transforms.json 14:36:07 SUCCESS Loaded 97 images of size 2976x3968 after 0s 14:36:07 INFO cam_aabb=[min=[0.394222,-0.798393,0.153654], max=[1.87405,1.80321,1.1615]] 14:36:07 ERROR Uncaught exception: Could not allocate memory: /media/hyx/code/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:112 cudaMalloc(&rawptr, n_bytes+DEBUG_GUARD_SIZE*2) failed with error out of memory
I tried to change the aabb_scale and add params such like --width 10 --height 10 or specify the json file but it did not work. How can I compute how many images can I reconstruct? Why did I success on some scenes which have more images but fail in scenes with less images?
Thank you for your time.

[smtabatabaie]

I'm getting the same error when I try to run "G:\NERF\instant-ngp>build\testbed.exe --scene data/toy_truck --width 10 --height 10"
I get the following error:

"failed with error CUDA_ERROR_OUT_OF_MEMORY, could not free memory ...../gpu_memory.h:465 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE"
I'm trying to run it on NVIDIA GeForce GTX 980 with 16GBs RAM.
I tried hard coding
list(APPEND CUDA_NVCC_FLAGS "-gencode=arch=compute_52,code=\"sm_52,compute_52\"") in my CMakeLists.txt files but still no luck. I've followed the instructions and everything worked and compiled and ran fine but still, I get this error. I'd appreciate it if someone could help me with this.

[cookins]

let me add my GPU too, with same issue. GTX970. but its kinda in supported list. it showing architecture 52..

[minch-dev]

GeForce GT 1030 here with 2gb of vram (architecture 61). I reduced the fox dataset to 3 images and yet it still fails. Is it absolutely necessary to load all images at once in uncompressed form in vram? Maybe at least doing it in chunks might fix the issue...

[Wang-fighter]

The same error occurred on the GTX1650 with 4GB. I reduced the number of images to two, but still reported this error.

17:07:49 SUCCESS Initialized OpenGL version 4.6.0 NVIDIA 516.94
17:07:57 INFO Loading NeRF dataset from
17:07:57 INFO D:\application\Instant-NGP-for-GTX-1000\Instant-NGP-for-GTX-1000\data\s2\transforms.json
17:07:57 SUCCESS Loaded 2 images after 0s
17:07:57 INFO cam_aabb=[min=[1.70659,0.570126,0.484637], max=[1.87223,0.88163,0.532478]]
17:07:57 INFO Loading network config from: .\configs\nerf\base.json
17:07:57 INFO GridEncoding: Nmin=16 b=3.28134 F=4 T=2^19 L=8
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
17:07:57 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
17:07:57 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
17:07:57 INFO total_encoding_params=13194816 total_network_params=10240
Attempting to free memory arena while it is still in use.
Could not free memory arena: D:/a/instant-ngp/instant-ngp/dependencies/tiny-cuda-nn/include\tiny-cuda-nn/gpu_memory.h:489 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE
17:07:58 ERROR Uncaught exception: D:/a/instant-ngp/instant-ngp/dependencies/tiny-cuda-nn/include\tiny-cuda-nn/gpu_memory.h:592 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY