out of memory cuda

[NoUserNameForYou]

INFO Loading NeRF dataset from
INFO data\nerf\fox\transforms.json
SUCCESS Loaded 8 images of size 216x384 after 0s
INFO cam_aabb=[min=[1.13457,0.406441,0.398352], max=[1.84657,1.63633,0.546498]]
INFO Loading network config from: configs\nerf\base.json
INFO GridEncoding: Nmin=16 b=1.66248 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 52. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 52. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
INFO total_encoding_params=13623184 total_network_params=9728
ERROR Uncaught exception: E:\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:531 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: E:\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:452 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE

I reduced the number of images as well as their resolution and crated a new json yet I still get memory error. Tried the --width --height trick as well. I'm on gtx970 and I allwoed testbed, python and related executables with Optimize Cuda in nvcpl. Cuda 11.6 + optix 7.3 (no issues with building) + python 3.8 + win 11

[metantonio]

Try to down abbscale (open the json file) to 1.

[walt-artist]

I'm getting this also. Which json file do you mean?

[walt-artist]

Nevermind, I found it. It seemed to be in the transforms.json on the fox example ,but not on hotdog.

[dino200606]

I got the same issue, it only happon on fox
14:27:40 SUCCESS Loaded 50 images of size 1080x1920 after 0s
14:27:40 INFO cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
14:27:40 INFO Loading network config from: configs\nerf\base.json
14:27:40 INFO GridEncoding: Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
14:27:40 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
14:27:40 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
14:27:40 INFO total_encoding_params=13074912 total_network_params=9728
14:27:40 ERROR Uncaught exception: G:\GitHub\instant-ngp\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:565 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: G:\GitHub\instant-ngp\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:469 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE

[walt-artist]

I get this too:

[mm23mm]

Hi, I'm having the same issue , do you have any idea to solve it? Thank you!!

[dino200606]

Hi!  ProjectPath/data/nerf/fox/transforms.json. at the line14,  the value "aabb_scale" , it default 4,change it to 1.

…

------------------ 原始邮件 ------------------ 发件人: "NVlabs/instant-ngp" ***@***.***>; 发送时间: 2022年3月9日(星期三) 晚上10:29 ***@***.***>; ***@***.******@***.***>; 主题: Re: [NVlabs/instant-ngp] out of memory cuda (Discussion #283) Hi, I'm having the same issue , do you have any idea to solve it? Thank you!! — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: ***@***.***>

[kajc10]

Same issue, modified abbscale to 1, reduced number of images, but probably my GPU is not enough (GTX 1050 notebook version).

[leventt]

yes 32 GB should be plenty... not sure what else to suggest :/

[kajc10]

Can't it be some CUDA misconfiguration? I already had an issue #347 due to lost env variables, is there anything else I could check with CUDA?

[kajc10]

Or anything to do with the fact I'm working in a docker container.

[kajc10]

Update: i used the provided Dockerfile, now the issue vanished in my new container (same resources)

[Ash93196]

excuse me @kajc10 i have the same issue(CUDA_OUT_OF_MEMORY)

could you please explain in details how did you fix the problem on windows machine that you have as i have nearly the same hardware like you "NVIDIA MX150 4Gb" ?
i didn't get what is meant by "docker container" ?
thanks in advance....

[leventt]

To get a better idea where memory is allocated and where to cut from to accommodate this model for your GPU, define TCNN_VERBOSE_MEMORY_ALLOCS

you may find if you are on an older GPU as I am, most of the memory footprint is sensitive to the multi-res hash encoding hyper parameters, which will affect the memory usage exponentially. I believe memory grows O(n^3) with some of those hyper parameters.

[NoUserNameForYou]

8 images of size 216x384 with aabscale modded to 1 and leventt's tweaks vmem usage was at 3.9 gb and outcome looked unrecognizable.

Nonrtx gpus are sol.

[Komti13]

Hello i am trying to run testbed on bunny.obj using SDF and keep getting this error.
root@400d274af02c:/volume# ./build/testbed --scene data/sdf/bunny.obj
22:44:06 SUCCESS Loaded mesh "data/sdf/bunny.obj" file with 1 shapes.
22:44:06 SUCCESS Built TriangleBvh: nodes=1365
22:44:06 WARNING OptiX was not built. Falling back to slower TriangleBVH::ray_intersect.
22:44:06 SUCCESS Built TriangleOctree: depth=10 nodes=287672 dual_nodes=1150344. Populating dual nodes...
22:44:07 SUCCESS Loaded mesh: triangles=4968 AABB=[min=[-0.0956273,0.032063,-0.0629261], max=[0.0620257,0.188243,0.0599615]] after scaling=[min=[0.00790903,0.0125811,0.118168], max=[0.992091,0.987419,0.881832]]
22:44:07 INFO Loading network config from: configs/sdf/base.json
22:44:07 INFO GridEncoding: Nmin=16 b=1.38191 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
22:44:07 INFO Model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=4)]-->1
22:44:07 INFO total_encoding_params=12196240 total_network_params=6656
Segmentation fault (core dumped)
i did some research and i think it's memory error and i have plenty of memory available around 54Go
does anyone have this issue before?

[sipink]

I am running a gtx3080 set it to one and still nothing, to be fair the images are huge... but not sure what to do to process such images in the future if it crashes,,,, can we not process large photos or many of them. Seems like a big limitation... hmm