[BUG/Discussion] Single RTX 3080 shutdowns computer during training after a few epochs

[timothylimyl]

Describe the bug
train script causes auto-shutdown, train.py

To Reproduce
Steps to reproduce the behavior:

1. python train.py /data/imagenette/ --model resnet34

Tested 10 times, behavior is roughly the same, after 10-20epochs, desktop will shutdown.

Expected behavior
Computer to not shutdown/restart

Desktop (please complete the following information):

• OS: Ubuntu 20.04
• PyTorch version w/ CUDA/cuDNN: 1.8.1 / 1.7, cu11.1
• Nvidia Driver: 470.82
• EVGA PSU 850W GOLD

Additional context
This is is not really related to this repository completely and it is a GPU hardware problem. However, after searching the web, it seems like a common problem for 3080/3090 and it is potentially caused by power fluctuation which this repository training script triggers. Many forums are all dedicated to gaming issues faced by 3080/3090 which is caused by power fluctuation in games only.

I saw that @rwightman actually manage to train on 3090 and was mainly wondering did u face any issue with the power fluctuation issues that made your desktop shut down? I am sure it is not a PSU issue (850W is over the requirement by 100W), the NVIDIA driver is new which supposedly fixes this issue. Mainly posting this issue, in case anyone has faced the same issue and fixed it.

Power fluctuation: Steady training state is roughly 340W, it can go down to 100-200W.

Additional Test

1. Does it work on another single GPU?

Yes, tested it with GTX1080, the power fluctuates but it works (never shutdown).

2. How did you confirm is GPU power fluctuation causing the issue?

a. I monitored the GPU using nvidia-smi and see that when the power drops by 100-200W, the computer will hang and restart.

Edit: sometimes it does go for a few more fluctuation cycle before shutting down.

b. I ran a script drawing ~340W consistently for an hour without any issues.

[rwightman]

@timothylimyl this should be a discussion, not a bug

It is most likely power or heat. Having a power supply with big enough numbers doesn't mean it delivers. Evga is usually good but I only trust their G/P/T2 and G3 lines that use superflower, any other series is a big?

[timothylimyl]

@rwightman, noted with thanks.

The PSU is 850W GR (EVGA GOLD), I think the power should be fine unless there is a sudden spike over the power limit that I am not noticing. I have tested it running on this script that draws ~340W consistently and there has not been any issue for an hour:

import torchvision.models as models
import torch

model = models.densenet121(pretrained=False).cuda()

x1 = torch.ones((31, 3, 224, 224), dtype= torch.float32).cuda()

while True:
	y  = model(x1)

Meanwhile, running the training with power fluctuating will cause auto-restarts roughly 5-10minutes in.

The temperature as of nvidia-smi is <80c which should be fine too, it usually auto-restarts only when the power fluctuates (from observation).

[rwightman]

@timothylimyl since others will probably wander upon this thread with similar issues, I'll repeat a few things I've learned over the years and tears related to unstable systems:

1. It is usually power with a sprinkle of heat, at least if you get a sudden shutdown or restart under load
2. If the shutdown under load happens quickly, the PSU is either undersized or cannot meet the transient demand of the workload (deep learning is worst case here), quite likely that over current or undervoltage protection was triggered
3. If the shutdown happens after a long time, it could still be the above, but more likey that continued high temps have contributed to derating of the PSU output. You may also have triggered and over temp protection
4. Your power cord and home circuit can contribute (if the voltage at the outlet is on the low side), this is more of a problem when you're using 1600W supplies with multiple GPU
5. I have come to ONLY trust Supwer Flower Leadex designs in recent years, I've been bit in the ass several times by other manufacturers. GR? I don't know that one, but G5 and some of the other G{x} supplies in EVGA line are FSP (I don't trust), and G6 is Seasonic (Used to trust, no longer). The G2 and G3 are Super Flower and I have a number of those running fine, even under spec'd. It's unfortunate that EVGA doesn't use Super Flower in their new lines. But Super Flower does appear to sell directly now, at least in the USA, I'm out of luck in Canada so far.
6. Seasonic and higher end Corsair (AXi - built by Flextronics) should be top quality supplies, and I'm sure for most use cases they are. However, they both seem to have current/voltage/thermal protections that GPUs like to trigger. Super Flower Leadex seem much more tolerant (some reviews state they don't provide enough protection, but with NVIDIA GPU it would seem you need a supply that won't freak out the moment the demand is out of spec in order to avoid shutdowns)
7. If your crashes aren't sudden power down / restarts, but a hang of some sort, or a protection fault with some error / memory addresses dumped on the screen and maybe a little different every time, it's likely bad RAM or CPU, possibly timing failures under high temps

[timothylimyl]

Hi @rwightman , thank you for the insights, I really appreciate it.

From reading what you said, I will approach the sellers on this, I initially thought it could be a software issue.

[timothylimyl]

Hi all, in case anyone is facing the same issue as me, the company figure that the issue is that the ventilation was insufficient and the gpu was overheating. I guess the nvidia-smi temperature data is not very accurate.

The seller fixed it by adding more fans and removing the top case to allow more airflow.