Overview of Nvidia Drivers for Lotus

[Reiers]

Good, tested and works:

• Driver Version 440.100
• Driver Version 450.19.03
• Driver Version 450.66
• Driver Version 450.80.02
• Driver Version 450.119.0
• Driver Version 455.38
• Driver Version 460.32.03
• Driver Version 460.56
• Driver Version 460.67
• Driver Version 460.84
• Driver Version 460.91.03

Stay on 460 for now.

Bad, tested:

• Driver Version 465.24.02
• Driver Version 465.27
• Driver Version 465.31
• Driver Version 470.57.02
• Driver Version 470.63.01
• Driver Version 495.44

If you need to downgrade your Nvidia Driver:

First:
sudo apt-get purge nvidia*
then:

1. wget https://us.download.nvidia.com/XFree86/Linux-x86_64/460.x.x/NVIDIA-Linux-x86_64-460.x.x.run
2. sudo chmod +x NVIDIA-Linux-x86_64-460.x.x.run
3. sudo ./NVIDIA-Linux-x86_64-460.x.x.run
4. reboot
5. check nvidia-smi that it is the correct driver
6. Downgrade complete (done)

On step 2,3 and 4 change x to whatever driver that you want, and remember 460 driver is needed to run a 3080/3090/30xx.

---

Warning: Arch Linux comes with the 465 non working drivers.

These drivers where tested with 2080ti, 3080, 3090 and TITAN RTX.
On Ubuntu 18.04, Ubuntu 20.04. ZorinOS 15.3.

Remember to turn off auto updates on Nvidia Drivers if you are on a stable version.
Disclaimer: Packages might depend on the driver that comes with the Linux distribution.

How to align kernel - driver - openCL:

apt install -y -t buster-backports nvidia-opencl-icd nvidia-opencl-common
#TODO this is optional
apt install -y -t buster-backports nvtop nvidia-smi

Please comment, if you have tested drivers that are not on the list - so we can add them.
Also incl. what Card and Linux distribution you are using.

[stuberman]

I can confirm these drivers work on my systems which run Ubuntu Server 20.04.2 LTS

1080 Ti 455.38
2080 Ti 455.38
3090 460.67

UPDATE Oct 26 2021
Now running Ubuntu Server 20.04.3 LTS

1080 Ti 455.38
2080 Ti 460.91.03
3090 460.91.03

BTW - after updating Ubuntu make sure you test your drivers again (run nvidia-smi) and reinstall if needed.

[Shekelme]

Any issues with PC2 on 3090s with this driver?

[stuberman]

Not at all.

[Shekelme]

That is extremely strange.

[Shekelme]

Do you use NVME mdadm arrays for sealing? If so, then what fylesystem do you use?

[stuberman]

For sealing I use NVMe disks running PCIe-Gen4x4
No arrays

[Angelo-gh3990]

I can confirm the following : running Ubuntu 20.04.1 LTS (slackID = Angelo)

GeForce RTX 3090 - version 460.73.01
GeForce GTX 1080 Ti - version 450.102.04

Thanks for putting this together

[TippyFlitsUK]

Running Ubuntu 18.04.5 LTS and no problems with the following setup:

Nvidia Tesla T4 - Version 450.19.03

[sharli91]

excuseme can you helpme how run lotus in ubuntu 18.04 LTS

[SBudo]

Running Ubuntu 20.04.2 LTS and have no issues with
Tesla T4 - version 460.32.03

Was running on version 450.80.02 without problems before too

[f8-ptrk]

can confirm the 460 driver woking on ubuntu2004 and debian 10 for 2080ti, 3080, 3090 Tesla T4 and nvidia A40, A6000

[neondragon]

Neonix's tested working list:

GPU	OS	GPU Driver	Kernel Version
RTX 3080	Debian 10	455.38 (from .run file)	5.5.0-0.bpo.2-amd64
RTX 3090	Debian 10	455.38 (from .run file)	5.5.0-0.bpo.2-amd64
Titan V	Debian 10	460.73.01 (from .run file)	5.5.0-0.bpo.2-amd64
Titan V	Debian 10	450.80.02 (from .run file)
RTX A5000	Debian 10	460.73.01 (from .run file)	5.5.0-0.bpo.2-amd64
Quadro M6000	Debian 10	390.138 (from .run file)

Does not work properly:
I have had issues with some 465.x and 460.x versions. In 'dmesg' you may see messages from the NVIDIA driver such as these:

[63442.509084] NVRM: GPU at PCI:0000:26:00: GPU-bd610676-d7d9-46b0-795d-3a4c82c1162f
[63442.509087] NVRM: GPU Board Serial Number:
[63442.509089] NVRM: Xid (PCI:0000:26:00): 13, pid=583, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Address
[63442.509162] NVRM: Xid (PCI:0000:26:00): 13, pid=583, Graphics Exception: ESR 0x514730=0xc01000e 0x514734=0x20 0x514728=0xc81eb60 0x51472c=0x1174
[63442.509474] NVRM: Xid (PCI:0000:26:00): 43, pid=3164, Ch 00000008

Tips
CLEAR YOUR OPENCL KERNEL CACHE!
The first time Lotus runs on a new GPU, OpenCL code (an OpenCL 'kernel') is compiled and cached. Using this same cached kernel after updating drivers or graphics libraries can result in undefined behaviour. For example, I have seen the above Graphics SM Warp Exception due to old cached kernels, and I have seen a WinningPoSt that runs and completes but instead of taking 4 seconds to run it took 2 minutes 38 seconds. So every time you change drivers or OpenCL libraries or lotus versions (when lotus has updated extern/filecoin-ffi), it can help to avoid problems by clearing the kernel cache and letting lotus re-generate it. You may want to run lotus-bench one time (as the lotus user) to let it generate the kernel cache, instead of waiting for it to be generated e.g. when WindowedPoSt starts.

The caches are in the home directory of the user running Lotus, at ~/.nv/ComputeCache and ~/.rust-gpu-tools. It's safe to delete these two directories.

[vmx]

Thanks a lot for the note about the kernel caching. I opened an issue to track it at filecoin-project/rust-gpu-tools#36.

[funktafide]

I can confirm 450.119.03 for 1080Ti on Ubuntu 20.04.2 LTS

[s0nik42]

GeForce RTX 2080 Ti / 455.45.01
GeForce RTX 3080 / 455.45.01
GeForce RTX 3090 / 460.32.03

[Shekelme]

RTX 3090, driver 460.84. It is working OK. Mostly OK.

[Shekelme]

Bad, tested:

Driver Version 470.57.02 (with RTX3090)

[j1v]

450.80.02 and 460.91.03 both work well with 2080ti

[cryptowhizzard]

I am running these ( 460.91.03 ) on 3090 .... Lot's of compute errors.

[nickboot]

Ubuntu 18.04 / 455.23.04 / GeForce RTX 3090
Ubuntu 20.04 / 460.91.03 / GeForce RTX 3090
Ubuntu 18.04 / 460.56 / GeForce RTX 3080

[chuanlong99]

Ubuntu 20.04 __ 470.63.01 ___ RTX 2080Ti bad
Ubuntu 20.04 __ 470.57.02 ___ RTX 2080Ti bad
Ubuntu 20.04 __ 465.27 ___ RTX 2080Ti bad
Ubuntu 20.04 __ 465.24.02 ___ RTX 2080Ti bad
Ubuntu 20.04 __ 460.67 ___ RTX 2080Ti Work well！

[dayou5168]

Os_not_clear_460.91_Locked arithmetic power RTX 3080TI
this is feedback from a slack community member

[stuberman]

@Reiers You can add 460.91.03 as tested good - see my update in original comment

[Reiers]

Thanks - I updated the post now with all the new information from the comments.

[scaseye]

Does NOT work: NVIDIA-SMI 470.74 Driver Version: 470.74 CUDA Version: 11.1 w/ RTX A6000

Does WORK: NVIDIA-SMI 470.74 Driver Version: 470.74 CUDA Version: 11.4.r11.4 w/ RTX A6000

[Angelo-gh3990]

Nope, stay on the 460 version for now. Also when you compile with CUDA support.

[scaseye]

compiled with CUDA support :)

[Shekelme]

11.4.0 or 11.4.x?

[Reiers]

Thanks guys !
I updated the post with the new information that you have provided.
Also added how to: downgrade.

Keep posting your findings and I will update the overview.

[micross]

Bad:

Driver Version 495.44 (with RTX3090)

[cheeseburgercod]

Ubuntu 20.04 - 460.91.03 - RTX 3070 - unknown if good/bad

I am new, just reading thru everything, lotus node working, still syncing 5 trillion messages, chain backup & snapshot work...

[Reiers]

460.91.03 is tested and a good driver. Thanks for reporting :)

[Shekelme]

Ubuntu 20.04, RTX3090 - 525.60.13, CUDA 12.0 - working perfectly.

[scaseye]

Running 525 and cuda 11.8 as well

On Sun, Mar 19, 2023 at 1:27 AM Shekelme ***@***.***> wrote: Ubuntu 20.04, RTX3090 - 525.60.13, CUDA 12.0 - working perfectly. — Reply to this email directly, view it on GitHub <#172 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQO55ATOLFG4TKUPU4HHZYLW427Q7ANCNFSM45R6U7JQ> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- -- Best, Casey Elliott HomeMate Realty Broker/Owner Direct|Txt: 949.667.3558 Office|Txt: 949.899.6002 Facebook: https://www.facebook.com/Casey-Elliott-Realtor-182611649482/ Twitter: https://twitter.com/SCaseyE YouTube: https://www.youtube.com/user/scaseye Website: http://www.newhomemate.com/scaseye BRE#01503233

[Hobbe81]

I tried the latest 555 driver with CUDA 12.5 and they seem to work. Running an nVidia RTX 3090.