Resizable BAR on unsupported NVIDIA GPUs

[ghost]

Sorry to bother but lspci shows 4G BAR and nvidia-smi shows only 256MB?

	Memory at 13e00000000 (64-bit, prefetchable) [size=4G]
	Memory at 13f00000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at fb000000 [virtual] [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

[alex@fedora ~]$ nvidia-smi -q -d memory

==============NVSMI LOG==============

Timestamp                                 : Sun Oct  8 11:58:15 2023
Driver Version                            : 535.113.01
CUDA Version                              : 12.2

Attached GPUs                             : 1
GPU 00000000:03:00.0
    FB Memory Usage
        Total                             : 4096 MiB
        Reserved                          : 59 MiB
        Used                              : 247 MiB
        Free                              : 3788 MiB
    BAR1 Memory Usage
        Total                             : 256 MiB
        Used                              : 7 MiB
        Free                              : 249 MiB
    Conf Compute Protected Memory Usage
        Total                             : 0 MiB
        Used                              : 0 MiB
        Free                              : 0 MiB

What do others get?

Edit by xCuri0

Locking this discussion, for any further discussion go to the NvStrapsRebar repo https://github.com/terminatorul/NvStrapsReBar

The latest version does not require any compiling like you may have seen in outdated guides/discussion

[xCuri0]

What GPU is this ? I don't think NVIDIA even made any 4GB VRAM cards with Resizable BAR. You probably need to use full size

[ghost]

That one is a GTX1050Ti configured to have a different sized BAR, in this case 4GiB. Answer noted, sorry for the inconvenience of the question.

[xCuri0]

@Xelafic how did you get 4GB BAR on a GTX1050Ti ?

It's probably some issue with the NVIDIA driver because it doesn't expect a 1050Ti to have 4GB BAR

[ghost]

Nvidia drivers are an issue. Cards post under Linux (F38) but am thinking range is truncated. However could be a bug and as I don't have anything else to test that I ask for help with a simple question. Windows is another story. Early Pascal driver sees code 43 when installed, later 5xx drivers and maybe some 4xx cause instant lock or reboot, no blue screen and happen early at start of OS boot if installed. GT640 with 2GiB aperture loads though, not sure if that's truncated or not, can also set down to 64MB for testing. A simple synthetic bench that shows differences would be nice but not sure of best way to go or if even practical.

Configuration of aperture has seemingly been around on Nvidia cards years before Rebar spec appeared in 2008.
See https://envytools.readthedocs.io/en/latest/hw/io/pstraps.html
Note that Pascal bits are 22:20 at dword of offset 0x10100c and give an aperture range from 256MiB to 32GiB.

Why would Nvidia have this option if it was never meant to be used. Possibly a driver bug but it's not like Nvidia are new to large apertures.

I'm only looking at this through curiosity and have no real need for a large aperture myself.

[ghost]

Before OS loads.

[xCuri0]

@Xelafic have you confirmed BAR isn't overlapping with anything ?

ideally it should be done before PCI initialization like ReBarUEFI does, can help you with that if you want.

Windows atleast AMD drivers don't really like if you change BAR after PCI initialization, causes the control panel to show resizable BAR incorrectly but it still works.

If you want best chance of this working, modify ReBarDxe so it can configure NVIDIA GPU BAR size. Can help with this if you need

[ghost]

Checked mapping was okay and have already tried a hacked RebarUEFI just in case I missed something with using post mod or UEFI driver (bcfg) add on. Sorry about that but your skills are way better than mine. I just do a little programming as a hobby and use asm as I get lost with C. I do like the way you hooked though, very nice.

I need to take a closer look at the Nvidia driver, not appealing really but a good chance your right with unsupported aperture I feel.

[xCuri0]

@Xelafic You need to modify your BIOS to add the ffs driver, it works by hooking the PCI initialization.

Edit: seems like it will be complicated as you have to set a temporary BAR to access PSTRAPS

[ghost]

Yes, that is what I did.
In the executable part of the ffs I nop'd after

    pciAddress = EFI_PCI_ADDRESS(addrInfo.Bus, addrInfo.Device, addrInfo.Function, 0);
    pciReadConfigWord(pciAddress, 0, &vid);
    pciReadConfigWord(pciAddress, 2, &did);

    if (vid == 0xFFFF)
        return;

which in asm is

...
   lea r8, [rbp + 0x58]                  ; ret value   
   mov edx, 2                            ; offset 2,did
   mov rcx, rsi                          ; pciAddress  
   call pciReadConfigWord                ; read did    
   mov eax, 0xffff                                     
   cmp word [rbp + 0x50], ax                           
   je 0x180000706    

and added asm after to check for PCID of Video card and if detected assign some temp values to bridge and video card so as to access strap then set them back to original values. ffs checksums corrected and inserted into BIOS. Also some other stuff removed such as GetVariable...

What I did was quite horrible but gets the test done, after aperture set BIOS takes care of addressing instead of having to do it ourself. Thank you for RebarDxe, I would not have come up with such an elegant method.

[xCuri0]

@Xelafic did it make any difference (especially on Windows) ?

Also maybe could share the code in case anyone else finds it useful

[ghost]

No difference.

Using Windows Display Adapter



Note there is no resizable aperture in the traditional sense of a capability and no dynamic sizing although it could be possible if support was given which is unlikely.

Note: added asm code is hard coded for convenience to my hardware so not very useful. I may have one too many changes too, not sure and haven't tested. The Debugger was to put out some numbers on the Debug LEDs, used while checking things earlier so I can see what is happening. Doesn't really need to be there.

 _256M  = 0
 _512M  = 1
 _1G    = 2
 _2G    = 3
 _4G    = 4
 _8G    = 5
 _16G   = 6
 _32G   = 7 

        cmp     word[rbp+0x50],0x10de
        jne     NotNv
        cmp     word[rbp+0x58],0x1b80
        jne     NotNv

        mov     al,0x77   ; output 77 to the debug LEDs
        call    Debugger
        ;;;            Cmd           Bus            Dev           Fun      Reg
        mov     esi,(1 shl 31) or (0 shl 16) or (03 shl 11) or (0 shl 8) or 4 ; 80001804h  80866f08 ; Bridge

        sub     rsp,8*4

        call    ReadDword               ; 0x4
        push    rax
        or      eax,7
        mov     ecx,eax
        call    WriteDword

        add     esi,0x14                ; 0x18
        call    ReadDword
        push    rax
        or      eax,0x30300
        mov     ecx,eax
        call    WriteDword

        add     esi,0x4                 ; 0x1c
        call    ReadDword
        push    rax
        mov     ax,0xe0e0
        mov     ecx,eax
        call    WriteDword

        add     esi,0x4                 ; 0x20
        call    ReadDword
        push    rax
        mov     eax,0xfb00fa00
        mov     ecx,eax
        call    WriteDword

        mov     edi,esi
;--------------------------------
        ;;;            Cmd           Bus            Dev           Fun      Reg
        mov     esi,(1 shl 31) or (3 shl 16) or (00 shl 11) or (0 shl 8) or 4 ; 80030004h  10de1b80 ; GTX 1080

        call    ReadDword               ; 0x4
        push    rax
        or      eax,3
        mov     ecx,eax
        call    WriteDword

        add     esi,0xc                 ; 0x10
        push    rax
        mov     [rsp+0x30],rax
        mov     eax,0xfa000000
        mov     ecx,eax
        call    WriteDword

;--------------------------------  ; set strap

        mov     edx,0xfa10100c
        mov     eax,[rdx]
        and     eax,0xff8fffff
        or      eax,(_8G shl 20) or 0x80000000
        mov     [rdx],eax

;-------------------------------- ; restore values for GTX 1080

        pop     rcx
        call    WriteDword              ; 0x10

        sub     esi,0xc
        pop     rcx
        call    WriteDword              ; 0x4

;-------------------------------- ; restore bridge values
        mov     esi,edi
        pop     rcx
        call    WriteDword              ; 0x20

        sub     esi,4
        pop     rcx
        call    WriteDword              ; 0x1c

        sub     esi,4
        pop     rcx
        call    WriteDword              ; 0x18

        sub     esi,0x14
        pop     rcx
        call    WriteDword              ; 0x4


        add     rsp,8*4
        jmp     NotNv

 ReadDword:
        mov     eax,esi
        mov     dx,0xcf8
        out     dx,eax
        mov     dx,0xcfc
        in      eax,dx
        ret

 WriteDword:
        mov     eax,esi
        mov     dx,0xcf8
        out     dx,eax
        mov     dx,0xcfc
        mov     eax,ecx
        out     dx,eax
        ret

 Debugger:
        out     80h,al
        mov     edx,0x80000000
 @@:    dec     edx
        jnz     @b
        ret


 NotNv:

[xCuri0]

Also try 1 or 2GB BAR, it could be NVIDIA using a 32-bit uint on Windows maybe

[ghost]

Anything over 256MiB gives the same result.

[xCuri0]

@Xelafic able to send a full dmesg log (with the increased BAR size) ? maybe it will have something useful

[ghost]

I'm going to take a break for a while.

Thank you for your help and suggestions. Appreciated.

[xCuri0]

My guess is that NVIDIA hardcoded 256MB for all pre-Ampere GPUs (except maybe Turing because they support Resizable BAR to smaller sizes), but the NVIDIA Linux open source driver now supports Pascal it may be fixable there at least.

Or there could be another bit needed to be set for >256MB BARs to function properly.

I think nvidia-smi uses the function nvmlDeviceGetBAR1MemoryInfo to get BAR info.

Probably this https://github.com/NVIDIA/open-gpu-kernel-modules/blob/f59818b7517a913d93adf714f1f45ae69906e207/src/nvidia/src/kernel/gpu/mem_sys/kern_mem_sys_ctrl.c#L246

[terminatorul]

@Xelafic I have always wanted ReBAR on my RTX 2000 card, do you know the BAR size bits (and offset) for Turing ? Are they the same as Pascal ?

[terminatorul]

Also, is there a missing call to ReadDword for the GTX 1080 second register ? (register at 0x10, after line 60) ?

        add     esi,0xc                 ; 0x10
                                        ;   Missing call to ReadDword here ?
        push    rax
        mov     [rsp+0x30],rax          ; also, this appears to do nothing ?
        mov     eax,0xfa000000
        mov     ecx,eax
        call    WriteDword

[UnidentifiedTag]

if anyone knows how to add resize to a gtx 1660 let me know, must be some tweaking we can do

[terminatorul]

GTX 1660 should be a Turing architecture card, like my RTX 2080 Ti (can you show a CPU-Z screenshot ?).

Well trying ReBAR on it is like the summary of this entire discussion thread:

More then a decade ago, video cards from this manufacturer were reverse-engineered in order to develop the Linux open-source driver (nouveau). The resulting documentation is available under envytools on readtthedocs (link posted by @Xelafic in this thread). This documentation indicates at the time it was possible to configure ReBAR on the PCIe bus by writing to configurations registers (exposing straps bits) that are exposed by the GPU on the PCI bus. The motherboard UEFI could do this if properly modded, before the BAR size of the GPU is being negotiated during PCIe bring-up.

This is what @Xelafic discovered and implemented (in assembly language) on some older hardware:

• GTX 1050 Ti, GTX 1080 (Pascal architecture)
• GTX 650 (pre-Fermi or Fermi architecture, I don't remember)

Unfortunately the PCIe bring-up and configuration is only the first part of the story. The manufacturer does not support ReBAR in the driver (in software). So when the Windows driver is loading, @Xelafic already reports the computer just reboots with the Pascal card... ! With the (pre-)Fermi card, it appears to work, but the Linux driver utilities report the original BAR size of 256MB (despite the PCIe bus being configured to 2 or 4GB). The Windows driver also shows no performance improvement, so it is not clear if it is using the new BAR size of 2GB, or how to test for it...

But my newer Turing card advertises ReBAR support in the registers exposed on the PCIe, and can be configured with RebarUefi to use 256, 128 or 64 MB BAR size. So for Turing maybe there is a glimmer of hope that the manufacturer did not directly hard-code and lock-down the BAR size, but maybe is still using the value reported by the vBIOS. It is very likely though that BAR size is still capped (limited) by the vBIOS or the software driver to a maximum of 256 MBytes. So I am still eager to try an UEFI mod for changing BAR size, but this is not the time to get my hopes up yet... :(

If only I could find the time to just focus on this issue for about a day ... but life (the "real life" if you're a gamer or a nerd) keeps throwing at me all sorts of issues like problems at my day job, just to throw me off-track. I have put together by now a list with most of the pieces I need, by bugging :) the original author in this thread, but it's still difficult for me as I am not an UEFI modder myself (I work mostly as an embedded developer, but not with UEFI drivers).

If you or someone else can write C code (or assembly), then I can detail here the steps I collected, so you can try this on your bord. For starters you need to extract the bus/device/function numbers for your GPU model, and for the associated PCI-to-PCI bridge inside the CPU (from the CPU-Z .txt report) and put the numbers into the C code that is linked in this thread. Then the RebarUefi DXE driver needs to be modified, to insert the new C code into PreprocessControllerOverride() function, most likely right after the nested call to the overriden preprocessController function. RebarUefi driver must be recompiled according to the readme page of the github project -- this was quite easy --, then the motherboard UEFI should be re-flashed according to instructions on the same readme page of the project.

Anyhow ... look at me talking about what needs to happen instead of actually doing it ... they should assign me to management already. So wish me luck with all this ... !

[UnidentifiedTag]

mine a GTX 1660 SUPER, currently im not at the system but i believe i got the motherboard to add resizebar but then i found out the gpu doesn't support resizebar, must be some way to add it. if it works in ubuntu then it must be able to work in windows, just needs the driver code changing.

always a catch with these things, i have no idea how to program in c code, just know if you do it, you'll be helping alot of people get 20% extra enjoyment for free. thats what i always think when i write a guide online, good luck to you and for us all i hope?

[ghost]

Newest card I have is Pascal, I haven't seen Turing. Sorry.

Umm, that code was just quick and dirty for testing, give me a moment to look as I have forgotten it.

mov [rsp+0x30],rax ; also, this appears to do nothing ?
was probably left over from earlier coding when changes were made.

; Missing call to ReadDword here ?
BAR0. Empty so nothing to read.

[terminatorul]

As you know I think many people would like to enable ReBAR on their cards, and the manufacturer just doesn't care.

So can you please please give me an overview some context on what your assembly code is doing ? Really just any details and more information to get me started on how does it work ? Like where did you get the 0x0cf8 / 0x0cfc I/O port addresses ? PCI specs by chance ?

Also see here my translation of your assembly to C++:
https://pastebin.com/2CWsYLjq#jqvfEb1C

[terminatorul]

My issue with the missing call to ReadDword also goes to the corresponding call to WriteDword, when the values for GTX 1080 are being restored.

If the original value was never read, is it correct to write there the original value of the other register at offset 0x0004 ?

;-------------------------------- ; restore values for GTX 1080

        pop     rcx                     ; so what is being pop'ed and written here ?
        call    WriteDword              ; 0x10

        sub     esi,0xc
        pop     rcx
        call    WriteDword              ; 0x4

Hope I am not bothering you too much with these ... ! :)

[ghost]

idk c++, even asm is just a hobby. Never worked in software or computer industry. cf8/cfc are standard IO for x86 to read PCI.

The code was used this way because of implementation into @xCuri0 hooking but can also do it at post by reallocating addresses I think. RUEFI is a good tool to use for experimenting, probably best suited with 2 graphics cards, one for display and one for testing.

pop rcx ; so what is being pop'ed and written here ?
Whatever was pushed before, a bug. I never really paid it much attention as it worked and code only posted as was requested by @xCuri0

I did update it but can't remember where, dementia kicking in :(

[terminatorul]

About the PCI bridge, is it the same as the motherboard chipset ? Do you have your GPU (GTX 1080) connected directly to the CPU (usually the first PCIe slot), or to the chipset (last PCIe slot on the motherboard) ? Does the assembly code work the same in both cases ?

[ghost]

That version is hard coded to the first PCIe slot which on my Taichi x99 is 8086:6f08. I can put some more info together but would be later if your interested. Don't be surprised if buggy though ;)

[terminatorul]

@xCuri0

To my understanding this is how to get bus numbers assigned (may be wrong)
[...]

If the GPU(s) are found assign the correct memory ranges to parent bridge(s).

For bridge I/O range I guess use can just use some fixed numbers because we aren't using it.

I still think the bus numbers must be already assigned in the ProcesssControllerOverride hook, otherwise the GPU would not even be accessible. So I need to enumerate all PCIe busses, looking for my GPU, but without assigning the bus numbers .... (like a read-only enumeration).

Would it be ok to do the enumeration inside the ProcesssControllerOverride hook , right where I need to check and set BAR size ?

Where can I find the AllocateMemorySpace() function please ? Edk2 appears to have gDS->AllocateMemorySpace() and CoreAllocateMemorySpace(). Or do I use AllocateMemorySpace() directly from <Pi/PiDxeCish.h> header ?

For the I/O range, different people report different base address for this range on their computers, although the range so far is always empty. So I have to wonder why is the range not always 00-00 in this case, and how is it allocated ... ?

Another issue is it looks for some hardware the remaining low-order bit, after truncating the least-signifiant BYTE in the IO address limit, is also reassigned, and I need to set the right value for the bit in this case ... how would I even know ?

I read in the UEFI specification that I/O port ranges must come from functions from the ACPI tables... ?

So can I use AllocateMemorySpace() and AllocateIoSpace() in the hook as well, even if the UEFI specification calls them pre-UEFI services ?

[xCuri0]

@terminatorul

Would it be ok to do the enumeration inside the ProcesssControllerOverride hook , right where I need to check and set BAR size ?

yeah I think it should be fine, just make sure it only gets run once with a global variable. I recommend testing this in a VM first

Where can I find the AllocateMemorySpace() function please ? Edk2 appears to have gDS->AllocateMemorySpace() and CoreAllocateMemorySpace(). Or do I use AllocateMemorySpace() directly from <Pi/PiDxeCish.h> header ?

It's gDS->, I recommend searching for it in the Pci modules of EDK2 if you need examples of it with MMIO.

For the I/O range, different people report different base address for this range on their computers, although the range so far is always empty. So I have to wonder why is the range not always 00-00 in this case, and how is it allocated ... ?
Another issue is it looks for some hardware the remaining low-order bit, after truncating the least-signifiant BYTE in the IO address limit, is also reassigned, and I need to set the right value for the bit in this case ... how would I even know ?

BIOS assigns I/O port ranges for stuff that use it, I have no idea how it's range is decided tho but it's not of any relevance to this since we aren't using any PCI port I/O.

Actually looking at the code again it appears that only the I/O base is being set while the limit is untouched, which shows that it's not required. If what Xelafic said about requiring I/O base to be set is true then just set it to some fixed value like for example E0 E0 taken from my system it won't matter what it is (besides being invalid I guess) because we aren't using it.

So can I use AllocateMemorySpace() and AllocateIoSpace() in the hook as well, even if the UEFI specification calls them pre-UEFI services

Yes, the PCI module we are hooking is full of calls to these functions. There's no need to use I/O space for this btw.

Also btw you can easily calculate the base/limit for PCI bridge using the allocated BAR address and size.
And you have to add DxeServicesTableLib to the .inf configuration file and include Library/DxeServicesTableLib.h to use gDS->

[terminatorul]

@xCuri0

For the README page I'd say remove the ReBarUEFI stuff and instead replace it with a link saying to first modify your BIOS with ReBarUEFI if it doesn't natively support Resizable BAR.

I would like separate NvStrapsReBar code from ReBarUEFI code, but I still need NvStrapsReBar functions to execute before ReBarUEFI. So the GPU has a chance to expose supported BAR sizes on the PCI extended configuration registers.

Do you know how to achive that ? Is it possible to build NvStrapsReBar as some sort of library (package) for UEFI code, and would you be interested to update ReBarUEFI to check if the library is present, and call it at specific points inside the ReBarUEFI handler ? The other way around works too, if ReBarUEFI could expose hooks that NvStrapsReBar could use ....

But then I hope a bug in NvStrapsReBar will not affect ReBarUEFI ...

[xCuri0]

@terminatorul

I would like separate NvStrapsReBar code from ReBarUEFI code, but I still need NvStrapsReBar functions to execute before ReBarUEFI. So the GPU has a chance to expose supported BAR sizes on the PCI extended configuration registers.
Do you know how to achive that ? Is it possible to build NvStrapsReBar as some sort of library (package) for UEFI code, and would you be interested to update ReBarUEFI to check if the library is present, and call it at specific points inside the ReBarUEFI handler ? The other way around works too, if ReBarUEFI could expose hooks that NvStrapsReBar could use ....

I think you should let NvStrapsReBar handle Resizable BAR for NVIDIA GPUs by itself, it will keep things simple. You can remove all the rebar code except for this (I changed to not need rebarState) after you set the strap within the scope of the function so it only gets run on NVIDIA GPUs that got their BAR size changed.

	epos = pciFindExtCapability(NVpciAddress, PCI_EXT_CAP_ID_REBAR);
    if (epos)
    {
        for (UINT8 bar = 0u; bar < PCI_STD_NUM_BARS; bar++)
        {
            UINT32 rBarS = pciRebarGetPossibleSizes(NVpciAddress, epos, bar);
            if (!rBarS)
                continue;
            // start with size from fls
            for (UINT8 n = fls(rBarS); n > 0; n--) {
                // check if size is supported
                if (rBarS & (1 << n)) {
                    pciRebarSetSize(NVpciAddress, epos, bar, n);
                    break;
                }
            }
        }
    }

Remove the vid and did arguments from the pciRebarGetPossibleSizes function, then remove the 5600XT override because it's not needed.

Also is the following needed ?

    if (NvStraps_CheckBARSizeListAdjust(pciAddress, vid, did, bar))
        cap = NvStraps_AdjustBARSizeList(pciAddress, vid, did, bar, cap);

Because doesn't the GPU update it's supported BAR size list after setting strap ?

[terminatorul]

Looking at what edk2 does, I have to find available memory myself (like by reading the memory map from UEFI), add it, and immediately allocate it so it will not be reused. As if the standard allocation function does nothing for my use case... it must be meant for something else

[xCuri0]

@terminatorul

Iassume I need to hook PciEnumerationEnd to check resulting BAR size

I meant write the address given from AllocateMemorySpace to some NVRAM variable, should be able to figure out if it's giving a proper one or not then. You did mention the Status variable doesn't show anything wrong so this is the last thing that can be checked

Looking at what edk2 does, I have to find available memory myself (like by reading the memory map from UEFI), add it, and immediately allocate it so it will not be reused. As if the standard allocation function does nothing for my use case... it must be meant for something else

This is exactly what AllocateMemorySpace does, if it's not working then the UEFI memory map also won't be of any use.

There are probably other ways to get a memory map like BIOS e820 function but those are very complicated to use and should only be tried if UEFI ones can't be made working no matter what.

[terminatorul]

So what does the AddMemory() function do ?

[xCuri0]

@terminatorul AddMemorySpace is used internally by the UEFI firmware to tell what regions are MMIO/memory/IO

Btw one of the UEFIPatches for LGA1155 systems involves correcting the AddMemorySpace call to use the full physical address range instead of a fixed 8GB value

[terminatorul]

Also is the following needed ?

if (NvStraps_CheckBARSizeListAdjust(pciAddress, vid, did, bar))
cap = NvStraps_AdjustBARSizeList(pciAddress, vid, did, bar, cap);

Because doesn't the GPU update it's supported BAR size list after setting strap ?

Tested again and now it works without the override. Also added new value to Status code from the UEFI DXE driver, to tell if the GPU does not expose the requested BAR size on PCI, after configuration is complete.

So now I removed the BAR size override shown above, but I made sure it can still be applied if user inputs new value 64 to the reBarState variable (special value used to change BAR size on PCI for the selected GPU only) ...

[xCuri0]

@terminatorul

Also you could try removing all the Resizable BAR code and leaving it to the BIOS to resize it (ReBarUEFI on older systems). Might solve the issue @UnknownGuyzs had.

If it doesn't work then maybe need to call the original PreprocessController function after setting straps instead of before.

[UnknownGuyzs]

So which option should I try once I rebuild?

[terminatorul]

First 65. Then 64.

[UnknownGuyzs]

Yeah, I did. Still doesn't work.

[terminatorul]

@UnknownGuyzs
If you're still onto this, @Cancretto raised a good question: is the CPU-Z report that you sent me generated with "Above 4G Decoding" on, and with CSM off ? In general, was it generated with the same settings (in UEFI setup) that you run daily ?

[UnknownGuyzs]

It was indeed generated with the bios settings I use daily.

[Maidvelia]

So I have ResizableBAR support in all aspects except Turing.

OS: Windows 11 Home 23H2
CPU: AMD Ryzen 7 3800XT
Mobo: MSI X470 GAMING PLUS MAX MS-7B79 PCI-e 3.0 w/ ResizableBAR and Above 4G in BIOS

GPU: MSI ARMOR NVIDIA RTX 2070 8GB

I just need to turn ResizableBAR on for Turing.

This PC is expensive, and I'm not about to brick my BIOS. Is there a tutorial that poses minimal risk and is easy to follow? (all on one page/all in one video) Would be very helpful right now.

[UnidentifiedTag]

@terminatorul or @xCuri0 could we have this answered?

[xCuri0]

@Maidvelia if you want something easy to follow wait till the module is finished development, currently it requires you configure and compile it yourself

[Maidvelia]

Sounds good. I'll keep checking this periodically. Thank you!

[Cancretto]

@terminatorul Hi, i'm trying to enable res bar on my 2080 Ti, though i'm unsure on what values i should set in LocalPciGPU.h.

I know my device and vendor ids by looking at the cpuz file report though i'm not sure about the pci registers, target bar address, mem base limit, io base limit and similar.

I'd like to try flashing the module on my board but can't figure out the values of that header.

If you want to take a look i attached my cpuz report file to the post

btw we all should thank you and the other guys for your awesome work :))

report.txt

[xCuri0]

I think the setup variable might change itself but I'm not sure about that.

Tbh I'd recommend against this since u will have to set it back every time changing bios setting and Windows doesn't really like when you change hardware configuration so often and might have to reinstall drivers or something.

[xCuri0]

I think I'll wait for version 0.2 before locking this, don't need another thread full of people asking how to compile it

[Cancretto]

@terminatorul Yeah, you're right i haven't thought that it can be an issue for people trying this on a laptop.

Also wanted to ask you or if anyone knows, do we have to force reBAR on the global nv profile or is it possible to enable it on a per application basis?

Thanks :)

[terminatorul]

Per-application settings in the Profile Inspector will override the global settings. But only if your game / application is listed there and has a profile in the Profile Inspector.

Beware the settings in the Profile Inspector only affect the values reported to the game by the NVIDIA GeForce driver for Windows. The values reported by the PCI bus (in Device Manager) are not affected.

[Cancretto]

@terminatorul Hey, upon further testing i can say that i was right with my theory, i can safely restore my UEFI's settings back and forth by first disabling the target BAR size and clearing the PCI BARs from NvStrapsReBar.exe, then if i disable 4G decoding or clear CMOS everything works fine, no PCI slot required, it disables fine.

[Black6spdZ]

Something helpful may be for someone to write a script to parse the CPU-Z txt file and output the needed info or even update the LocalPciGPU.h automatically. Also, a quick and easy build environment can be setup with gitpod.io/login and the only missing package is NASM which can be installed "sudo apt update && sudo apt install nasm"

[terminatorul]

Yes, a script would help, and other users asked for it before in this thread (@UnidentifiedTag). Back then I chose to focus on more difficult parts instead of the script, and then just could not get any progress untill now ( I mean the target BAR size is no longer hard-coded, but that was it...).

Would you be willing and able to write such a python on PowerShell script (or other language), maybe with regular expressions, that can search the .txt report for a PCI device of type "VGA Controller", and with a PCI Vendor ID of 0x10DE (for NVIDIA) ?

[UnidentifiedTag]

This project is getting more light on it, I’m noticing a up trend in incoming users. Comments on my video like why is this an hour long 😅

a script would definitely bring down that time, I remember there being an endless supply of script kids. Where have they gone 😂

[Black6spdZ]

they grew up and started coding Raku

[terminatorul]

I'd love to see Raku getting more popular

[Black6spdZ]

is it possible to build a hybrid ReBarUEFI / NVStrapsReBar by setting up the windows executable to enter the required cpu-z PCI information for Turing GPUs to pass to the EFI driver like the BAR Size?

[CptSchwifty]

got no linux sorry.
you got me wrong. i told you the regex so you could update/use it for the exe.

isnt ffs being equal bad? i thought the driver had it hardcoded as well, which is why we use the config file?

[terminatorul]

@CptSchwifty

[...] could you try making the files and provide them to me?

Here, I built both the driver and the executable FOR YOUR LocalPciGpu.h. I had to fix the regexp to accept both space or dash [ -] characters after the word "PCI"

NvStrapsReBar-CptSchwifty.zip

[CptSchwifty]

i tried.. sadly not working as well..
https://pastebin.com/xHBE5hAL
does this help you?
"Name: NvStrapsReBar
GUID: e3ee4a27-e2a2-4435-bba3-184ccad935a8
Attributes: (0)
"
im currently downloading linux and will add to this.
i'm very grateful for you help, just to let you know.

[CptSchwifty]

dont ask me why

its working now..
after modding the bios using mmtool and flashing that one it worked.
thank you so much!

[LawandaSRT]

Hello. Can you please provide instructions on how you turned on Re-BAR?

[Sid127]

would like some help with filling out my header as well, here's how far I've gotten

#if !defined(NV_STRAPS_REBAR_LOCAL_PCI_GPU_H)
#define NV_STRAPS_REBAR_LOCAL_PCI_GPU_H

// Hard-coded values that must match the local system

#include <stdint.h>

// Update these values to match the output from CPU-Z .txt report
// for your GPU and for the associated PCI-to-PCI bridge

#define TARGET_GPU_PCI_VENDOR_ID        0x10DEu
#define TARGET_GPU_PCI_DEVICE_ID        0x2191u

#define TARGET_GPU_PCI_BUS              0x01u
#define TARGET_GPU_PCI_DEVICE           0x00u
#define TARGET_GPU_PCI_FUNCTION         0x00u

// PCIe config register offset 0x10
#define TARGET_GPU_BAR0_ADDRESS         UINT32_C(0xB300'0000)               // Should fall within memory range mapped by the bridge

// Secondary bus of the bridge must match the GPU bus
// Check the output form CPU-Z .txt report

#define TARGET_BRIDGE_PCI_VENDOR_ID     0x8086u
#define TARGET_BRIDGE_PCI_DEVICE_ID     0x1901u

#define TARGET_BRIDGE_PCI_BUS           0x00u
#define TARGET_BRIDGE_PCI_DEVICE        0x01u
#define TARGET_BRIDGE_PCI_FUNCTION      0x00u

// Memory range and I/O port range (base + limit) mapped to bridge
// from CPU-Z .txt report of the bridge and GPU

// PCIe config register offset 0x20
#define TARGET_BRIDGE_MEM_BASE_LIMIT  UINT32_C(0xB400'B300)                 // Should cover the GPU BAR0

// PCIe config register offset 0x1C
#define TARGET_BRIDGE_IO_BASE_LIMIT   0x8181u

static_assert((TARGET_GPU_BAR0_ADDRESS & UINT32_C(0x0000'0001)) == 0u, "Unexpected GPU BAR0 in the I/O port range");
static_assert((TARGET_GPU_BAR0_ADDRESS & UINT32_C(0x0000'0006)) == 0u, "Unexpected GPU BAR0 in the 64-bit memory address space");

#endif          // !defined(NV_STRAPS_REBAR_LOCAL_PCI_GPU_H)

and I've attached my cpu-z report
HBCD_PE.txt

This is a 1660Ti laptop with an Insyde BIOS and 6 gigs of VRAM

[Sid127]

Unfortunately on linux I still have a max bar of 256MB, despite an 8gig region being allotted

[Sid127]

update: tried Pop!_OS' live ISO, and I correctly have a resized BAR there. Meaning this is an Arch Linux specific thing

[Sid127]

update 2: it was either my hugepages related config or pci power management related stuff that was breaking rebar on linux, it's been sorted

thank you very much for your time!

[LawandaSRT]

Hello. Can you please provide instructions on how you turned on Re-BAR?

[terminatorul]

If you have an NVIDIA card from the GTX 1660 and RTX 2000 line, the instructions are here NvStrapsReBar.
If you don't have an NVIDIA card, and you need to enable ReBAR on an old motherboard, use the instructions here: ReBarUEFI.

The instructions are not easy, so make sure you understand them. User @UnidentifiedTag made a video tutorial about them, and it ended up 1-hour long. Please go through all the instructions and give more details for what steps do not work or are not clear.

[UnidentifiedTag]

https://youtu.be/ldnC18u8FI0

@terminatorul and @xCuri0 you are welcome to share the video, i hope this helps more people get onboard

[xCuri0]

@terminatorul often resetting CMOS will reset the date to the year a board was first produced, for example my ivy bridge board will become 2012.

using <2023 as the reset date is an issue because recent boards will reset to 2023/24

[UnidentifiedTag]

@xCuri0 is correct, that’s the behaviour I’ve noticed when removing the cmos battery and performing a cmos reset over 10 odd boards I’ve worked with dating back to 1995!

[terminatorul]

@xCurio
@Maidvelia

I saw someone say their board didn't reset time, when just using clear CMOS headers/button. They had to physically remove the battery so I added that to ReBarUEFI wiki.

I think this might explain everything ... it makes sense. So hopefully I don't need to panic and revert the project version any more.

I didn't know this about the Clear CMOS button so I might have given some bad advice about it here in this discussion, or elsewhere... :(

@Cancretto can you still tell which of these did you use, before you had to change GPU slot ?

• the Clear CMOS button
• shorting the pins on the Clear CMOS header
• removing the battery from the motherboard

[Maidvelia]

My clear CMOS button reverts my entire BIOS to defaults the last few times I've used it. As long as I'm not doing any hardware changes I'm sure my setup is safe?

I also used the 65 / 0 option for PCI.

[terminatorul]

You need to check the button also resets the current date (which is the key to recovering the board), but in your case with option 0 you are safe anyway

[Black6spdZ]

no one given any thought to setting up a build environment on gitpod.io? a single script could be used to setup all missing dependencies and paths

[Sid127]

my fork has github actions enabled

[Maidvelia]

@terminatorul Did everything in @UnidentifiedTag 's video + the readme's, still unable to enable ReBAR.

Here's what I did:

I had 2 entires like @vigilant-hylian

#define TARGET_BRIDGE_IO_BASE_LIMIT 127 (forgot to capture this value)

INTERCEPTOR.txt

(INTERCEPTOR.txt is my CPU-Z report)

EDIT: Have tried all 3 methods of LocalPciGPU.h values (decimal, reversed hexadecimal, and @xCuri0's values that are supposed to work for 99% of systems), still not working.

[terminatorul]

@terminatorul What information do you need to record my configuration as working?

Motherboard: MSI X470 GAMING PLUS MAX
CPU: AMD Ryzen 7 3800XT
GPU: MSI ARMOR NVIDIA GeForce RTX 2070 8G

• GPU chipset or GPU-Z screenshot with the main page (GPU)
• NVIDIA driver version
• output from the following command: nvidia-smi -q -d memory

[Maidvelia]

@terminatorul What information do you need to record my configuration as working?
Motherboard: MSI X470 GAMING PLUS MAX
CPU: AMD Ryzen 7 3800XT
GPU: MSI ARMOR NVIDIA GeForce RTX 2070 8G

• GPU chipset or GPU-Z screenshot with the main page (GPU)
• NVIDIA driver version
• output from the following command: nvidia-smi -q -d memory

Sorry to bug you again despite getting a full BAR size to work, but I'm getting EFI Error using the 2^x settings. I'd like to drop to 4GB to have buffer (I stream and video edit, want some VRAM left over for that).

Is there something I can do to fix this?

[terminatorul]

When you follow the video, you are using an older version of NvStrapsReBar (v0.1). You should build the latest version instead, and re-flash your motherboard.

The process is the same as in the video, but you skip the command that reads git checkout v0.1. Also for the latest version, the target BAR size option was removed from the header file LocalPciGpu.h, so you don't have to set it there. Instead, you have to set BAR size when you run NvStrapsReBar.exe.

But you don't have to restart the entire setup process. Instead, you can manually delete the LocalPciGpu.h file, then run the commands:

git -C "%UserProfile%\edk2\NvStrapsReBar" checkout master
git -C "%UserProfile%\edk2\NvStrapsReBar" pull

and re-populate the new LocalPciGpu.h file. Then continue with the build commands for DXE driver (NvStrapsReBar.ffs) and for the executable (NvStrapsReBar.exe), from github.

Remember to set "Above 4G decoding" again (and disable CSM), after your re-flash the motherboard.

[Maidvelia]

Thank you so much, I'll try this out!

[terminatorul]

@Cancretto found some issues in the latest version, so now I have reverted the github project to the same v0.1 version, until I can properly check things out

[Jonny5555555]

Hello People just wanted to double check, I run a z690 mobo with a ole 1080ti I take it ReBAR with Pascal is not gonna happen ? :D

[terminatorul]

Ah yes that’s right it only works on the gtx 1660 super and up right ?

Yes.

Well technically it would work on pre-Pascal cards as well (Xelafic used a Keppler card)... So it's just Pascal that does not work. And not because of the UEFI driver or PCI configuration, but because of the NVIDIA GeForce driver for Windows

[UnidentifiedTag]

What about amd, do we know which is supposed and which isn’t?

[terminatorul]

Don't know about AMD cards. My oldest card from AMD is a Radeon RX 580 and it has ReBAR support from the manufacturer (unofficial, need to manually change a registry key, but still, AMD provides the option). So ReBAR support for old AMD cards is simply not a problem most of the time.

Also the list of working motherboards in this project would be usefull for finding the oldest supported card...

[xCuri0]

@terminatorul the newest AMD cards that don't support rebar are the Rx 2xx from 2013 and even some AIBs on those support rebar. No idea how the driver support is tho.

AMD also exposes a similar straps interface to set BAR size ask @5l1v3r1 about it.

I don't think it's worth doing though because I hardly see anyone using AMD GPUs older than Polaris (2016 and supports it).

[xCuri0]

For older NVIDIA cards I can't imagine anything older than Maxwell (9xx 2014) benefiting from it. Don't think anyone has tried Maxwell tho.

Really sucks Pascal can't use it, they definitely would benefit alot like.

[xCuri0]

Locking this discussion, for any further discussion go to the NvStrapsRebar repo https://github.com/terminatorul/NvStrapsReBar

The latest version does not require any compiling like you may have seen in outdated guides/discussion