This is my post. Basically, I needed P2P between four R9700s for tensor parallel serving, which means a 32 GB BAR on each card. With all six GPUs in, the BIOS couldn't map them, and Linux sized the switch's window too small, so only two cards got their BARs. The module in the post works around that. With P2P off, prefill was about half as fast.
Odd. I'm using 7 GPUs (8 if you include the onboard AST2500) and haven't had this issue, but these are directly connected to the mainboard
I did disable CSM which used base memory resources needed by REBAR, but aside from that I've 6x Tesla P4s, 1x 750TI, and the AST2500 all happily working together.
My board is a Cloud Star C612 (6x DDR4 variant), 2 PCIE ports bifurcated (so 4 GPUs in total, the others sit on dedicated ports), with a Xeon E5 2696v3 and TBU.
That is because you are using a server Xeon E5 Motherboard at a time when Teslas and Large BARs were becoming a thing whereas he is using a consumer ASUS TUF GAMING X570-PLUS with a PCIe Switch hooked in that makes the MMIO resources from a single PCIe Port suddently becoming gigantic. Unlikely to have ever been tweaked, much less tested, by vendor for that use case.
I am literally also using an X570 ASUS TUF motherboard and run into resize BAR issues with Linux able to deal with and the bios -- which seems RGB fan optimised rather than memory bandwidth optimised -- unable to directly help.
The gpu_unlock is a bit hacky (but I have many GPUs that the unlock won't work on and that will cause issues), and there are two BIOS settings that matter: above 4GB decode enabled and resizeable bar enabled. That's it.
This affects me and I'm on an older Epyc server. It had >4G decode and resizable BAR already, and IOMMU disabled (not running any VMs). I'm hoping that with the aforementioned workaround, I'll finally be able to get P2P DMA working for my R9700s.
>> I think a lot of folks use open air aluminum extrusion frames that were once meant for mining.
That is what i am using. It works well enough, and is much much quieter than the server rackmount case I started with, but mining rigs do not fit AI very well. The full PCI risers need more space than the USB-PCI things used in mining rigs. So, anyone reading this, make sure to buy a frame that is a couple inches taller than the standard squat mining rigs.
Before anyone asks, my rig is 3x5060ti(16gb) for 48GB VRAM, then 64GB DDR4 and a 1000w power supply.
I alternate between Ollama for LLMs and ComfyUI for image/video creation.
MCIO 8i cables are easier to deal with than the big ribbon riser cables, and also help with airflow. I'm glad I was forced to switch to them to work with my switch, even though they're pricey.
Also, insert obligatory "friends don't let friends use Ollama" speech here.
I'm also using a mining frame. Not really happy with it because it does very little to secure my GPUs, so they aren't properly spaced and are prone to falling. I wish there were decent case options that I could properly rackmount.
This is amazing, I'm curious about two things
1. How does one find motherboards that are compatible with this technique
2. What's your take on TinyBox, as they have been pushing and playing with AMD early with limitations of getting things like this running?
Thank you for posting this! I've been fighting with this exact issue for days and didn't know what it was until now. Going to try your module, and what luck that I also run the PVE kernel!
These kind of things should be fun to test in Coreboot. When Dasharo was released for the MSI PRO Z690-A DDR4 more than four years ago, I had issues with two Radeon 5600XT installed that was fixed after developer tuned some MMIO related parameters: https://github.com/Dasharo/dasharo-issues/issues/245
How many PCIe Switches you could chain and how many cards with absurd MMIO requeriments you can get working would be an interesing experiment.
In the last decade or so I have seen plenty of Firmware MMIO related issues. I remember certain SR-IOV supporting NICs (Usually Intel ones like i350 and other enterprise parts) not being able to reserve enough MMIO to create all the supported SR-IOV Virtual Functions in consumer Motherboards, in many cases you could get SR-IOV working but were limited to create like 4 VFs even if hardware supported 16 because that is what you could fit. This was essentially all a Firmware issue, not Hardware, but few people has the required skills to do any BIOS modding to fix rather convoluted things like MMIO allocation. Basically, you're trying to workaround Firmware issues elsewhere.
It feels ridiculous that a person has to go to these kind of lengths to do what is in the end a trivial thing: assigning little pieces of a gigantic virtual address space to specific peripherals.
The only thing that's complex here is having to influence a bunch of hidden software through three layers of indirection. The software is all standing in the way of a person just writing some values to registers.
Another important benefit is that transfers between cards flow across the switch and bypass the CPU, that ain't nothing. At least for the AMD cards. Nvidia firmware refuses nach
The bottleneck with AI and hardware is really pushing things.
Even with six capable GPUs that have plenty of VRAM with enough PCIe bandwidth, you then discover that memory address allocation is preventing the hardware from doing what it's physically capable of.
Most CPUs can address 48 bits of physical memory (hundreds of terabytes), why is that even a problem ? Is it just some hardware limitation somewhere else or badly written firmware ?
There was never a concerted effort in the PC ecosystem to force all peripherals and firmware to drop 32-bit compatibility. At some point after a decade or two of x86-64 CPUs, we should have started making GPUs that simply exposed all of their VRAM to be mapped by the host by default, instead of retaining compatibility hacks like only mapping 256MB by default. Resizable BARs should have happened at least a decade earlier, and should have been phased out by now in favor of just always mapping all the VRAM.
Even as late as the PCIe 2.0 era, it was very plausible to still have peripherals that couldn't be relied upon to support 64-bit addressing (eg. a motherboard I had back then had Firewire provided by a PCI controller behind a PCIe bridge). I'd say it was the PCIe 3.0 era when consumer PC hardware should have started making 32-bit addressing support an explicit non-goal; motherboards and peripherals should have both expected to be able to use a much wider address space to allow a GPU's entire VRAM capacity to be mapped by default. That would have been starting around the GeForce 600 and Radeon HD 7000 series 28nm GPUs circa 2012. At the very least, there should have been a push to start using Resizable BAR around then, not eight years later as actually happened. PCIe has theoretically supported 64-bit addressing from the beginning, but consumer hardware didn't start trying to actually take advantage of that in a real way until 2020.
They DID do that, but not for consumer GPUs, only pure compute ones. 2013 Kepler based Tesla K40 had 12 GiB VRAM and a 16 GiB BAR1 requeriment (Also called Large BAR), and was stated in the datasheet itself: https://hpc.llnl.gov/sites/default/files/Tesla-K40-Board-Spe...
...and is fun to read issues from a decade ago about failing to get those working in desktop systems. Makes perfect sense why they never did that with no control over the Firmware ecosystem.
IIRC, there was at least one particular Tesla or Quadro model where nVidia was providing tools to flash its VBIOS because there were two versions, one with standard 256 MiB BAR and other with Large BAR, and you needed to reflash VBIOS with the version you wanted to use. This was of course pre ReBAR. ReBAR pretty much solved the issue because if it doesn't support ReBAR you get standard 256 MiB BAR compatibility mode, and if it does, then it is assumed it works. Large BAR GPUs tended to either made the system to not POST or failed to be initialized in anything not intended for it.
The limits weren't always exactly 32 bits, but it was common to find that the usable address space supported by a motherboard or its chipset was at most one bit wider than necessary for fully populating its DRAM slots; ie. there wasn't physical address space to spare for a significant fraction of it to be assigned to peripherals. That meant there was continued pressure for peripherals to map only increasingly small portions of their memory into the host address space, even though the CPUs themselves usually supported physical addressing that was comfortably spacious enough.
My first desktop with DDR3 had four DIMM slots supporting 8GB modules, but the total address space was limited to 32GB including all address space used by peripherals, and the peripherals were all mapped below 4GB. So it was effectively a 35-bit system, even though Intel never shipped an x86-64 CPU with less than 36-bit physical address space and AMD never shipped less than 40-bit. Significant parts of the PC ecosystem were being stingy with address lines.
Yeah, it's the firmware. It's an AM4 board, and the BIOS only gives PCI devices the space from the top of RAM up to 512 GB. Each 32 GB BAR has to start on a 32 GB boundary, and its small doorbell BAR spills into the next slot, so each card needs 64 GB. Six won't fit. So I turn Resizable BAR off in the BIOS and resize the four R9700s from Linux, where I hit the kernel bug.
The problem is that with P2P every card will map every other card into its address space. So you're not mapping 128G but likely 512! It depends a bit on what software you use, some software is smart about this and uses a smaller sliding window if it can't use all of it at once and some software is really rigid.
Please let us know if AI patches to your firmware does anything other than brick your hardware. I'm willing to let an LLM handle a lot of things, but the firmware I need to boot is not among them.
This is my post. Basically, I needed P2P between four R9700s for tensor parallel serving, which means a 32 GB BAR on each card. With all six GPUs in, the BIOS couldn't map them, and Linux sized the switch's window too small, so only two cards got their BARs. The module in the post works around that. With P2P off, prefill was about half as fast.
Odd. I'm using 7 GPUs (8 if you include the onboard AST2500) and haven't had this issue, but these are directly connected to the mainboard
I did disable CSM which used base memory resources needed by REBAR, but aside from that I've 6x Tesla P4s, 1x 750TI, and the AST2500 all happily working together.
My board is a Cloud Star C612 (6x DDR4 variant), 2 PCIE ports bifurcated (so 4 GPUs in total, the others sit on dedicated ports), with a Xeon E5 2696v3 and TBU.
That is because you are using a server Xeon E5 Motherboard at a time when Teslas and Large BARs were becoming a thing whereas he is using a consumer ASUS TUF GAMING X570-PLUS with a PCIe Switch hooked in that makes the MMIO resources from a single PCIe Port suddently becoming gigantic. Unlikely to have ever been tweaked, much less tested, by vendor for that use case.
I am literally also using an X570 ASUS TUF motherboard and run into resize BAR issues with Linux able to deal with and the bios -- which seems RGB fan optimised rather than memory bandwidth optimised -- unable to directly help.
Cool!
Interesting. This sort of thing is highly bios specific, I'm running 240G of VRAM on a single box using a very simple config:
GRUB_CMDLINE_LINUX_DEFAULT="pcie_aspm=off amd_iommu=on iommu=pt pci=realloc,firmware noapic nomodeset pci=nommconf gpu_unlock.targets=0000:66:00.0,0000:69:00.0,0000:6d:00.0,0000:6e:00.0"
The gpu_unlock is a bit hacky (but I have many GPUs that the unlock won't work on and that will cause issues), and there are two BIOS settings that matter: above 4GB decode enabled and resizeable bar enabled. That's it.
This affects me and I'm on an older Epyc server. It had >4G decode and resizable BAR already, and IOMMU disabled (not running any VMs). I'm hoping that with the aforementioned workaround, I'll finally be able to get P2P DMA working for my R9700s.
Good luck, if it doesn't work feel free to mail me.
sidebar: Why don't your posts include pics of your rig?
Asking because I really want to see your kit.
I'll get a post together with some photos of the rat's nest soon. It's definitely not fancy. :-)
I think a lot of folks use open air aluminum extrusion frames that were once meant for mining.
>> I think a lot of folks use open air aluminum extrusion frames that were once meant for mining.
That is what i am using. It works well enough, and is much much quieter than the server rackmount case I started with, but mining rigs do not fit AI very well. The full PCI risers need more space than the USB-PCI things used in mining rigs. So, anyone reading this, make sure to buy a frame that is a couple inches taller than the standard squat mining rigs.
Before anyone asks, my rig is 3x5060ti(16gb) for 48GB VRAM, then 64GB DDR4 and a 1000w power supply.
I alternate between Ollama for LLMs and ComfyUI for image/video creation.
MCIO 8i cables are easier to deal with than the big ribbon riser cables, and also help with airflow. I'm glad I was forced to switch to them to work with my switch, even though they're pricey.
Also, insert obligatory "friends don't let friends use Ollama" speech here.
I'm also using a mining frame. Not really happy with it because it does very little to secure my GPUs, so they aren't properly spaced and are prone to falling. I wish there were decent case options that I could properly rackmount.
You into liquid cooling?
because the llm who wrote and illustrated it doesn't have a camera
This is amazing, I'm curious about two things 1. How does one find motherboards that are compatible with this technique 2. What's your take on TinyBox, as they have been pushing and playing with AMD early with limitations of getting things like this running?
Thank you for posting this! I've been fighting with this exact issue for days and didn't know what it was until now. Going to try your module, and what luck that I also run the PVE kernel!
What model/engine are you running? QFN/radiance? DSF?
These kind of things should be fun to test in Coreboot. When Dasharo was released for the MSI PRO Z690-A DDR4 more than four years ago, I had issues with two Radeon 5600XT installed that was fixed after developer tuned some MMIO related parameters: https://github.com/Dasharo/dasharo-issues/issues/245 How many PCIe Switches you could chain and how many cards with absurd MMIO requeriments you can get working would be an interesing experiment.
In the last decade or so I have seen plenty of Firmware MMIO related issues. I remember certain SR-IOV supporting NICs (Usually Intel ones like i350 and other enterprise parts) not being able to reserve enough MMIO to create all the supported SR-IOV Virtual Functions in consumer Motherboards, in many cases you could get SR-IOV working but were limited to create like 4 VFs even if hardware supported 16 because that is what you could fit. This was essentially all a Firmware issue, not Hardware, but few people has the required skills to do any BIOS modding to fix rather convoluted things like MMIO allocation. Basically, you're trying to workaround Firmware issues elsewhere.
The word or acronym BAR is mentioned 34 times but never defined.
thanks, fixed
It feels ridiculous that a person has to go to these kind of lengths to do what is in the end a trivial thing: assigning little pieces of a gigantic virtual address space to specific peripherals.
The only thing that's complex here is having to influence a bunch of hidden software through three layers of indirection. The software is all standing in the way of a person just writing some values to registers.
Feels nostalgic. Kind of like Linux in the 90s. :)
Another important benefit is that transfers between cards flow across the switch and bypass the CPU, that ain't nothing. At least for the AMD cards. Nvidia firmware refuses nach
The bottleneck with AI and hardware is really pushing things.
Even with six capable GPUs that have plenty of VRAM with enough PCIe bandwidth, you then discover that memory address allocation is preventing the hardware from doing what it's physically capable of.
What are you running on this? I have a very similar set up to you, a skylake beast 32GiB of DDR4 goodness and a monster 8GB RX570.
All jokes aside I would love to know more about your set up and what you do with it and what speeds you get token wise.
Most CPUs can address 48 bits of physical memory (hundreds of terabytes), why is that even a problem ? Is it just some hardware limitation somewhere else or badly written firmware ?
There was never a concerted effort in the PC ecosystem to force all peripherals and firmware to drop 32-bit compatibility. At some point after a decade or two of x86-64 CPUs, we should have started making GPUs that simply exposed all of their VRAM to be mapped by the host by default, instead of retaining compatibility hacks like only mapping 256MB by default. Resizable BARs should have happened at least a decade earlier, and should have been phased out by now in favor of just always mapping all the VRAM.
> we should have started making GPUs that simply exposed all of their VRAM to be mapped by the host by default
Believe this is what Intel GPUs already do.
> Resizable BARs should have happened at least a decade earlier
We have had "Above 4G Decoding" for years.
The 256MB BAR is a leftover from AGP GPUs. They are talking about when we moved to PCI Express GPUs.
Even as late as the PCIe 2.0 era, it was very plausible to still have peripherals that couldn't be relied upon to support 64-bit addressing (eg. a motherboard I had back then had Firewire provided by a PCI controller behind a PCIe bridge). I'd say it was the PCIe 3.0 era when consumer PC hardware should have started making 32-bit addressing support an explicit non-goal; motherboards and peripherals should have both expected to be able to use a much wider address space to allow a GPU's entire VRAM capacity to be mapped by default. That would have been starting around the GeForce 600 and Radeon HD 7000 series 28nm GPUs circa 2012. At the very least, there should have been a push to start using Resizable BAR around then, not eight years later as actually happened. PCIe has theoretically supported 64-bit addressing from the beginning, but consumer hardware didn't start trying to actually take advantage of that in a real way until 2020.
They DID do that, but not for consumer GPUs, only pure compute ones. 2013 Kepler based Tesla K40 had 12 GiB VRAM and a 16 GiB BAR1 requeriment (Also called Large BAR), and was stated in the datasheet itself: https://hpc.llnl.gov/sites/default/files/Tesla-K40-Board-Spe...
...and is fun to read issues from a decade ago about failing to get those working in desktop systems. Makes perfect sense why they never did that with no control over the Firmware ecosystem.
IIRC, there was at least one particular Tesla or Quadro model where nVidia was providing tools to flash its VBIOS because there were two versions, one with standard 256 MiB BAR and other with Large BAR, and you needed to reflash VBIOS with the version you wanted to use. This was of course pre ReBAR. ReBAR pretty much solved the issue because if it doesn't support ReBAR you get standard 256 MiB BAR compatibility mode, and if it does, then it is assumed it works. Large BAR GPUs tended to either made the system to not POST or failed to be initialized in anything not intended for it.
I don't think it's related to any of the 32 bit limits
The limits weren't always exactly 32 bits, but it was common to find that the usable address space supported by a motherboard or its chipset was at most one bit wider than necessary for fully populating its DRAM slots; ie. there wasn't physical address space to spare for a significant fraction of it to be assigned to peripherals. That meant there was continued pressure for peripherals to map only increasingly small portions of their memory into the host address space, even though the CPUs themselves usually supported physical addressing that was comfortably spacious enough.
My first desktop with DDR3 had four DIMM slots supporting 8GB modules, but the total address space was limited to 32GB including all address space used by peripherals, and the peripherals were all mapped below 4GB. So it was effectively a 35-bit system, even though Intel never shipped an x86-64 CPU with less than 36-bit physical address space and AMD never shipped less than 40-bit. Significant parts of the PC ecosystem were being stingy with address lines.
Yeah, it's the firmware. It's an AM4 board, and the BIOS only gives PCI devices the space from the top of RAM up to 512 GB. Each 32 GB BAR has to start on a 32 GB boundary, and its small doorbell BAR spills into the next slot, so each card needs 64 GB. Six won't fit. So I turn Resizable BAR off in the BIOS and resize the four R9700s from Linux, where I hit the kernel bug.
The problem is that with P2P every card will map every other card into its address space. So you're not mapping 128G but likely 512! It depends a bit on what software you use, some software is smart about this and uses a smaller sliding window if it can't use all of it at once and some software is really rigid.
"640kB ought to be enough for everyone" moment.
I wonder if AI could RE the BIOS enough to fix that limit
Please let us know if AI patches to your firmware does anything other than brick your hardware. I'm willing to let an LLM handle a lot of things, but the firmware I need to boot is not among them.