The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applications noted above eventually got their own flavors.
I think the Von Neumann bottleneck is mostly a feature. The fact that communication of information across distances is expensive should not be immediately assumed to mean that it is universally flawed to do this. You are paying for something when you use all those joules. I'd argue we are usually wasting our energy with regard to information communication (e.g., lighting up a network interface & copper because we couldn't be bothered to use SQLite), but other times this stuff is fundamentally required for practical solutions to exist.
I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.
Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.
Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.
Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.
Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.
Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM.
I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).
I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands?
I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM access ports and the PIM register state.
A lot of SIMD code can be on two array operands, and I expect this PIM approach only works well if both are stored locally in the same memory "local" to the PIM and where it can efficiently interleave at the natural addresisng and access widths. Too much random access or needing "remote" data sounds like the point where PIM fails and you need the elaborate memory IO controllers and caching subsystems of CPUs sitting on top of the distributed memory modules..?
> PIM would have a full blown IO controller and cache subsystem to fetch remote operands
Don't we already have those in mainstream computing in the form of dedicated silicon in DMA controllers? Programmed input–output performance is often low throughput, high jitter and uses a lot of CPU.
Yes, but a DMA controller sit out on the memory bus to do the same kind of work the CPU could be doing, controlling bus traffic between memory modules.
I think the whole idea of PIM is to be able to scale up and do work locally within the memory module, without the bottleneck of the system memory bus. This only works for embarrassingly parallel workloads that don't actually depend on the bisection bandwidth across the global memory.
If you start talking about PIM that is all interconnected, your application is back to being bound by the system memory bus. Maybe it's a new package design, but you're basically building yet another NUMA interconnect architecture, not a PIM architecture.
Data movement and local operations are still bottlenecked today on memory bandwidth. Butterfly primitives, sorting/fft/1D-convolution, the whole cub library, could be ported there and have great performance wins. But the pain of programming and maintaining code using this...
> run a sum over megabytes of data like for accounting
It's been many decades since the last time somebody ran a sum over megabytes of data for accounting and though "damn that's a bottleneck I need to optimize".
> Personally I'm a big fan of the "in-ram accelerator" idea especially for server space.
The operations this model supports are so extremely limited that you would be hard pressed to find applications where it's worth it.
> Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
I think we are essentially reinventing SSE, AVX & friends from first principles. This is already being done. Compare the speed of libjpegturbo to a non-vectorized implementation and you'll find a 2-4x difference in throughput.
If Infiniband does this for MPI on the network and realizes Sun's "The network is the computer" dream, I believe we can do this for other parts of the hardware, as well. Not only for AI, HPC will love this idea.
One thing this makes possible – accelerated memcpy. Of course, it is only worthwhile once the memory buffer to be copied is large enough. But the copy speed could be greatly increased if it could all be done inside the memory module. And of course, that only works if the source and destination buffers are physically on the same module. But, if your application allocates 1GB pages, the OS could attempt to ensure they are on the same memory module.
Accelerated memcpy is already offered by some Intel server systems (QuickData aka IOATDMA aka CBDMA aka SDMA), but it uses a memory-to-memory DMA engine on the CPU die, so the memory to be copied still has to travel back and forward between the CPU and the memory modules via the memory controller, even though using a DMA engine means it isn't consuming a CPU core. With this, the same process could be made much faster, bypassing the CPU entirely, provided the source and destination are on the same module.
Two decades ago it was a challenge to get people to see that what they were doing was preventing horizontal scaling. Today horizontal scaling is table stakes and people don't even always register that they are doing it. It's just how we do things, no thoughts.
PIM requires problems to be decomposed into horizontal scaling problems. Then what you should do with PIM is take a problem that used to be solved by 2 racks of computers and squeeze it down to less than half a rack by stuffing a bunch of these into a single box to do 8-10x as much work per box (and double the cluster size to offset Jevons' Paradox because it's so cheap now that you'll do 2x as much of it)
There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.
Maybe PIM will push this forward, but I still think we're doing something fundamentally wrong by not just embracing NUMA and trying to do something Sun tried decades ago, which is have number of cores share 4GB of semiprivate working
memory.
We've kind of half-assed it with DDR memory banks, but it mostly introduces mysterious slowdowns that are difficult to reason about and I think we would be better served I think by making a formal thing. Instead of introducing an L4 cache we could do this instead, and reduce the size of the L1-L3 caches, which shortens lookup time and thus latency.
For legacy apps, you could provide facilities for the OS to 'page' blocks in from main memory, but the speed would come from managing the workload imperatively, starting loads in the background before the data is actually needed, and dumps after it is last touched.
why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.
You could still do map-reduce operations, but for it to really fly, what you'd want is a side channel between the memory chips that allows the reduce to happen out of band from the front-side bus and the reduction to be sent to the CPU. And any workflow where you can stream the reduction to the CPU that would be even better for latency.
That really doesn't make sense because your system is already using memory that needs to be physically contiguous but mapped virtually. Also, would you say pinned memory is not part of the virtual memory system??
You're only restricted by the fragmentation of the system memory which is an issue yes, but it's dealt with in other ways.
I remember taking VLSI design as part of my Comp. Sci. degree at Bristol, UK c.1980, using the Conway & Mead book, and "Commingling of Processing and Memory" was mentioned even back then.
Obviously you (eventually) need your data where the compute is, especially in a non-von-Neumann architecture where moving data around isn't an option even if you were OK with the performance drop.
It seems kinda obvious that eventually AI will be implemented as low power dataflow custom chips integrating memory/state & compute, but who knows!
I saw them present a similar concept at Hot Chips in 2020 or 2021. It's still a cool idea, however people should remember that there are like 20 of these exotic accelerators designs pitched at trade shows every year that go nowhere.
Whilst processing in memory is clearly the future, I am unconvinced by this implementation.
Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2).
To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.
"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.
If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.
Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
With the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.
Exactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference.
If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.
People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.
At this point processing-in-memory has taken on the aura of fusion power.
OLED also has a deep history with a pretty famous opinions that it is impossible and a waste of time and money for companies to invest in the development. So does AI. From inception to product it is normal for decades to pass.
Compute needs change. AI needs are pretty unique in terms of scale and type of compute requirements to anything else so far.
> Whilst processing in memory is clearly the future
How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.
Maybe. The use cases have always been around FP arithmetic over arrays, because that's what is easy to parallelize. I saw a standalone systolic array box attached to a MicroVAX circa 1990. Lots of PIM approaches in the mid-90s, too, but mostly what survived from that era are shared memory NUMA multiprocessors and using GPUs for general purpose computing.
This is really just restating the assertion I'm asking about. Previous work also thought their use cases fit the needs.
Everything changes all the time in computing, so there are lots of differences. I'm not asking if things are different or claiming it won't work this time either. I'm asking what it is this time that makes it different / obvious. Not rhetorical, I'm interested if someone can actually explain. Preferably with trends and numbers that have $ and flops and picojoules and gates and bits in their units.
Not if you have duplicates of rows on the first matrix, which can be done very efficiently if you build specialized hardware. Then its all just forward in parallel.
you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.
what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.
that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.
Might as well just go whole hog and change the entire computer architecture, then. A lot of the arguments against this change boil down to computers and software code don’t work well with this today.
What I find amusing about moving compute to a RAM bank is it _almost_ resembles where we were with ISA-based extended RAM back in the 1980's. Some cards featured a CPU that took over the whole system and/or functioned like an upgrade. Others were a "computer on a card" that provided other features. I think this goes to show how cyclic tech can be. So, something like Samsung's invention here might have gained traction, as overcoming the slow PC ISA bus would have been a huge accelerator, kind of like where we are now.
So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
> So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
This is a temporary issue. JEDEC's LPDDR6-PIM is going to add defined commands for Processing-in-Memory operations. Once there are standardised commands, it will be possible for the CPU vendors to make the CPU cache aware of what is happening.
Of course, that doesn't solve it for this generation of the technology. But I think this generation is more of a demo for early adopters to gain experience with it. It will likely take a few years for all these issues to be solved, but there is no principled reason why they can't be.
It has been a prominent "next major shift" idea in computer architecture since the 90s, to deal with the memory wall. Eg David Patterson advocating it in the 1990 and 1997 articles. or CRAM [1].
Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot.
That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required.
Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place.
How would you do map-reduce across multiple DIMMs w/o extra reads/writes?
PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.
> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
Yes, but running out of RAM is impractical due to low memory bandwidth.
According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width:
> Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s.
And that's just for single 64-bit IC. So way faster and more power efficient.
You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly.
Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
We can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)
You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip.
614 GB/s x 96 = 58,944 GB/s.
Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast single digit gigabyte models. Without MoE you could reach 100 tokens per second with an 8B fp8 model on a single channel. With MoE you might break 500 tokens per second.
10 years ago hpe labs had something similarly including the operating system ! and the cto got ousted and a new ceo camea nd the whole new direction is always networking !
“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.”
So how big are these banks? If you can’t fit the weights of a layer into one bank then presumably you lose a lot of the speed gains.
The reason is quite simple. You can split the matrix along both dimensions so you just tile it into 64x64 or whatever fits into the bank and just fill it up. The biggest problem is load balancing the tiles across all banks for maximum parallelism.
For GEMM I believe there is no point in doing PIM, you are better off with a GPU or NPU.
This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.
There's been a ton of optimizations already, it hasn't remotely reduced demand even temporarily. More efficiency just makes the compute have even higher ROI per $ and watt spent.
With sufficient optimisation, there ought to be a tipping point beyond which local inference is good enough. And, sure, datacentre compute will still be needed for training but one of the biggest current uses will begin to taper off.
The question really is how soon we reach that tipping point, and whether it's before or after the current bubble runs out of steam for some other reason.
>there ought to be a tipping point beyond which local inference is good enough
There's no such ought really. Even at current levels you'd need like a 100x gain from here to approach current top proprietary models (probably a lot more for say Mythos or Mythos 2), and it's not like they are stoppng to improve. This is before we even account that you'd just be running 1 agent then, and not a swarm like you'd be able to in the cloud or that you can do only so much compression before you are losing out
It's not just inference, some things done in data centers like simulations, testing, are complementary to inference.
And in these types of hardware, the time between a successful prototype and a fully deployed product is pretty long. Maybe they count on that to know when to stop?
The better question is: what is the net gain for the overall system? If PIM reduces the net thermal load and power consumption of the system for the same workload, then it’s a win regardless of where the heat sinks end up. The customers Samsung has in mind for this today are not limited to commodity designs. They’re using novel designs with each new hardware generation, so moving heat sinks around is not a deal-breaker.
I thought DMR and Venice were supporting memory encryption by default. With the keys living on the CPU side and no standards for key sharing, I wonder how this will gain traction.
...and it's been long enough now, that I can say there was an effort to implement this on standard x86 memory controllers and have the existing string instructions do so, back in the days of SDR SDRAM, but the tradeoffs weren't (yet) in favour.
Feels like the most realistic/short-term way to make use of this would be to set up some barebones RTOS to run from CPU cache with the PIM memory being used for compute only and use the device as a network attached accelerator.
The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applications noted above eventually got their own flavors.
I think the Von Neumann bottleneck is mostly a feature. The fact that communication of information across distances is expensive should not be immediately assumed to mean that it is universally flawed to do this. You are paying for something when you use all those joules. I'd argue we are usually wasting our energy with regard to information communication (e.g., lighting up a network interface & copper because we couldn't be bothered to use SQLite), but other times this stuff is fundamentally required for practical solutions to exist.
I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.
Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.
Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.
Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.
Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.
Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM.
I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).
I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands?
I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM access ports and the PIM register state.
A lot of SIMD code can be on two array operands, and I expect this PIM approach only works well if both are stored locally in the same memory "local" to the PIM and where it can efficiently interleave at the natural addresisng and access widths. Too much random access or needing "remote" data sounds like the point where PIM fails and you need the elaborate memory IO controllers and caching subsystems of CPUs sitting on top of the distributed memory modules..?
> PIM would have a full blown IO controller and cache subsystem to fetch remote operands
Don't we already have those in mainstream computing in the form of dedicated silicon in DMA controllers? Programmed input–output performance is often low throughput, high jitter and uses a lot of CPU.
Yes, but a DMA controller sit out on the memory bus to do the same kind of work the CPU could be doing, controlling bus traffic between memory modules.
I think the whole idea of PIM is to be able to scale up and do work locally within the memory module, without the bottleneck of the system memory bus. This only works for embarrassingly parallel workloads that don't actually depend on the bisection bandwidth across the global memory.
If you start talking about PIM that is all interconnected, your application is back to being bound by the system memory bus. Maybe it's a new package design, but you're basically building yet another NUMA interconnect architecture, not a PIM architecture.
Data movement and local operations are still bottlenecked today on memory bandwidth. Butterfly primitives, sorting/fft/1D-convolution, the whole cub library, could be ported there and have great performance wins. But the pain of programming and maintaining code using this...
> run a sum over megabytes of data like for accounting
It's been many decades since the last time somebody ran a sum over megabytes of data for accounting and though "damn that's a bottleneck I need to optimize".
> Personally I'm a big fan of the "in-ram accelerator" idea especially for server space.
The operations this model supports are so extremely limited that you would be hard pressed to find applications where it's worth it.
> Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
I think we are essentially reinventing SSE, AVX & friends from first principles. This is already being done. Compare the speed of libjpegturbo to a non-vectorized implementation and you'll find a 2-4x difference in throughput.
If Infiniband does this for MPI on the network and realizes Sun's "The network is the computer" dream, I believe we can do this for other parts of the hardware, as well. Not only for AI, HPC will love this idea.
One thing this makes possible – accelerated memcpy. Of course, it is only worthwhile once the memory buffer to be copied is large enough. But the copy speed could be greatly increased if it could all be done inside the memory module. And of course, that only works if the source and destination buffers are physically on the same module. But, if your application allocates 1GB pages, the OS could attempt to ensure they are on the same memory module.
Accelerated memcpy is already offered by some Intel server systems (QuickData aka IOATDMA aka CBDMA aka SDMA), but it uses a memory-to-memory DMA engine on the CPU die, so the memory to be copied still has to travel back and forward between the CPU and the memory modules via the memory controller, even though using a DMA engine means it isn't consuming a CPU core. With this, the same process could be made much faster, bypassing the CPU entirely, provided the source and destination are on the same module.
Yes, and no.
Two decades ago it was a challenge to get people to see that what they were doing was preventing horizontal scaling. Today horizontal scaling is table stakes and people don't even always register that they are doing it. It's just how we do things, no thoughts.
PIM requires problems to be decomposed into horizontal scaling problems. Then what you should do with PIM is take a problem that used to be solved by 2 racks of computers and squeeze it down to less than half a rack by stuffing a bunch of these into a single box to do 8-10x as much work per box (and double the cluster size to offset Jevons' Paradox because it's so cheap now that you'll do 2x as much of it)
In a world where AI is writing all of the code, the difficulty of the task may no longer be a blocker.
Indeed. SIMD type code used to be too hard for me to write very often, but now it’s easy to crank out tons of it.
There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.
Maybe PIM will push this forward, but I still think we're doing something fundamentally wrong by not just embracing NUMA and trying to do something Sun tried decades ago, which is have number of cores share 4GB of semiprivate working memory.
We've kind of half-assed it with DDR memory banks, but it mostly introduces mysterious slowdowns that are difficult to reason about and I think we would be better served I think by making a formal thing. Instead of introducing an L4 cache we could do this instead, and reduce the size of the L1-L3 caches, which shortens lookup time and thus latency.
For legacy apps, you could provide facilities for the OS to 'page' blocks in from main memory, but the speed would come from managing the workload imperatively, starting loads in the background before the data is actually needed, and dumps after it is last touched.
why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.
You could still do map-reduce operations, but for it to really fly, what you'd want is a side channel between the memory chips that allows the reduce to happen out of band from the front-side bus and the reduction to be sent to the CPU. And any workflow where you can stream the reduction to the CPU that would be even better for latency.
That really doesn't make sense because your system is already using memory that needs to be physically contiguous but mapped virtually. Also, would you say pinned memory is not part of the virtual memory system??
You're only restricted by the fragmentation of the system memory which is an issue yes, but it's dealt with in other ways.
And the security folks will have tons of fun with this enormously leaky abstraction too.
Hey, just don't store your pagetables in PIM capable sections :)
Linux has been dealing with this kind of thing for over a decade now. Specialized SoCs love their memory carveouts.
I remember taking VLSI design as part of my Comp. Sci. degree at Bristol, UK c.1980, using the Conway & Mead book, and "Commingling of Processing and Memory" was mentioned even back then.
Obviously you (eventually) need your data where the compute is, especially in a non-von-Neumann architecture where moving data around isn't an option even if you were OK with the performance drop.
It seems kinda obvious that eventually AI will be implemented as low power dataflow custom chips integrating memory/state & compute, but who knows!
I saw them present a similar concept at Hot Chips in 2020 or 2021. It's still a cool idea, however people should remember that there are like 20 of these exotic accelerators designs pitched at trade shows every year that go nowhere.
that go nowhere
But some do end up in the industry: Mythic AI, Encharge AI, d-Matrix.
Whilst processing in memory is clearly the future, I am unconvinced by this implementation.
Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2).
To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.
"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.
I was under the impression they're doing a chiplet thing to get heterogenous processes for that reason.
If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.
Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
With the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.
Turning RAM into an active coprocessor does feel like an awkward model for that though.
Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit.
But Samsung doesn't have that option so they're playing the card they have.
I think useful flops per dollar will outweigh any "awkwardness" factor.
Elegant designs don't win anymore.
Exactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference.
If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.
People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.
At this point processing-in-memory has taken on the aura of fusion power.
I think Wright brothers had long history of failed attempts.
And all of sudden, no passengers are crossing Atlantic on ship.
Well, when you fail at everything, you want other's to fail too. Luckily for us, your failure doesn't affect others.
OLED also has a deep history with a pretty famous opinions that it is impossible and a waste of time and money for companies to invest in the development. So does AI. From inception to product it is normal for decades to pass.
Compute needs change. AI needs are pretty unique in terms of scale and type of compute requirements to anything else so far.
> Whilst processing in memory is clearly the future
How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.
Because now the usecase fits the needs.
Maybe. The use cases have always been around FP arithmetic over arrays, because that's what is easy to parallelize. I saw a standalone systolic array box attached to a MicroVAX circa 1990. Lots of PIM approaches in the mid-90s, too, but mostly what survived from that era are shared memory NUMA multiprocessors and using GPUs for general purpose computing.
This is really just restating the assertion I'm asking about. Previous work also thought their use cases fit the needs.
Everything changes all the time in computing, so there are lots of differences. I'm not asking if things are different or claiming it won't work this time either. I'm asking what it is this time that makes it different / obvious. Not rhetorical, I'm interested if someone can actually explain. Preferably with trends and numbers that have $ and flops and picojoules and gates and bits in their units.
Not if you have duplicates of rows on the first matrix, which can be done very efficiently if you build specialized hardware. Then its all just forward in parallel.
you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.
what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.
that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.
Might as well just go whole hog and change the entire computer architecture, then. A lot of the arguments against this change boil down to computers and software code don’t work well with this today.
What I find amusing about moving compute to a RAM bank is it _almost_ resembles where we were with ISA-based extended RAM back in the 1980's. Some cards featured a CPU that took over the whole system and/or functioned like an upgrade. Others were a "computer on a card" that provided other features. I think this goes to show how cyclic tech can be. So, something like Samsung's invention here might have gained traction, as overcoming the slow PC ISA bus would have been a huge accelerator, kind of like where we are now.
Maybe we can connect multiple Samsung RAM sticks to a grid and call it RAMsputer.
So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
> So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
This is a temporary issue. JEDEC's LPDDR6-PIM is going to add defined commands for Processing-in-Memory operations. Once there are standardised commands, it will be possible for the CPU vendors to make the CPU cache aware of what is happening.
Of course, that doesn't solve it for this generation of the technology. But I think this generation is more of a demo for early adopters to gain experience with it. It will likely take a few years for all these issues to be solved, but there is no principled reason why they can't be.
I guess you also get very high bandwidth that way? I‘m not sure that would come for free though.
It comes at a cost of a fragmented memory space, which is fine for some applications, like LLMs
if your working set fits in cache PIM is irrelevant
Caches are important for cpu core performance even when the the working set doesn't fit.
So instead of putting more cache on the cpu you just put the cpu on the cache.
Sort of, but the cache architecture is replaced with memory architecture. But you could look at it either way.
In memory compute has been in talks since LLMs took up. I remember few flocks were trying to get RISC V cores in the memory like these papers
https://arxiv.org/abs/2602.01827
It has been a prominent "next major shift" idea in computer architecture since the 90s, to deal with the memory wall. Eg David Patterson advocating it in the 1990 and 1997 articles. or CRAM [1].
[1] https://www.eecg.toronto.edu/~stumm/Theses/Elliott-PhD98.pdf
Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot.
That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required.
Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
> You could run MACs directly in RAM
Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes?
Also, doesn’t it mean that you forgo batching?
Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place.
How would you do map-reduce across multiple DIMMs w/o extra reads/writes?
PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.
Re-arrange x+(y+(z+w)) into (x+y)+(z+w) to accomplish a distributed sum. Shared bus, interconnect fabric. This is a supercomputer on a chip.
> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
Yes, but running out of RAM is impractical due to low memory bandwidth.
According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width:
> Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s.
And that's just for single 64-bit IC. So way faster and more power efficient.
You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly.
Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.
How do you scale past 16 channels though? 16 channels give you around 614 GB/s, while PIM can do that per chip, so it can achieve 58TB/s.
You: "AI, please write me $COOL_APP."
AI: "Sorry, all the hardware is made for running AI."
We can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)
The idea is this:
You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip.
614 GB/s x 96 = 58,944 GB/s.
Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast single digit gigabyte models. Without MoE you could reach 100 tokens per second with an 8B fp8 model on a single channel. With MoE you might break 500 tokens per second.
> load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip.
Won’t you have a bunch of extra reads/writes via the CPU because these DIMMs won’t be able to compute matrix multiplications?
And now the 96 memory slots need individual cooling.
Which might be easier since the surface is larger
> That being said, I am not sure what's the killer application for this technology...
Build it, and they will come ;)
10 years ago hpe labs had something similarly including the operating system ! and the cto got ousted and a new ceo camea nd the whole new direction is always networking !
“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.”
So how big are these banks? If you can’t fit the weights of a layer into one bank then presumably you lose a lot of the speed gains.
Not necessarily. If your weights have to go across 2 banks, you just have to split and transfer the input and output vectors, which are much smaller.
For GEMV you lose nothing.
The reason is quite simple. You can split the matrix along both dimensions so you just tile it into 64x64 or whatever fits into the bank and just fill it up. The biggest problem is load balancing the tiles across all banks for maximum parallelism.
For GEMM I believe there is no point in doing PIM, you are better off with a GPU or NPU.
This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.
There's been a ton of optimizations already, it hasn't remotely reduced demand even temporarily. More efficiency just makes the compute have even higher ROI per $ and watt spent.
With sufficient optimisation, there ought to be a tipping point beyond which local inference is good enough. And, sure, datacentre compute will still be needed for training but one of the biggest current uses will begin to taper off.
The question really is how soon we reach that tipping point, and whether it's before or after the current bubble runs out of steam for some other reason.
>there ought to be a tipping point beyond which local inference is good enough
There's no such ought really. Even at current levels you'd need like a 100x gain from here to approach current top proprietary models (probably a lot more for say Mythos or Mythos 2), and it's not like they are stoppng to improve. This is before we even account that you'd just be running 1 agent then, and not a swarm like you'd be able to in the cloud or that you can do only so much compression before you are losing out
> (probably a lot more for say Mythos or Mythos 2)
Not everyone needs that large of a model, though.
It's not just inference, some things done in data centers like simulations, testing, are complementary to inference.
And in these types of hardware, the time between a successful prototype and a fully deployed product is pretty long. Maybe they count on that to know when to stop?
Jevons Paradox shows that increasing efficiency can increase demand for a product by making it cost effective for more uses.
What about energy consumption? Wouldn't active cooling be needed for RAM as well as for CPU and GPU?
The better question is: what is the net gain for the overall system? If PIM reduces the net thermal load and power consumption of the system for the same workload, then it’s a win regardless of where the heat sinks end up. The customers Samsung has in mind for this today are not limited to commodity designs. They’re using novel designs with each new hardware generation, so moving heat sinks around is not a deal-breaker.
I wrote up a theoretical post here about LMM performance of a MacBook Pro with PIM memory:
https://ben3d.ca/blog/m5-max-samsung-lpddr5-pim-650-tokens-p...
Interesting read. Definitely hoping this works out so we can have cheap LLM machines at home
I thought DMR and Venice were supporting memory encryption by default. With the keys living on the CPU side and no standards for key sharing, I wonder how this will gain traction.
In-memory computation was already possible with regular DRAM: https://news.ycombinator.com/item?id=22712811
Add a new set of CPU instructions like “rep macb”
...and it's been long enough now, that I can say there was an effort to implement this on standard x86 memory controllers and have the existing string instructions do so, back in the days of SDR SDRAM, but the tradeoffs weren't (yet) in favour.
that's not at all comparable, you're still paying memory latency and not getting any extra bandwidth
That game is still going? I have a char from 06!
Is this about the fake craters in moon photos?
Self changing RAM and a complex way to interact with it in software. A new security nightmare is emerging.
It doesn't suffer from the inherent security issues of the von Neumann architecture. Memory will only be data.
memory plus memory bandwidth > GPU, this is the gist of it.
2021 https://patents.google.com/patent/US11600340B2/en
https://x.com/xennygrimmato_/status/2025376089607209218
If we could buy a 64gb stick and run a 32b model with 30tps on it. This could sell
The discussion about cpu caching challenges etc seems like it's missing the point. Wouldn't it be more likely to DMA the results to a GPU anyway?
Feels like the most realistic/short-term way to make use of this would be to set up some barebones RTOS to run from CPU cache with the PIM memory being used for compute only and use the device as a network attached accelerator.