> This makes them AMD’s first desktop chips to qualify for Microsoft’s Copilot+ PC label, which enables a handful of unique Windows 11 features like Recall and Click to Do.
This is not the selling point they think it is.
The problem I see with the AM5 socket is simply the fact that DD5 RAM to support it is just too expensive. So this will not really make the big impact they were hopin for.
I have a hard time believing ANYONE thinks this is a selling point. Literally anyone, including marketing and execs at Microsoft. I think they have just sunk too much money in it to quit, so they keep doubling down.
"The problem I see with the AM5 socket is simply the fact that DD5 RAM to support it is just too expensive."
DDR4 is basically just as expensive. At least DDR5 gives on-chip error correction (not as good as full ECC).
For a geneal computer, there's not that much difference between AM4 and AM5 unless you really want the extra speed of DDR5, PCI 5, and the newest processors. You can build a very capable AM4 machine for slightly less money, but that savings is found on the CPU and motherboard, not on the RAM.
The ECC in DDR5 is there just to make it work because of the errors caused by the density and the data rates. The ECC isn’t there for you, it is for the manufacturers.
I am still running DDR2 & DDR3 machines! I was going to finally make the big upgrade this year but am now holding off until the market finds a little bit more sanity.
DDR4 looks to be around half the price of DDR5 on the used market to me. I wouldn't call that slightly less money unless you weren't planning to install much RAM.
On the new market, they are really close. Most consumers are buying new. If we want to get pedantic, we can compare buying used systems on Facebook marketplace to cannibalize the parts and resell the others to see net cost.
I don't think most people building AM4 systems currently are buying new, or at least not everything new, simply because depending on what you're looking for there might not even be any new parts.
I think for brand new computers/builds that's correct but where it hit me was wanting to upgrade an existing desktop. I already have more DDR4 RAM than I need and would have been willing to purchase a new CPU/motherboard and being forced to also purchase new RAM at the same time made it too big of a price tag all at once. I just found the best zen 3 cpu I could on ebay and called it a day.
I think your point still stands overall for AMD's business though, I assume a vast majority of CPUs are purchased in new desktops?
Agree, i just built a desktop for the first time in ages, it is a leap and change from using laptops with numerous components pplugged into them. i made the leap to desktop. everything was comparably reasonable except the RAM or anything that has memory chip on it ( RAM, NVME etc) so i did some research just to make sure. All in all i happy with the result i went with AMD 9900x no graphics card in this option, i skipped the graphics card for now.
I would like to add that, looking for bundles helps a lot. If you have micro center near you, utilize it to you full advantage they are the only ones given promotional items with bundles at the moment from what seen. The main objective is to skip the price gouge of RAM chips, they cost more than the CPU at the moment. I got CPU and motherboard plus 32g RAM fro $600 and that was a save. the RAM was $445 alone.
HN is overloaded with AI stuff, its hard to break through all the noise. I say this as someone very interested in AI. Even I skip some links because its just too much.
I see it making claims about 10x efficiency, but how is tokens / second / watt? The only machines I've seen with the memory bandwidth to effectively do local inference are Mx arm chips on mac.
because it's not faster than the Ryzen 395's GPU. power efficiency doesn't matter as much as TTFT for desktop users, especially when they're tasking their AMD box as a dedicated inference machine.
some older pre-395 AMD articles suggested it'd be possible to use the NPU for prefill and the GPU for decoding and this would be faster than using either alone, but we have yet to see that (even on Windows) for any usefully sized models, just toys like LLaMA-8B.
To really take advantage of those gpu cores you need memory bandwidth. Modern transformer based LLMs are really bandwidth hungry. I am really happy to see this first push. NVIDIA having discrete GPU/memory/etc is an option, but not great for a lot of different reasons. Unified memory architectures like what AMD and Apple have are the way to go for the future. Put 256GB of ram on the main board and be able to access it at speed for LLM use please.
Meanwhile, the corresponding "non-standard" desktop PC is the Framework Desktop, which with the Ryzen AI Max+ 395 can use 120GB of its 128GB RAM for the GPU: How to Run a One Trillion-Parameter LLM Locally: An AMD Ryzen™ AI Max+ Cluster Guide https://www.amd.com/en/developer/resources/technical-article...
The Framework Desktop motherboard does actually have a PCIe x4 slot, their case just doesn't expose it for whatever reason. But you can buy the board separately and put it in your own choice of Mini-ITX case which does.
First I've heard of this thread (I haven't spent much time on the forums for a while), but my Framework Desktop definitely has the PSU noise issue, and it doesn't seem correlated to system load. Otherwise a really solid machine.
That PCI slot has low power output, not suitable for what people would plug into it. Easier to cover it than run into support problems. It won't run a GPU directly. It will run an Oculink card which will allow use of an egpu.
I can't imagine why you'd want to buy a beefy SOC with unified memory, only to have it host a discrete GPU through it's narrow PCIe x4 interface. You'd be better off with a traditional CPU that supports a proper x16 slot.
They're a small shop compared to the big laptop makers and their focus is on user-repairability. Between those two you are naturally going to have a slightly higher price point than simply buying a laptop from one of the usual companies. I wouldn't call it junk or gimmickery. It has a purpose and a niche. It might not be your niche, but it is for some people.
Yeah, but all of this is pointless when RAM is as expensive as two CPUs by itself - if it's even in stock. AMD/Intel should focus on that first if they want to save their DIY business at all - which I'm starting to doubt they don't
The RAM shortage is... a shortage. It's temporary by nature. RAM didn't suddenly get 4x more expensive to produce, it's just in high demand right now. Supply will eventually catch up even if it takes a few years.
Most of us are not really doing detailed half-decade hardware purchasing plans, for most of us "the next few years" is really the only relevant time frame.
Right now I'm on a 2021 MacBook Pro, I'm debating whether to upgrade to the new M5 Pro laptops or to wait until next year; so that's 5-6 years. My desktop gets upgraded over time, so it's not really on a "cycle"; it's more that every few years, I may decide to get a more powerful CPU, or more RAM, or a new GPU, or more storage, or whatever else. Though I recently (as in, during the past year) upgraded from an AM4 CPU to an AM5 CPU, so that meant I replaced more than usual; everything other than the GPU, storage, power supply and peripherals. (I switched from ITX to mATX for an extra PCI slot so the case had to go; though it still lives on in the form of a lab PC of sorts, along with my old i7 6700k)
But common among all these replacements is that they're not really planned in detail years before. I may weigh up factors like "I don't really need an upgrade right now" against factors like "the market looks like it'll probably get really shitty later this year", or "I could really really use an upgrade right now" against "but the market is shitty now and we're right before a product launch which will shake things up". But I always react to my current or near-future needs/wants and current or near-future market conditions.
So hearing "the RAM market will be good again in 5 years" is completely irrelevant to me. My decisions are entirely based around how the RAM market is right now and how I believe it will look throughout the year or the next.
Also supply will only increase if producers believe the current level of demand is going to be sustainable in the long term. Which I don't think anyone really believes.
The same way nobody wanted to invest in mask supply in the US or Western Europe during covid, because producers knew the demand spike wouldn't last and they'll be left with useless equipment to pay after the crisis passed.
> demand is going to be sustainable in the long term. Which I don't think anyone really believes.
I am one who think there is a chance it will sustain. AI is useful tool, Opus unlocks N times productivity gain for devs since Opus 4.5, which is available just for 3 months.
This means adaptation is just started, it could expand on all kind of usecases, niches, solving problems, products, etc, which could be N times more demand for compute from what we have now.
Correct, but if they're right in that belief then it's still just a matter of time before demand goes down and the shortage is solved.
If they're wrong and this is actually a permanent spike in demand, then it'll take the industry a while to realize it but eventually they'll collectively figure it out and increase supply. The ones who figure it out soonest and increase supply fastest will profit the most. The ones who figure it out slowest will lose market share.
If demand declines then the shortage is still solved regardless.
If it doesn't decline, than anyone who took that risk and increased their production capacity will benefit greatly, and those who didn't will lose market share.
And if they invest in new fabs and the bubble doesn't pop then they make a whole lot of that money they're in business to make.
The incentives here are naturally very well aligned with solving the shortage. If doing nothing is likely to solve the shortage, then they'll do nothing. If increasing supply is likely to solve the shortage, then they'll increase supply. If there's a 50/50 chance of both, then some will increase supply and some will do nothing, and the market will reward whichever group was right and punish the other.
You forgot to say “Amen” at the end of your gospel.
I'm always baffled how seriously some people take that “market and incentives always lead to the greater good” religion despite plenty of empirical evidence against it.
But hey, there are creationists out there too so it's not too surprising I guess.
Let me describe this in the most simple terms possible: You have speculators speculating about AI products. The speculators are not very smart when it comes to technology, and think RAM is RAM. There is at least three kinds of RAM that are important to this: DDR for system RAM, GDDR for GPUs, and HBM for high density enterprise products, and they are not interchangeable, there is no one-die-fits-all solution.
So, these speculators are like "oh no, more GPUs requires more RAM!", and then just start speculating on all RAM. Which of these RAMs are the ones that they need to worry about? Exclusively HBM, which is a minority in production, DDR and GDDR dominate production.
If you're into inference, and have older machines, you're buying Hxxx or Bxxx cards that use HBM, fit into dual slot x16 configurations, and you're jamming (optimally) 8 of them in. If you're into hardware that is newer, somewhere in the middle of the inference boom, you're using MXM cards. In either situation, the host machine has DDR, but if you're OpenAI, Anthropic, Microsoft, or Google, you're not building (more) inference machines like this.
The first two are buying Nvidia's all in one SBC solution: unified HBM, onboard ARM CPU to babysit the dual GPUs, has its own dual QSFP network controller that can RDMA, etc. No DDR or GDDR involved. Any machines built before this platform are being phased out entirely.
Microsoft is doing the same, but with AMD's products, the MI series that co-locates Epyc-grade Zen 4/5 CCDs with CDNA compute chiplets, running the entire thing off HBM, thus also unified and no DDR/GDDR needed. They, too, are phasing out machines older than this.
Google has a mix: they offer Nvidia all in one SBCs as part of GCP for legacy inference tasks (so your stack that can't run on AMD yet still can run), but also offer the same MI products that Microsoft offers via Azure's inference product, but also has their own TPUs that some of Gemini runs on; the TPUs run on HBM afiact. No DDR or GDDR here.
So, what does AMD or Intel do here? Lets say they waste fab time to make their own dies on the wrong process (TSMC and Intel-Foundry do not have for-RAM optimized processes)... they would be producing DDR and GDDR for a market that almost has its entire demand met. Intel lacks the die stacking technology required to build HBM, and TSMC I think can't do it for that many layers (HBM has 8 to 16 layers in current gen stuff iirc).
Micron, for example, already is bringing two large factories online here in the US to meet the projected growth in demand for the next 20+ years. When these factories finally start producing, it will not change the minds of speculators: they still seem to think AI datacenters need RAM, of any kind, and refuse to understand even the most basics of nuance. Also, when they come online, HBM will be a minority product; the AI inference boom is still just a bump in the road for them.
Nvidia kinda screwed their consumer partners, btw: they no longer bundle the GDDR required for the card with the purchase of the die. There is a slight short term bump in GDDR spot prices as partners are building up warchests to push series 60 GPUs into production, and once that is done, spot prices return to normal (outside of the wild speculation manipulation).
One last thing: what about LPDDR, used by AMD Strix Halo and Apple stuff? Speculation seems to have not actually effected it. I consider it as a sub-category of DDR (and some dies seem to work as either DDR or LPDDR as of DDR5, due to the merger of the specs by JEDEC), but since it isn't something you find in datacenters, it seems to have avoided speculation.
The Ryzen Max CPUs mentioned in the linked article? Uses LPDDR. Doubling down on the Ryzen Max product line might be a brilliant move.
> The speculators are not very smart when it comes to technology, and think RAM is RAM. There is at least three kinds of RAM that are important to this: DDR for system RAM, GDDR for GPUs, and HBM for high density enterprise products, and they are not interchangeable, there is no one-die-fits-all solution.
The commenter is also not very smart and does not realize companies making the RAM can trade capacity of one for another and any re-tooling at current price is still profitable.
The commenter also does not realize that is also true for lines currently making SSDs
They can trade capacity, but they generally aren't. The huge storage-only fabs owned by Samsung and Micron do runs that go for 9 months to 12 months.
Flash chips haven't been speculated on nearly as hard, and are suffering from the same sort of weird lack-of-nuance. Samsung, for example, isn't reassigning capacity to meet some sort of phantom datacenter demand that isn't already there, generically, across all datacenters, AI or not.
A lot of SSD price skyrocketing is largely "SSDs have RAM on them for cache", not "SSDs have flash chips, and they're both made at the same fabs"... which oddly effects low end SSDs that don't have external cache.
To make it worse, for the speculators who do understand this, because it isn't some universal homogeneous group, the flash chips that go into enterprise SSDs aren't the same that go into consumer SSDs.
The Big Three still aren't doing some major re-tasking of capacity, as the actual global demand isn't outstripping supply any more than normal. There is no short term problem to fix, speculators are just gonna have to stop hoarding toilet paper like its the start of Covid.
Edit: Oh, and if you want to ask how AMD/TSMC or Intel solve this? They can't, same reason why making their own in-house HBM isn't happening.
I'm glad Kioxia (formerly Toshiba) have been able to do that. However, I also know they've been having problems meeting demand for quite some time, and haven't been able to scale up nearly as fast as the big three have. There was an incident in 2019 and another in 2022 that killed entire runs of chips and screwed them during the Covid datacenter rush.
Micron killed Crucial because Crucial was a weird offering that competed with their own partners. This was always a weird problem, and it just didn't make financial sense to continue with it. One of the analyses I read was Crucial was less than 12% of sales.
Like, don't get me wrong, I've liked many Crucial products over the years, and even recommended some of them, but it was always weird they were trying to out-compete companies like Adata and other major ODMs.
The counterexample of this is Nvidia absolutely trying to kill their partners, and going to first party assembly and sales of products. Nvidia isn't even going to PNY anymore for ODM needs, but going directly to Foxconn.
Micron execs claiming its because of AI is a bit weird and revisionist, because they've been working on exiting the Crucial brand since long before they publicly announced it. The public didn't learn of any such plans until right before the Ballistix brand sunsetting was announced in 2021, but started years before that. Like, I know they're just playing to their shareholders, but its still a bit weird.
As far as I know, the current lineup is PNY still makes the workstation cards, possibly also the x16 server cards, but Foxconn is doing the Blackwell SBCs and MXMs, and those SBCs are a pretty big chunk of Nvidia's income right now. I also believe they have moved to Foxconn for the Founders Edition consumer cards.
Also, with the FEs, their partners are disallowed from making their own FEs, even if they make their own PCB from scratch and not based on any existing Nvidia design. Doesn't matter who makes the FE, it immediately puts partners at a great disadvantage if they can't make one too.
So where are all these speculators storing DDR5, flash, and even spinning hard disks? Asking for a friend.
As a small buyer of all of those things supply at nearly any price has gotten very difficult to reliably predict week to week. When a lot of 100 64GB DDR5 sticks shows up available at a vendor, it’s at a take it or leave it price good for a couple hours. If I don’t pull the trigger they have another buyer for it and I might be waiting another month before anything becomes available again. We can no longer JIT for even failure replacement on our edge nodes.
Then you have the NVMe and even SATA SSD shortages. Still a bunch of very useful hardware out there I would love to find a decent deal on 8TB sata so I could repurpose it. Just doesn’t make any sense right now at current pricing and availability. Good luck trying to even find a batch of 12 of these disks at a time.
This goes for both enterprise and even prosumer I was willing to take for some of these uses.
There is no such thing as "consumer grade RAM". Servers still take DIMMs, ECC DIMMs just has more chips on it (previously 9 instead of 8, but now 10 instead of 8 as of DDR5; you'll see some DDR5 DIMMs with 5 instead of 4 because they're double die packages).
Micron, Samsung, and Hynix just basically sell you chips that comply with the JEDEC spec, and the DIMM manufacturers further bin them according to purpose. The highest end chips (that are stable at high clocks and acceptable voltages) end up in enthusiast performance products, the ones that don't work well at all but still meet JEDEC spec are sold to Dell/HP/Lenovo/etc for Grandma's Facebook machine, and the ones that are exceptionally stable at thermal design limits are plunked onto ECC DIMMs and sold to servers.
Also, as others have mentioned, its just a fab, and it can make any of the dies they're able to make. Whatever needs to be made to meet demand, they make, they just can't turn on a dime and react to quarterly concerns, and are locked into cycles that may range from 6 months to 18 months.
Side note that is also worth mentioning, sometimes you can order special bins of parts with features that wouldn't normally be available if you're willing to order enough. Recent example being Nvidia buying overclocked GDDR6 chips from Micron with additional features enabled; Micron was more than happy to become Nvidia's exclusive supplier for the custom GDDR chip if Nvidia was willing to buy out the entire run. Stuff like this happens every so often, but isn't the norm.
The announcement means that they're closing Crucial - just like it says in the title and the first paragraph. The rest of that press release is outlining the mechanics of how that works + some fluff. Micron is going to continue producing the exact same memory chips in the exact same fabs. They're just not soldering it to a board, slapping the Crucial logo on it, and selling it directly to consumers. There's nothing stopping downstream vendors from buying Micron chips, soldering them to a board, and selling them to consumers as Micron was doing previously.
There's nothing in that press release that implies that the memory was somehow different (or "consumer-grade"). The _only_ thing they're saying is that they're ending their B2C business and focusing on B2B.
Didn’t you just describe the literal difference between consumer grade RAM that is soldered to a consumer format board vs a memory chip sold to a company to be soldered onto a product of the other company?
You just need an additional chip to move from "consumer grade" (ie no parity) to "server grade" (ie have parity). ECC support is actually in the memory controller which is in the CPU for the last 15 years. No magik.
> what about LPDDR, used by AMD Strix Halo and Apple stuff? Speculation seems to have not actually effected it
Good luck actually finding them on stock with 128GB+ RAM. I got strix laptop while ago, now price in EU is technically the same, but no stock. Maybe month or three
There is also claw hype. And large gwen3.5 models can run very well on DDR5 CPUs or mac minis...
I find the panic over RAM prices to be overestimated. 32GB DDR5 RAM is around $500 which is comparable to to the 9800x3D. Sure it sucks that it increases by around 4x, but when you factor in the overall price of a top end PC at around 1000-2000, especially for the lion's sum of the GPU, the increase is marginal.
This only effects a very narrow slice of highly budget conscious consumers trying to build high end PCs at razor thin margins.
$500 for 32GB is about $15/GB which is a high we haven't seen since the mid-2000s. This is a big deal, it turns RAM and to some extent storage (especially fast storage) into a massive economic bottleneck.
Adjusted for inflation, the last time prices (/GB) were this high was May 2011; the tail end of the 2009/2010 shortage. Aside from a brief glut in 2008, it wasn't really cheaper before (than it is now) though. Of course RAM is much faster these days, but also in 2011 most people had no more than 4 GB of system memory and 512 MB VRAM.
Not the point - my point utterly of arithmetical nature - dollar has substantial inflation, and any comparison more that 5 years apart, let alone 20 warrants adjusting of prices, as error is substantial, 2x in the case of 2000s
That was not my point entirely; my point that citing prices from 2000s and comparing with modern ones |(with indexing about 2x times), regardless of underling reason is either a demonstration of lazyness or innumeracy, or even worse - an attempt to manipulate.
It’s not laziness, innumeracy, or manipulation when it can be taken at face value that the cost increase vastly outstrips anything that could be attributed to inflation. You don’t even need to look it up to know that.
> when it can be taken at face value that the cost increase vastly outstrips anything that could be attributed to inflation
But that was not my point _whatsoever_. What I said is - every time you bring the explicit numbers (like in GP "$500 for 32GB is about $15/GB which is a high we haven't seen since the mid-2000s") you _absolutely_ have to adjust for inflation to have a meaningful conversation. This is it.
That was not my point entirely; my point that citing prices from 2000s and comparing with modern ones |(with indexing about 2x times), regardless of underling reason is either a demonstration of lazyness or innumeracy, or even worse - an attempt to manipulate.
I disagree with you. The issue does not only affect a “very narrow slice” of consumers. https://www.techspot.com/news/111472-hp-warns-ram-now-makes-... A major brand is now suggesting that this is a “new normal” and one solution is to just offer systems with less ram. This is an issue when lots of modern software seems to expect an unending supply.
That is an insane amount of money for just 32GB of RAM! That's what we were paying back when it was hard to use more than 32-64GB in a desktop setting. These days with all the electron and node bloatware, containers everywhere and AI - 32GB doesn't get you far.
$500 is 5x what it cost less than a year ago, just for context. It turns a $1600 computer build into a $2000 one. That’s a huge difference.
Edit: I don’t get your math. If we’re using a very generous definition of “top end,” even neglecting Nvidia and going AMD - which some would argue makes it not top end - you’re talking conservatively: $600 for a GPU, $500 for 32gb of ram, and $500 for a CPU. $1600 before PSU, case, SSD, fan(s), mobo…there’s no world in which you’re coming in under $2k. The SSD and board will put you over immediately.
You’re talking 3/2025 prices, not 3/2026. A compromise, mid-range computer is $1500 to build now.
Look at my comment above and see what i said about my build. I was unwilling to pay that for a moderate build that can sustain my computer use. Utilize bundles that can save you $ on RAM or chipset, CPU to skip some of the costs.
You aren't specific in your comment. Where are these bundles? What do you do with all the parts you don't need/end up swapping out? How much are you actually saving?
That's a ridiculously implausible and sensationalized rumor. At most, Microsoft may make a NPU a requirement for OEMs to use the Windows 12 logo on new PCs. Actually refusing to support the existing install base of recent and hughly-capable desktops is not at all likely. It would be far more drastic than the hardware deprecations brought by Windows 11, which were already quite controversial (and loosely enforced).
Hoe much dedicated cache do these NPUs have? Because it's easy enough to saturate the memory bandwidth using the CPU for compute, never mind the GPU. Adding dark silicon for some special operations isn't going to make out memory bandwidth faster.
It's streaming access, and no not as far as I'm aware. APUs have always been hilariously bottlenecked on memory bandwidth as soon as your task actually needed to pull in data. The only exception I know of is the PS5 because it uses GDDR instead of desktop memory.
I'm not sure what you mean - I think the mobile 300 series can do quad channel already for its APU at least. I'd assume it can do more but do you not need more slots beyond that?
There are a single-digit number of products using the AMD Strix Halo mobile parts that have a 256-bit memory bus. All other mobile x86 processors (including AMD's mainstream Mobile silicon used for these desktop processors) have the usual 128-bit bus.
Is this fast GPU like instructions that anyone can use in any operating system to run any open sourced LLMs on CPU with all your RAM rather than on a discrete GPU with its own limited amount of VRAM?
Or is this a proprietary thing that only works in Windows for some specific use cases and irrelevant for Linux users?
it's a distinct piece of hardware based on AMD XDNA architecture, which, coincidentally, much like CPUs, can tap into your RAM pool. there are XDNA drivers (`amdxdna`) for Linux.
AMD marketing is hoping the “AI” branding is a positive. Antidotally, I know many consumers who are not sold on AI. This branding could actually hurt sales.
We are dealing with a hype, but the reality is that AI would change everything we do. Local models will start being helpful in [more] unobtrusive ways. Machines with decent local NPUs would be usable for longer before they feel too slow.
> the reality is that AI would change everything we do
Your true believer convictions don't matter here. Those AI accelerators are merely just marketing stunts. They won't help your local inference because they are not general purpose enough for that, they are too weak to be impactful, most people won't ever run local inference because it sucks and is a resource hog most can't afford, and it goes against the interests of those scammy unprofitable corporations who are selling us LLMs as AI as the silver bullet to every problem and got us there in the first place (they are already successful in that, by making computing unaffordable). There's little to no economical and functional meaning to those NPUs.
> You have fallen headfirst into the "Not now, so never" fallacy.
Perhaps. Though we have empirical evidence of how much we can quantize and distillate models to the point of practical uselessness. That sets a bar for how large a local model needs to be for general-use as to compete with the could ones. We are talking in the area of 60GB for GPT-OSS/Qwen3.5, which is what enthusiasts are running on 32GB DDR5 + 24GB VRAM RTX 3090.
> As if consumer hardware won't get more powerful
Now I will let you, with that last fact in hand, plot a chart of how much it's been costing to provision that over the past 2 years and use it to prove me wrong about the affordability of local models.
> most people won't ever run local inference because it sucks and is a resource hog most can't afford
a) Local inference for chats sucks. Using LLMs for chatting is stupid though.
b) Local inference is cheap if you're not selling a general-purpose chatbot.
There's lots of fun stuff you can get with a local LLM that previously wasn't economically possible.
Two big ones are gaming (for example, text adventure games or complex board games like Magic the Gathering) and office automation (word processors, excel tables).
It surprises me that semantic search never gets mentioned here.
If you can use the NPU to process embeddings quickly, you get some incredible functionality — from photo search by subject to near match email search.
For consumer applications that’s what I’m most excited for. It takes something that used to require large teams, data, and bespoke models into commodity that any app can use.
> Ask your friends or a small business owner if they are going to spend $1k on a new laptop because "there's lots of fun stuff".
Do people not buy gaming PCs and game consoles? Isn't that buying something because "there's lots of fun stuff?"
And while sure a business owner wouldn't be buying it for "fun stuff", if it was about being able to run the AI tools they want without the business risk of sending your most important data and intellectual property to an AI provider wouldn't some think about it?
Are you kidding? A good ratio of ST folks run finetunes of Mistral Nemo (if it tells you anything). Anyway your core statement is simply wrong ("local chat sucks").
> If you intend to do LLM inference on your local machine, we recommend a 3000-series NVIDIA graphics card with at least 6GB of VRAM, but actual requirements may vary depending on the model and backend you choose to use.
Also, please be respectful when discussing technical matters.
> we recommend a 3000-series NVIDIA graphics card with at least 6GB of VRAM
...which is not by any means a powerful GPU, and besides the AMD Ryzen AI CPUs in question have a plenty enough capacity to run local LLMs esp. MoE ones; with 3b active MoE parameters miniPC equipped with these CPUs dramatically outperform any "3000-series NVIDIA graphics card with at least 6GB of VRAM".
> please be respectful when discussing technical matters.
That is more applicable to your inappropriately righteous attitude than to mine.
> Parent comment is fair and technically accurate.
In what way precisely? That local LLMs "suck"? Is that a technical argument?
Or this statement "there's little to no economical and functional meaning to those NPUs." - is that actual factual statement or a emotionally charged verbal flatulence? and what "they won't help your local inference because they are not general purpose enough for that" even means? People succesfully run largeish MoE llms on AMD Ryzen AI miniPCs.
> Do you have a real argument, especially a technical one, that you can contribute?
What kind of argument do you want me to "contribute" wrt the ideological rant the "parent comment" had managed to produce?
Hey, OP here with their (apparently) controversial views. I stand firmly within those lines:
- I shouldn't be paying more for my next CPU because it has a NPU that I won't ever use. Give me the freedom of choice.
- Given that freedom of choice, it would seem that a majority would opt-out (as seen recently by Dell), so the morals of all that are dubious.
- NPUs may not be completely stupid as a concept, in theory, but at this point in time they are proprietary black-boxes purpose-built for marketing and micro-benchmarks. Give me something more general-purpose and open, and I will change my mind
- …but the problem is, you can only build so much general-purpose computing in bespoke processor. That's kind of its defining trait. So I won't hold my breath.
- Re: local-inference for the masses, putting aside the NPU shortcomings from above: how large do you think a LLM needs to be so it's deemed useful by your average laptop user? How would the inference story be like, in your honest opinion (in terms of downloading the model, loading it in memory, roundtrip times)? And how often would the user realistically want to suffer through all that, versus, just hopping to ${favorlite_llm.ai} from their browser?
Anyhow, if that makes me "antiai", please, sign me up!
> I shouldn't be paying more for my next CPU because it has a NPU that I won't ever use. Give me the freedom of choice.
There is a plenty to choose from.
> - NPUs may not be completely stupid as a concept, in theory, but at this point in time they are proprietary black-boxes purpose-built for marketing and micro-benchmarks. Give me something more general-purpose and open, and I will change my mind
In fact the linked article is not talking about NPUs in particular, but about Ryzen AI cpus. These have unified memory and more compute compared to normal ones which make them very useful for inference.
> how large do you think a LLM needs to be so it's deemed useful by your average laptop user?
Depend what they need it for. Useful autcomplete in IDE starts at around 4b weights.
> loading it in memory
Happens only once, usually takes around 10sec.
> roundtrip times
Negligible? it is loca after all.
> And how often would the user realistically want to suffer
I am bullish on AI being used in all sorts of useful and discreet and non-discreet ways in the present and future. However I am exceedingly skeptical of NPUs being some winning bet.
No one is running LLMs on current gen NPUs so if we will in the future its a long time coming. Unless they can demonstrate some real (and not marketing) wins I remain skeptical that a large NPU for LLMs is the future.
I can totally see NPU accelerating simple tasks, but to be worth the silicon they have a ways to go imo.
99% of people don't need or want a dev workstation. My travel laptop is 7+ years old and I couldn't tell you the difference between it and a current flagship in terms of browsing and everyday tasks.
I will not lie, I find LLMs useful but the desktop experience is pretty polished already. NPUs seem to be an attempt to ride the AI bandwagon with very little to show for it so far.
For some people maybe. I don't want to use local AI and NPU will be dead weight for me. Can't imagine a single task in my workflow that would benefit from AI.
It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
> Can't imagine a single task in my workflow that would benefit from AI.
You don't do anything involving realtime image, video, or sound processing? You don't want ML-powered denoising and other enhancements for your webcam, live captions/transcription for video, OCR allowing you to select and copy text out of any image, object and face recognition for your photo library enabling semantic search? I can agree that local LLMs aren't for everybody—especially the kind of models you can fit on a consumer machine that isn't very high-end—but NPUs aren't really meant for LLMs, anyways, and there are still other kinds of ML tasks.
> It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
Do you insist that your CPU cores must be completely homogeneous? AMD, Intel, Qualcomm and Apple are all making at least some processors where the smaller CPU cores aren't optimized for power efficiency so much as maximizing total multi-core throughput with the available die area. It's a pretty straightforward consequence of Amdahl's Law that only a few of your CPU cores need the absolute highest single-thread performance, and if you have the option of replacing the rest with a significantly larger number of smaller cores that individually have most of the performance of the larger cores, you'll come out ahead.
* You don't do anything involving realtime image, video, or sound processing?
I don't
* You don't want ML-powered denoising and other enhancements for your webcam,
Maybe, but I don't care. The webcam is good or bad as it stands.
* live captions/transcription for video,
YouTube has them. I don't need it for live calls.
* OCR allowing you to select and copy text out of any image,
Maybe
* object and face recognition for your photo library enabling semantic search?
Maybe but I think that most people have their photo library on a cloud service that does AI in the cloud. My photos are on a SSD attached to a little single board ARM machine at home, so no AI.
What I would like to be able to do is running the latest Sonnet locally.
In general I think that every provider will do their best to centralize AI in their servers, much like Adobe did for their suite and Microsoft did for Office so local AI will be marginal, maybe OCR, maybe not even blurring the room behind my back (the server could do it.)
There are alternatives to Adobe and Office, because I don't care about more than the very basic functionality: running Gimp and Libreoffice on my laptop costs zero. How much would it cost to run Sonnet with the same performances of Claude's free tier? Start with a new machine and add every piece of hardware. I bet that's not a trivial amount of money.
None of what I listed was in any way specific to "content creators". They're not the only ones who participate in video calls or take photos.
And on the platforms that have a NPU with a usable programming model and good vendor support, the NPU absolutely does get used for those tasks. More fragmented platforms like Windows PCs are least likely to make good use of their NPUs, but it's still common to see laptop OEMs shipping the right software components to get some of those tasks running on the NPU. (And Microsoft does still seem to want to promote that; their AI PC branding efforts aren't pure marketing BS.)
The issue is that the consumer strongly associates "AI" with LLMs specifically. The fact that machine learning is used to blur your background in a video call, for example, is irrelevant to the consumer and isn't thought of as AI.
Never wanted to do high quality voice recognition? No need for face/object detection in near instant speed for your photos, embedding based indexing and RAG for your local documents with free text search where synonyms also work? All locally, real-time, with minimal energy use.
That is fine. Most ordinary users can benefit from these very basic use cases which can be accelerated.
Guess people also said this for video encoding acceleration, and now they use it on a daily basis for video conferencing, for example.
Those usecases are at least 5 years if not 10 years out. They require software support which won't come until a significant part of the pc market has the necessary hardware for it. Until then, paying extra for the hardware is foolish.
This will only come if Windows 12 requires a TPU and most of the old hardware is decommissioned.
Also similar to GPU + CPU on the same die, yet here we are. In a sense, AI is already in every x86 CPU for many years, and you already benefit from using it locally (branch prediction in modern processors is ML-based).
> Also similar to GPU + CPU on the same die, yet here we are.
I think the overall trend is now moving somewhat away from having the CPU and GPU on one die. Intel's been splitting things up into several chiplets for most of their recent generations of processors, AMD's desktop processors have been putting the iGPU on a different die than the CPU cores for both of the generations that have an iGPU, their high-end mobile part does the same, even NVIDIA has done it that way.
Where we still see monolithic SoCs as a single die is mostly smaller, low-power parts used in devices that wouldn't have the power budget for a discrete GPU. But as this article shows, sometimes those mobile parts get packaged for a desktop socket to fill a hole in the product line without designing an entirely new piece of silicon.
Your comment is almost completely irrelevant to what the parent is saying. "AI would change everything we do" has nothing to do with "This new chip along with bloat from Windows enables new workflows for you". If you have been paying attention, you'd know that NPUs from these new CPUs barely made any difference from a consumer's perspective.
So I’ve got a lot warmer to believing that AI can be a better programmer than most programmers these days. That is a low bar :). The current approach to AI can definitely change how effective a programmer is: but then it is up to the market to decide if we need so many programmers. The talk about how each company is going to keep all the existing programmers and just expect productivity multipliers is just what execs are currently telling programmers; that might change when the same is execs are talking to shareholders etc.
But does this extrapolate to the current way of doing AI being in normal life in a good way that ends up being popular? The way Microsoft etc is trying to put AI in everything is kinda saying no it isn’t actually what users want.
I’d like voice control in my PC or phone. That’s a use for these NPUs. But I imagine it is like AR- what we all want until it arrives and it’s meh.
Indeed, I was buying a laptop for my wife, and she was viscerally against "Ryzen AI": I don't want a CPU with builtin AI to spy on my screen all the time!
That is already the case with datacenter "GPUs". A A100, MI300 or Intel PVC/Gaudi does not have useful graphics performance nor capabilities. Coprocessors ala NPU/VPU are also on the rise again for CPUs.
This is kind of true back in the day though. Uninformed people would buy Quadro cards because they were the most expensive GPU on Newegg only to realize this thing sucks for gaming.
Yes, this has already been the case for years on mobile devices, CoPilot+ PC design requires this approach as well.
Additionally, GPUs are going back to the early days, by becoming general purpose parallel compute devices, where you can use the old software rendering techniques, now hardware accelerated.
Even the latest NVIDIA Blackwell GPUs are general purpose, albeit with negligible "graphics" capabilites. They can run fairly arbitrary C/C++ code with only some limitations, and the area of the chip dedicated to matrix products (the "tensor units") is relatively small: less than 20% of the area!
Conversely, the Google TPUs dedicate a large area of each chip to pure tensor ops, hence the name.
This is partly why Google's Gemini is 4x cheaper than OpenAI's GPT5 models to serve.
Jensen Huang has said in recent interviews that he stands by the decision to keep the NVIDIA GPUs more general purpose, because this makes them flexible and able to be adapted to future AI designs, not just the current architectures.
That may or may not pan out.
I strongly suspect that the winning chip architecture will have about 80% of its area dedicated to tensor units, very little onboard cache, and model weights streamed in from High Bandwidth Flash (HBF). This would be dramatically lower power and cost compared to the current hardware that's typically used.
Something to consider is that as the size of matrices scales up in a model, the compute needed to perform matrix multiplications goes up as the cube of their size, but the other miscellaneous operations such as softmax, relu, etc.. scale up linearly with the size of the vectors being multiplied.
Hence, as models scale into the trillions of parameters, the matrix multiplications ("tensor" ops) dominate everything else.
I'm not aware of any major BLAS library that uses Strassen's algorithm. There's a few reasons for this; one of the big ones is Strassen is much worse numerical performance than traditional matrix multiplication. Another big one is that at very large dense matrices--which are using various flavors of parallel algorithms--Strassen vastly increases the communication overhead. Not to mention that the largest matrices are probably using sparse matrix arithmetic anyways, which is a whole different set of algorithms.
AFAIK the best practical matrix multiplication algorithms scale as roughly N^2.7 which is close enough to N^3 to not matter for the point that I'm trying to make.
I guess it depends on what games you play. I have an AI Max 395 (Framework Desktop) and it runs every game in my library flawlessly. I'm sure if I played this year's most resource-intensive games it might stutter, but I don't. For me, it's an amazing low power minpc doing triple duty as a gaming PC, development box, and running my self-hosted services for the rest of the house.
Yeah. I have one. People buying the Framework Desktop mainboard aren't buying it just for gaming. There are better and far cheaper options for gaming. What this does is everything, though. Good enough for 1440p gaming. 16c/32t powerful CPU, it can run LLM's. SFF main pc that can do everything in a tiny space is a win.
As far as I can find, Plex does not support AMD iGPU for transcoding. Jellyfin will work, but support seems rather spotty. For other AI/ML work, it seems like ROCm is up and coming, but support - e.g. for Frigate object detection - is still a work in progress, especially for newer chips.
Jellyfin supports it, but the resulting quality is noticeably poor compared to Intel QuickSync or software transcoding. Perhaps the newer chips are better, but if you're building a media server from scratch you'd probably build around an Intel CPU or ARC GPU anyway.
Is it actually using the iGPU, or just "brute forcing" it?
I've put it in quotes as the effort required from these chips for streaming transcoding is so low these days that brute force makes it sound like more effort than it really is.
Thats interesting. My 5 year old Ryzen laptop can transcode 4k faster than realtime, which is what I mean mean by "these chips". Modern Ryzen, which is what the subject is about.
Quick Sync is invaluable for low powered processors, my old Intel embedded Wyze can do several streams.
"faster than realtime" doesn't mean much if it's in a device that's supposed to do more than just transcoding (such as serving a web app) or if you need multiple transcodes, etc.
Even on modern chips, transcoding is quite expensive.
People who are running Plex generally are running on servers also serving files, web apps, and who knows what else. These devices are often running 24/7, so both overall cost and power efficiency are big concerns. I wouldn't want to rely on my server being at high CPU usage most of the time - for power, heat, and overall reliability concerns.
PC desktop chips have 128 bit busses to right ? From what I know theoretical maximum memory bandwidth of chips is less than 100gbps - which is less than like base M5/M4 chips.
So no matter how much compute you stuff in there it's going to be shit for AI ?
PC architecture is not adapting to AI workloads at all, and no signs of that changing in years to come. I would not be surprised if your phone was more capable of running AI models than an average desktop - especially given gpu pricing.
Apple has their high-bandwidth chips, the rest of the commercial desktop market is effectively running Windows, and Microsoft has no incentive to move towards local AI, their ideal case is that you use their cloud-based services and pay for them forever (you being enterprise clients with thousands of PCs).
Putting Strix Halo into the AM5 socket would make no sense. Half the memory controllers would be orphaned and the GPU would be severely bandwidth-starved (assuming that the memory controller on Strix Halo actually supports DDR5 and not just LPDDR5).
Yeah the next generation of Strix Halo is what would get me excited. I think right now TSMC has no capacity, so maybe we have to wait another year. Kinda ironic that all CPU/RAM capacity is being sold to LLM companies, and as a result we can't get the hardware needed for good local LLMs.
> all CPU/RAM capacity is being sold to LLM companies, and as a result we can't get the hardware needed for good local LLMs.
yeah... Ironic I guess. It's as if they've realised that it's only a matter of time until we get a "good enough" FOSS model that runs on consumer hardware. The fact that such a thing would demolish their entire business of getting VC hyped while giving out their service for a loss surely got lost to them. Surely they and Nvidia have not realised that the only thing that could stop this is to make good hardware unreachable for anything smaller than a massive corp
Mark my words: in less than one year, we'll probably get something akin to Opus 4.6 FOSS. China is putting as much money into that as they can because they know this would crash the US economy, which is in the green only thanks to big tech pumping up AI. China wants Trump either gone or neutered as soon as possible, which they know they can do by making Republicans as unelectable as possible - something that will probably do if the economy crashes and a recession happens
Since this is for desktop, the NPU is irrelevant. Consumer-grade NPUs are not made for high performance. They are optimized for low power consumption, which makes sense when you are trying to run basic AI tasks on a laptop without turning it into a frying pan. On consumer-grade gear, the GPU will outperform the NPU, and since a desktop is not constrained by battery and can dissipate far more heat, the NPU is mostly irrelevant (aside from using it to develop software for laptop NPUs).
The Ryzen AI line is actually great if deployed to an entire org as the bottom tier, as it garuantees every device has a 50 TOPs NPU. We deploy local software at $STARTUP and this makes deployment to a Windows corp more predictable.
> This makes them AMD’s first desktop chips to qualify for Microsoft’s Copilot+ PC label, which enables a handful of unique Windows 11 features like Recall and Click to Do.
Microsoft: "Friendship ended with Intel, now AMD is my best friend"
Actually it is Qualcom, as they keep trying to push for ARM, but due to the way PC ecosystem has been going since the IBM PC clones started, no one is rushing out to adopt ARM.
> This makes them AMD’s first desktop chips to qualify for Microsoft’s Copilot+ PC label, which enables a handful of unique Windows 11 features like Recall and Click to Do.
This is not the selling point they think it is.
The problem I see with the AM5 socket is simply the fact that DD5 RAM to support it is just too expensive. So this will not really make the big impact they were hopin for.
I have a hard time believing ANYONE thinks this is a selling point. Literally anyone, including marketing and execs at Microsoft. I think they have just sunk too much money in it to quit, so they keep doubling down.
"The problem I see with the AM5 socket is simply the fact that DD5 RAM to support it is just too expensive."
DDR4 is basically just as expensive. At least DDR5 gives on-chip error correction (not as good as full ECC).
For a geneal computer, there's not that much difference between AM4 and AM5 unless you really want the extra speed of DDR5, PCI 5, and the newest processors. You can build a very capable AM4 machine for slightly less money, but that savings is found on the CPU and motherboard, not on the RAM.
The ECC in DDR5 is there just to make it work because of the errors caused by the density and the data rates. The ECC isn’t there for you, it is for the manufacturers.
I am still running DDR2 & DDR3 machines! I was going to finally make the big upgrade this year but am now holding off until the market finds a little bit more sanity.
DDR4 looks to be around half the price of DDR5 on the used market to me. I wouldn't call that slightly less money unless you weren't planning to install much RAM.
On the new market, they are really close. Most consumers are buying new. If we want to get pedantic, we can compare buying used systems on Facebook marketplace to cannibalize the parts and resell the others to see net cost.
I don't think most people building AM4 systems currently are buying new, or at least not everything new, simply because depending on what you're looking for there might not even be any new parts.
I think for brand new computers/builds that's correct but where it hit me was wanting to upgrade an existing desktop. I already have more DDR4 RAM than I need and would have been willing to purchase a new CPU/motherboard and being forced to also purchase new RAM at the same time made it too big of a price tag all at once. I just found the best zen 3 cpu I could on ebay and called it a day.
I think your point still stands overall for AMD's business though, I assume a vast majority of CPUs are purchased in new desktops?
I built a workstation / gaming pc in 2024, and I feel like I was on the last chopper out of 'nam
Agree, i just built a desktop for the first time in ages, it is a leap and change from using laptops with numerous components pplugged into them. i made the leap to desktop. everything was comparably reasonable except the RAM or anything that has memory chip on it ( RAM, NVME etc) so i did some research just to make sure. All in all i happy with the result i went with AMD 9900x no graphics card in this option, i skipped the graphics card for now.
I would like to add that, looking for bundles helps a lot. If you have micro center near you, utilize it to you full advantage they are the only ones given promotional items with bundles at the moment from what seen. The main objective is to skip the price gouge of RAM chips, they cost more than the CPU at the moment. I got CPU and motherboard plus 32g RAM fro $600 and that was a save. the RAM was $445 alone.
If I can use it with Linux in any meaningful way, that would be a better selling point.
You in fact can now! In the past week, a transformer framework called FastFlowLM [0] supporting XDNA 2 NPUs officially started supporting Linux.
I posted it here the same day I found and started using it, to almost no reaction.
[0] https://github.com/FastFlowLM https://fastflowlm.com/ https://huggingface.co/FastFlowLM
> to almost no reaction.
HN is overloaded with AI stuff, its hard to break through all the noise. I say this as someone very interested in AI. Even I skip some links because its just too much.
I see it making claims about 10x efficiency, but how is tokens / second / watt? The only machines I've seen with the memory bandwidth to effectively do local inference are Mx arm chips on mac.
because it's not faster than the Ryzen 395's GPU. power efficiency doesn't matter as much as TTFT for desktop users, especially when they're tasking their AMD box as a dedicated inference machine.
some older pre-395 AMD articles suggested it'd be possible to use the NPU for prefill and the GPU for decoding and this would be faster than using either alone, but we have yet to see that (even on Windows) for any usefully sized models, just toys like LLaMA-8B.
Now all the features you don't use can perform 20% faster!
Upgrading to AM5 wasn't compelling to me even last summer before things went bonkers; I'm still very content with my 5800x and 64GB of DDR4.
Trying to take the plunge on that now sounds like a nightmare.
I got lucky on the timing and got 9800X3D + 64GB DDR5 before prices increased.
Your machine is sweet and probably runs just as fast on most tasks. I wouldn't be in n a hurry to upgrade.
I upgraded from an old Intel i7.
To really take advantage of those gpu cores you need memory bandwidth. Modern transformer based LLMs are really bandwidth hungry. I am really happy to see this first push. NVIDIA having discrete GPU/memory/etc is an option, but not great for a lot of different reasons. Unified memory architectures like what AMD and Apple have are the way to go for the future. Put 256GB of ram on the main board and be able to access it at speed for LLM use please.
Just like the previous generation of AI PC, consumers just need a usb/pcie NPU,
Mass adoption won't happen until we get those cheap, because there are no mass prosumers making software for them that is massively popular.
No, AI inference is mainly RAM/RAM speed constrained, we need more fast RAM to make local AI thrive.
Lol. Thanks to someone buying all the ram platters, before they became modules, that won't happen.
Meanwhile, the corresponding "non-standard" desktop PC is the Framework Desktop, which with the Ryzen AI Max+ 395 can use 120GB of its 128GB RAM for the GPU: How to Run a One Trillion-Parameter LLM Locally: An AMD Ryzen™ AI Max+ Cluster Guide https://www.amd.com/en/developer/resources/technical-article...
Minisforum MS-S1 is the same chip but has a PCIe slot suitable for a network card.
The Framework Desktop motherboard does actually have a PCIe x4 slot, their case just doesn't expose it for whatever reason. But you can buy the board separately and put it in your own choice of Mini-ITX case which does.
This case is cute and well built but it does have strange quirks, see also the noisy PSU [1]
[1] https://community.frame.work/t/noisy-psu-fan/74751
First I've heard of this thread (I haven't spent much time on the forums for a while), but my Framework Desktop definitely has the PSU noise issue, and it doesn't seem correlated to system load. Otherwise a really solid machine.
That PCI slot has low power output, not suitable for what people would plug into it. Easier to cover it than run into support problems. It won't run a GPU directly. It will run an Oculink card which will allow use of an egpu.
I can't imagine why you'd want to buy a beefy SOC with unified memory, only to have it host a discrete GPU through it's narrow PCIe x4 interface. You'd be better off with a traditional CPU that supports a proper x16 slot.
If you want to try both "Team Red" and "Team Green" stuff without having 2 hosts to manage, the x4 interface is a reasonable tradeoff.
I saw that, it's pretty strange.
Or a Beelink GTR9 Pro with same chip and memory, and two 10GbE LAN ports built in for clustering several together.
The MS-S1 also has 2x 10Gb Ethernet, the value of the slot is going to a card with more bandwidth + RDMA support.
You need five network cards per node to make proper hypercube. Don't settle for less!
A really good switch will do, too.
In their article, AMD chose to use the built-in 5Gbe NIC, but I read you could use the two USB4 @40Gbps ports as interconnects too (or for USB NICs).
[flagged]
They're a small shop compared to the big laptop makers and their focus is on user-repairability. Between those two you are naturally going to have a slightly higher price point than simply buying a laptop from one of the usual companies. I wouldn't call it junk or gimmickery. It has a purpose and a niche. It might not be your niche, but it is for some people.
What are you doing here? This is both an emotional and yet unsubstantiated comment. Would you like to try again?
Why do you think that? I'm looking at getting one.
Yeah, but all of this is pointless when RAM is as expensive as two CPUs by itself - if it's even in stock. AMD/Intel should focus on that first if they want to save their DIY business at all - which I'm starting to doubt they don't
The RAM shortage is... a shortage. It's temporary by nature. RAM didn't suddenly get 4x more expensive to produce, it's just in high demand right now. Supply will eventually catch up even if it takes a few years.
Most of us are not really doing detailed half-decade hardware purchasing plans, for most of us "the next few years" is really the only relevant time frame.
I have a 9 year cycle for my main machine. Not sure what's yours?
Right now I'm on a 2021 MacBook Pro, I'm debating whether to upgrade to the new M5 Pro laptops or to wait until next year; so that's 5-6 years. My desktop gets upgraded over time, so it's not really on a "cycle"; it's more that every few years, I may decide to get a more powerful CPU, or more RAM, or a new GPU, or more storage, or whatever else. Though I recently (as in, during the past year) upgraded from an AM4 CPU to an AM5 CPU, so that meant I replaced more than usual; everything other than the GPU, storage, power supply and peripherals. (I switched from ITX to mATX for an extra PCI slot so the case had to go; though it still lives on in the form of a lab PC of sorts, along with my old i7 6700k)
But common among all these replacements is that they're not really planned in detail years before. I may weigh up factors like "I don't really need an upgrade right now" against factors like "the market looks like it'll probably get really shitty later this year", or "I could really really use an upgrade right now" against "but the market is shitty now and we're right before a product launch which will shake things up". But I always react to my current or near-future needs/wants and current or near-future market conditions.
So hearing "the RAM market will be good again in 5 years" is completely irrelevant to me. My decisions are entirely based around how the RAM market is right now and how I believe it will look throughout the year or the next.
When someone is in the market for a new computer, they rarely look more than months ahead, no matter how long your cycle is.
> Supply will eventually catch up even if it takes a few years.
I am wondering if this so true. What resources and time are needed to increase supply by N times to catch demand.
The resources are certainly there; the high prices are providing them. It's just a matter of time.
Supply will catch up to demand when demand goes down.
Also supply will only increase if producers believe the current level of demand is going to be sustainable in the long term. Which I don't think anyone really believes.
The same way nobody wanted to invest in mask supply in the US or Western Europe during covid, because producers knew the demand spike wouldn't last and they'll be left with useless equipment to pay after the crisis passed.
> demand is going to be sustainable in the long term. Which I don't think anyone really believes.
I am one who think there is a chance it will sustain. AI is useful tool, Opus unlocks N times productivity gain for devs since Opus 4.5, which is available just for 3 months.
This means adaptation is just started, it could expand on all kind of usecases, niches, solving problems, products, etc, which could be N times more demand for compute from what we have now.
Correct, but if they're right in that belief then it's still just a matter of time before demand goes down and the shortage is solved.
If they're wrong and this is actually a permanent spike in demand, then it'll take the industry a while to realize it but eventually they'll collectively figure it out and increase supply. The ones who figure it out soonest and increase supply fastest will profit the most. The ones who figure it out slowest will lose market share.
Or they'll just enjoy the high margins and not invest significantly more than they normally would in new production capacity.
Or they're not convinced that betting on continued hypergrowth of AI is a good idea.
And thereby leave a bunch of profits on the table while simultaneously losing market share to competitors who do invest in more capacity?
Which competitor is going to invest in increasing their capacity when everyone expects the demand to decline sooner rather than later?
If demand declines then the shortage is still solved regardless.
If it doesn't decline, than anyone who took that risk and increased their production capacity will benefit greatly, and those who didn't will lose market share.
They're in business to make money not to solve the shortage. If they invest in new fabs and the bubble pops they're sitting on idle fabs.
And if they invest in new fabs and the bubble doesn't pop then they make a whole lot of that money they're in business to make.
The incentives here are naturally very well aligned with solving the shortage. If doing nothing is likely to solve the shortage, then they'll do nothing. If increasing supply is likely to solve the shortage, then they'll increase supply. If there's a 50/50 chance of both, then some will increase supply and some will do nothing, and the market will reward whichever group was right and punish the other.
You forgot to say “Amen” at the end of your gospel.
I'm always baffled how seriously some people take that “market and incentives always lead to the greater good” religion despite plenty of empirical evidence against it.
But hey, there are creationists out there too so it's not too surprising I guess.
China is coming online :(
They can't fix stupid.
Let me describe this in the most simple terms possible: You have speculators speculating about AI products. The speculators are not very smart when it comes to technology, and think RAM is RAM. There is at least three kinds of RAM that are important to this: DDR for system RAM, GDDR for GPUs, and HBM for high density enterprise products, and they are not interchangeable, there is no one-die-fits-all solution.
So, these speculators are like "oh no, more GPUs requires more RAM!", and then just start speculating on all RAM. Which of these RAMs are the ones that they need to worry about? Exclusively HBM, which is a minority in production, DDR and GDDR dominate production.
If you're into inference, and have older machines, you're buying Hxxx or Bxxx cards that use HBM, fit into dual slot x16 configurations, and you're jamming (optimally) 8 of them in. If you're into hardware that is newer, somewhere in the middle of the inference boom, you're using MXM cards. In either situation, the host machine has DDR, but if you're OpenAI, Anthropic, Microsoft, or Google, you're not building (more) inference machines like this.
The first two are buying Nvidia's all in one SBC solution: unified HBM, onboard ARM CPU to babysit the dual GPUs, has its own dual QSFP network controller that can RDMA, etc. No DDR or GDDR involved. Any machines built before this platform are being phased out entirely.
Microsoft is doing the same, but with AMD's products, the MI series that co-locates Epyc-grade Zen 4/5 CCDs with CDNA compute chiplets, running the entire thing off HBM, thus also unified and no DDR/GDDR needed. They, too, are phasing out machines older than this.
Google has a mix: they offer Nvidia all in one SBCs as part of GCP for legacy inference tasks (so your stack that can't run on AMD yet still can run), but also offer the same MI products that Microsoft offers via Azure's inference product, but also has their own TPUs that some of Gemini runs on; the TPUs run on HBM afiact. No DDR or GDDR here.
So, what does AMD or Intel do here? Lets say they waste fab time to make their own dies on the wrong process (TSMC and Intel-Foundry do not have for-RAM optimized processes)... they would be producing DDR and GDDR for a market that almost has its entire demand met. Intel lacks the die stacking technology required to build HBM, and TSMC I think can't do it for that many layers (HBM has 8 to 16 layers in current gen stuff iirc).
Micron, for example, already is bringing two large factories online here in the US to meet the projected growth in demand for the next 20+ years. When these factories finally start producing, it will not change the minds of speculators: they still seem to think AI datacenters need RAM, of any kind, and refuse to understand even the most basics of nuance. Also, when they come online, HBM will be a minority product; the AI inference boom is still just a bump in the road for them.
Nvidia kinda screwed their consumer partners, btw: they no longer bundle the GDDR required for the card with the purchase of the die. There is a slight short term bump in GDDR spot prices as partners are building up warchests to push series 60 GPUs into production, and once that is done, spot prices return to normal (outside of the wild speculation manipulation).
One last thing: what about LPDDR, used by AMD Strix Halo and Apple stuff? Speculation seems to have not actually effected it. I consider it as a sub-category of DDR (and some dies seem to work as either DDR or LPDDR as of DDR5, due to the merger of the specs by JEDEC), but since it isn't something you find in datacenters, it seems to have avoided speculation.
The Ryzen Max CPUs mentioned in the linked article? Uses LPDDR. Doubling down on the Ryzen Max product line might be a brilliant move.
> The speculators are not very smart when it comes to technology, and think RAM is RAM. There is at least three kinds of RAM that are important to this: DDR for system RAM, GDDR for GPUs, and HBM for high density enterprise products, and they are not interchangeable, there is no one-die-fits-all solution.
The commenter is also not very smart and does not realize companies making the RAM can trade capacity of one for another and any re-tooling at current price is still profitable.
The commenter also does not realize that is also true for lines currently making SSDs
They can trade capacity, but they generally aren't. The huge storage-only fabs owned by Samsung and Micron do runs that go for 9 months to 12 months.
Flash chips haven't been speculated on nearly as hard, and are suffering from the same sort of weird lack-of-nuance. Samsung, for example, isn't reassigning capacity to meet some sort of phantom datacenter demand that isn't already there, generically, across all datacenters, AI or not.
A lot of SSD price skyrocketing is largely "SSDs have RAM on them for cache", not "SSDs have flash chips, and they're both made at the same fabs"... which oddly effects low end SSDs that don't have external cache.
To make it worse, for the speculators who do understand this, because it isn't some universal homogeneous group, the flash chips that go into enterprise SSDs aren't the same that go into consumer SSDs.
The Big Three still aren't doing some major re-tasking of capacity, as the actual global demand isn't outstripping supply any more than normal. There is no short term problem to fix, speculators are just gonna have to stop hoarding toilet paper like its the start of Covid.
Edit: Oh, and if you want to ask how AMD/TSMC or Intel solve this? They can't, same reason why making their own in-house HBM isn't happening.
Both Western Digital and Kioxia have reported their 2026 Flash/Hard drive production capacity is sold out.
Micron killed Crucial to focus on AI.
I'm glad Kioxia (formerly Toshiba) have been able to do that. However, I also know they've been having problems meeting demand for quite some time, and haven't been able to scale up nearly as fast as the big three have. There was an incident in 2019 and another in 2022 that killed entire runs of chips and screwed them during the Covid datacenter rush.
Micron killed Crucial because Crucial was a weird offering that competed with their own partners. This was always a weird problem, and it just didn't make financial sense to continue with it. One of the analyses I read was Crucial was less than 12% of sales.
Like, don't get me wrong, I've liked many Crucial products over the years, and even recommended some of them, but it was always weird they were trying to out-compete companies like Adata and other major ODMs.
The counterexample of this is Nvidia absolutely trying to kill their partners, and going to first party assembly and sales of products. Nvidia isn't even going to PNY anymore for ODM needs, but going directly to Foxconn.
Micron execs claiming its because of AI is a bit weird and revisionist, because they've been working on exiting the Crucial brand since long before they publicly announced it. The public didn't learn of any such plans until right before the Ballistix brand sunsetting was announced in 2021, but started years before that. Like, I know they're just playing to their shareholders, but its still a bit weird.
When did nvidia drop PNY as ODM for their reference cards? I recall my A5000 (now 2 Gen old) was made by PNY.
As far as I know, the current lineup is PNY still makes the workstation cards, possibly also the x16 server cards, but Foxconn is doing the Blackwell SBCs and MXMs, and those SBCs are a pretty big chunk of Nvidia's income right now. I also believe they have moved to Foxconn for the Founders Edition consumer cards.
Also, with the FEs, their partners are disallowed from making their own FEs, even if they make their own PCB from scratch and not based on any existing Nvidia design. Doesn't matter who makes the FE, it immediately puts partners at a great disadvantage if they can't make one too.
Samsung and SK Hynix have moved all of their capacity over as well IIRC.
Reminds me of "false sharing" effect: hidden common dependency and bottleneck for what looks like independent variables on the surface.
So where are all these speculators storing DDR5, flash, and even spinning hard disks? Asking for a friend.
As a small buyer of all of those things supply at nearly any price has gotten very difficult to reliably predict week to week. When a lot of 100 64GB DDR5 sticks shows up available at a vendor, it’s at a take it or leave it price good for a couple hours. If I don’t pull the trigger they have another buyer for it and I might be waiting another month before anything becomes available again. We can no longer JIT for even failure replacement on our edge nodes.
Then you have the NVMe and even SATA SSD shortages. Still a bunch of very useful hardware out there I would love to find a decent deal on 8TB sata so I could repurpose it. Just doesn’t make any sense right now at current pricing and availability. Good luck trying to even find a batch of 12 of these disks at a time.
This goes for both enterprise and even prosumer I was willing to take for some of these uses.
Those micron factories won’t even be targeted at consumer-grade RAM though, right?
There is no such thing as "consumer grade RAM". Servers still take DIMMs, ECC DIMMs just has more chips on it (previously 9 instead of 8, but now 10 instead of 8 as of DDR5; you'll see some DDR5 DIMMs with 5 instead of 4 because they're double die packages).
Micron, Samsung, and Hynix just basically sell you chips that comply with the JEDEC spec, and the DIMM manufacturers further bin them according to purpose. The highest end chips (that are stable at high clocks and acceptable voltages) end up in enthusiast performance products, the ones that don't work well at all but still meet JEDEC spec are sold to Dell/HP/Lenovo/etc for Grandma's Facebook machine, and the ones that are exceptionally stable at thermal design limits are plunked onto ECC DIMMs and sold to servers.
Also, as others have mentioned, its just a fab, and it can make any of the dies they're able to make. Whatever needs to be made to meet demand, they make, they just can't turn on a dime and react to quarterly concerns, and are locked into cycles that may range from 6 months to 18 months.
Side note that is also worth mentioning, sometimes you can order special bins of parts with features that wouldn't normally be available if you're willing to order enough. Recent example being Nvidia buying overclocked GDDR6 chips from Micron with additional features enabled; Micron was more than happy to become Nvidia's exclusive supplier for the custom GDDR chip if Nvidia was willing to buy out the entire run. Stuff like this happens every so often, but isn't the norm.
If there’s no such thing as consumer grade RAM, then what did Micron’s announcement mean?
https://investors.micron.com/news-releases/news-release-deta...
The announcement means that they're closing Crucial - just like it says in the title and the first paragraph. The rest of that press release is outlining the mechanics of how that works + some fluff. Micron is going to continue producing the exact same memory chips in the exact same fabs. They're just not soldering it to a board, slapping the Crucial logo on it, and selling it directly to consumers. There's nothing stopping downstream vendors from buying Micron chips, soldering them to a board, and selling them to consumers as Micron was doing previously.
There's nothing in that press release that implies that the memory was somehow different (or "consumer-grade"). The _only_ thing they're saying is that they're ending their B2C business and focusing on B2B.
Didn’t you just describe the literal difference between consumer grade RAM that is soldered to a consumer format board vs a memory chip sold to a company to be soldered onto a product of the other company?
Re-read the previous comment again.
You just need an additional chip to move from "consumer grade" (ie no parity) to "server grade" (ie have parity). ECC support is actually in the memory controller which is in the CPU for the last 15 years. No magik.
> what about LPDDR, used by AMD Strix Halo and Apple stuff? Speculation seems to have not actually effected it
Good luck actually finding them on stock with 128GB+ RAM. I got strix laptop while ago, now price in EU is technically the same, but no stock. Maybe month or three
There is also claw hype. And large gwen3.5 models can run very well on DDR5 CPUs or mac minis...
I find the panic over RAM prices to be overestimated. 32GB DDR5 RAM is around $500 which is comparable to to the 9800x3D. Sure it sucks that it increases by around 4x, but when you factor in the overall price of a top end PC at around 1000-2000, especially for the lion's sum of the GPU, the increase is marginal.
This only effects a very narrow slice of highly budget conscious consumers trying to build high end PCs at razor thin margins.
$500 for 32GB is about $15/GB which is a high we haven't seen since the mid-2000s. This is a big deal, it turns RAM and to some extent storage (especially fast storage) into a massive economic bottleneck.
> since the mid-2000s.
Did you adjust for inflation ?
Adjusted for inflation, the last time prices (/GB) were this high was May 2011; the tail end of the 2009/2010 shortage. Aside from a brief glut in 2008, it wasn't really cheaper before (than it is now) though. Of course RAM is much faster these days, but also in 2011 most people had no more than 4 GB of system memory and 512 MB VRAM.
https://web.archive.org/web/20240805053759/https://jcmit.net...
https://thememoryguy.com/dram-prices-hit-historic-low/
Inflation applied manually; https://www.bls.gov/cpi/
https://www.neowin.net/forum/topic/983036-latest-steam-hardw...
Steam hardware survey GPU history: https://www.youtube.com/watch?v=wHTdnIviZTE
No, I didn't adjust for the huge inflation in average RAM requirements since the mid-2000s.
Not the point - my point utterly of arithmetical nature - dollar has substantial inflation, and any comparison more that 5 years apart, let alone 20 warrants adjusting of prices, as error is substantial, 2x in the case of 2000s
Inflation since the 2000s cannot possibly make up the difference in price we’ve seen in just the last 6 months.
That was not my point entirely; my point that citing prices from 2000s and comparing with modern ones |(with indexing about 2x times), regardless of underling reason is either a demonstration of lazyness or innumeracy, or even worse - an attempt to manipulate.
It’s not laziness, innumeracy, or manipulation when it can be taken at face value that the cost increase vastly outstrips anything that could be attributed to inflation. You don’t even need to look it up to know that.
> when it can be taken at face value that the cost increase vastly outstrips anything that could be attributed to inflation
But that was not my point _whatsoever_. What I said is - every time you bring the explicit numbers (like in GP "$500 for 32GB is about $15/GB which is a high we haven't seen since the mid-2000s") you _absolutely_ have to adjust for inflation to have a meaningful conversation. This is it.
Ram is clearly way more expensive now, yes?
Did you adjust for technological improvements that pumps out more chip per wafer compared to mid-2000s due to node-size shrink?
That was not my point entirely; my point that citing prices from 2000s and comparing with modern ones |(with indexing about 2x times), regardless of underling reason is either a demonstration of lazyness or innumeracy, or even worse - an attempt to manipulate.
> 32GB DDR5 RAM is around $500 which is comparable to to the 9800x3D.
Apples to oranges. Why are you comparing RAM prices to CPU prices? It's different hardware.
$500 for 32 GB is insane. Just 18 months ago, I bought 128 GB of DDR5 for only $480.
I disagree with you. The issue does not only affect a “very narrow slice” of consumers. https://www.techspot.com/news/111472-hp-warns-ram-now-makes-... A major brand is now suggesting that this is a “new normal” and one solution is to just offer systems with less ram. This is an issue when lots of modern software seems to expect an unending supply.
That is an insane amount of money for just 32GB of RAM! That's what we were paying back when it was hard to use more than 32-64GB in a desktop setting. These days with all the electron and node bloatware, containers everywhere and AI - 32GB doesn't get you far.
$500 is 5x what it cost less than a year ago, just for context. It turns a $1600 computer build into a $2000 one. That’s a huge difference.
Edit: I don’t get your math. If we’re using a very generous definition of “top end,” even neglecting Nvidia and going AMD - which some would argue makes it not top end - you’re talking conservatively: $600 for a GPU, $500 for 32gb of ram, and $500 for a CPU. $1600 before PSU, case, SSD, fan(s), mobo…there’s no world in which you’re coming in under $2k. The SSD and board will put you over immediately.
You’re talking 3/2025 prices, not 3/2026. A compromise, mid-range computer is $1500 to build now.
Look at my comment above and see what i said about my build. I was unwilling to pay that for a moderate build that can sustain my computer use. Utilize bundles that can save you $ on RAM or chipset, CPU to skip some of the costs.
You aren't specific in your comment. Where are these bundles? What do you do with all the parts you don't need/end up swapping out? How much are you actually saving?
>"overall price of a top end PC at around 1000-2000"
All 4 of my "top end PCs" have 128GB RAM. Me server (I self host everything is 512GB). Lucky for me all were bought before that insanity.
In time for Windows 12 that reportedly will require an NPU: https://tech4gamers.com/windows-12-reportedly-relasing-2026-...
That's a ridiculously implausible and sensationalized rumor. At most, Microsoft may make a NPU a requirement for OEMs to use the Windows 12 logo on new PCs. Actually refusing to support the existing install base of recent and hughly-capable desktops is not at all likely. It would be far more drastic than the hardware deprecations brought by Windows 11, which were already quite controversial (and loosely enforced).
Hoe much dedicated cache do these NPUs have? Because it's easy enough to saturate the memory bandwidth using the CPU for compute, never mind the GPU. Adding dark silicon for some special operations isn't going to make out memory bandwidth faster.
Does a cache help with inference workloads anyway?
I don't know much about it but my mental model is that for transformers you need random access to billions of parameters.
It's streaming access, and no not as far as I'm aware. APUs have always been hilariously bottlenecked on memory bandwidth as soon as your task actually needed to pull in data. The only exception I know of is the PS5 because it uses GDDR instead of desktop memory.
Are we going to see more memory channels for consumer desktop at some point from AMD or Intel? Apple seems to be the only one that offers it.
Am6 Socket isn't due for a couple of years. The current sTR5 used by Threadripper processors supports up to 8 channels of memory.
I'm not sure what you mean - I think the mobile 300 series can do quad channel already for its APU at least. I'd assume it can do more but do you not need more slots beyond that?
There are a single-digit number of products using the AMD Strix Halo mobile parts that have a 256-bit memory bus. All other mobile x86 processors (including AMD's mainstream Mobile silicon used for these desktop processors) have the usual 128-bit bus.
I'm talking about desktop computers, not mobile.
NPUs are more useful for prefill than decode anyway. Memory bandwidth is not the bottleneck for prefill.
What exactly is this:
Is this fast GPU like instructions that anyone can use in any operating system to run any open sourced LLMs on CPU with all your RAM rather than on a discrete GPU with its own limited amount of VRAM?
Or is this a proprietary thing that only works in Windows for some specific use cases and irrelevant for Linux users?
it's a distinct piece of hardware based on AMD XDNA architecture, which, coincidentally, much like CPUs, can tap into your RAM pool. there are XDNA drivers (`amdxdna`) for Linux.
> can tap into your RAM pool
lol no it can't - there's a small (40MB) SRAM that can DMA to DRAM and then each of the tiles is another DMA away from that SRAM.
AMD marketing is hoping the “AI” branding is a positive. Antidotally, I know many consumers who are not sold on AI. This branding could actually hurt sales.
Well, as long as it's one product of many, they may be able to segment the market and suffer no drawback.
We are dealing with a hype, but the reality is that AI would change everything we do. Local models will start being helpful in [more] unobtrusive ways. Machines with decent local NPUs would be usable for longer before they feel too slow.
> the reality is that AI would change everything we do
Your true believer convictions don't matter here. Those AI accelerators are merely just marketing stunts. They won't help your local inference because they are not general purpose enough for that, they are too weak to be impactful, most people won't ever run local inference because it sucks and is a resource hog most can't afford, and it goes against the interests of those scammy unprofitable corporations who are selling us LLMs as AI as the silver bullet to every problem and got us there in the first place (they are already successful in that, by making computing unaffordable). There's little to no economical and functional meaning to those NPUs.
> most people won't ever run local inference because it sucks and is a resource hog most can't afford
You have fallen headfirst into the "Not now, so never" fallacy. As if consumer hardware won't get more powerful, or models more economical.
> You have fallen headfirst into the "Not now, so never" fallacy.
Perhaps. Though we have empirical evidence of how much we can quantize and distillate models to the point of practical uselessness. That sets a bar for how large a local model needs to be for general-use as to compete with the could ones. We are talking in the area of 60GB for GPT-OSS/Qwen3.5, which is what enthusiasts are running on 32GB DDR5 + 24GB VRAM RTX 3090.
> As if consumer hardware won't get more powerful
Now I will let you, with that last fact in hand, plot a chart of how much it's been costing to provision that over the past 2 years and use it to prove me wrong about the affordability of local models.
> most people won't ever run local inference because it sucks and is a resource hog most can't afford
a) Local inference for chats sucks. Using LLMs for chatting is stupid though.
b) Local inference is cheap if you're not selling a general-purpose chatbot.
There's lots of fun stuff you can get with a local LLM that previously wasn't economically possible.
Two big ones are gaming (for example, text adventure games or complex board games like Magic the Gathering) and office automation (word processors, excel tables).
It surprises me that semantic search never gets mentioned here.
If you can use the NPU to process embeddings quickly, you get some incredible functionality — from photo search by subject to near match email search.
For consumer applications that’s what I’m most excited for. It takes something that used to require large teams, data, and bespoke models into commodity that any app can use.
> There's lots of fun stuff
Ask your friends or a small business owner if they are going to spend $1k on a new laptop because "there's lots of fun stuff".
For office automation, you'll get a lot more mileage with Claude and similar.
> Ask your friends or a small business owner if they are going to spend $1k on a new laptop because "there's lots of fun stuff".
Do people not buy gaming PCs and game consoles? Isn't that buying something because "there's lots of fun stuff?"
And while sure a business owner wouldn't be buying it for "fun stuff", if it was about being able to run the AI tools they want without the business risk of sending your most important data and intellectual property to an AI provider wouldn't some think about it?
> Local inference for chats sucks.
/r/SillyTavernAI would disagree with you.
Many people who use ST have a "serious" nvidia card.
We are talking about NPUs here.
Are you kidding? A good ratio of ST folks run finetunes of Mistral Nemo (if it tells you anything). Anyway your core statement is simply wrong ("local chat sucks").
From their own GitHub:
> If you intend to do LLM inference on your local machine, we recommend a 3000-series NVIDIA graphics card with at least 6GB of VRAM, but actual requirements may vary depending on the model and backend you choose to use.
Also, please be respectful when discussing technical matters.
P.S. I didn't say "local chat sucks".
> we recommend a 3000-series NVIDIA graphics card with at least 6GB of VRAM
...which is not by any means a powerful GPU, and besides the AMD Ryzen AI CPUs in question have a plenty enough capacity to run local LLMs esp. MoE ones; with 3b active MoE parameters miniPC equipped with these CPUs dramatically outperform any "3000-series NVIDIA graphics card with at least 6GB of VRAM".
> please be respectful when discussing technical matters.
That is more applicable to your inappropriately righteous attitude than to mine.
Am I reading /r/antiai?
Parent comment is fair and technically accurate.
Do you have a real argument, especially a technical one, that you can contribute?
> Parent comment is fair and technically accurate.
In what way precisely? That local LLMs "suck"? Is that a technical argument? Or this statement "there's little to no economical and functional meaning to those NPUs." - is that actual factual statement or a emotionally charged verbal flatulence? and what "they won't help your local inference because they are not general purpose enough for that" even means? People succesfully run largeish MoE llms on AMD Ryzen AI miniPCs.
> Do you have a real argument, especially a technical one, that you can contribute?
What kind of argument do you want me to "contribute" wrt the ideological rant the "parent comment" had managed to produce?
Hey, OP here with their (apparently) controversial views. I stand firmly within those lines:
- I shouldn't be paying more for my next CPU because it has a NPU that I won't ever use. Give me the freedom of choice.
- Given that freedom of choice, it would seem that a majority would opt-out (as seen recently by Dell), so the morals of all that are dubious.
- NPUs may not be completely stupid as a concept, in theory, but at this point in time they are proprietary black-boxes purpose-built for marketing and micro-benchmarks. Give me something more general-purpose and open, and I will change my mind
- …but the problem is, you can only build so much general-purpose computing in bespoke processor. That's kind of its defining trait. So I won't hold my breath.
- Re: local-inference for the masses, putting aside the NPU shortcomings from above: how large do you think a LLM needs to be so it's deemed useful by your average laptop user? How would the inference story be like, in your honest opinion (in terms of downloading the model, loading it in memory, roundtrip times)? And how often would the user realistically want to suffer through all that, versus, just hopping to ${favorlite_llm.ai} from their browser?
Anyhow, if that makes me "antiai", please, sign me up!
> I shouldn't be paying more for my next CPU because it has a NPU that I won't ever use. Give me the freedom of choice.
There is a plenty to choose from.
> - NPUs may not be completely stupid as a concept, in theory, but at this point in time they are proprietary black-boxes purpose-built for marketing and micro-benchmarks. Give me something more general-purpose and open, and I will change my mind
In fact the linked article is not talking about NPUs in particular, but about Ryzen AI cpus. These have unified memory and more compute compared to normal ones which make them very useful for inference.
> how large do you think a LLM needs to be so it's deemed useful by your average laptop user?
Depend what they need it for. Useful autcomplete in IDE starts at around 4b weights.
> loading it in memory
Happens only once, usually takes around 10sec.
> roundtrip times
Negligible? it is loca after all.
> And how often would the user realistically want to suffer
No suffering involved.
I am bullish on AI being used in all sorts of useful and discreet and non-discreet ways in the present and future. However I am exceedingly skeptical of NPUs being some winning bet.
No one is running LLMs on current gen NPUs so if we will in the future its a long time coming. Unless they can demonstrate some real (and not marketing) wins I remain skeptical that a large NPU for LLMs is the future.
I can totally see NPU accelerating simple tasks, but to be worth the silicon they have a ways to go imo.
99% of people don't need or want a dev workstation. My travel laptop is 7+ years old and I couldn't tell you the difference between it and a current flagship in terms of browsing and everyday tasks.
I will not lie, I find LLMs useful but the desktop experience is pretty polished already. NPUs seem to be an attempt to ride the AI bandwagon with very little to show for it so far.
For some people maybe. I don't want to use local AI and NPU will be dead weight for me. Can't imagine a single task in my workflow that would benefit from AI.
It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
> Can't imagine a single task in my workflow that would benefit from AI.
You don't do anything involving realtime image, video, or sound processing? You don't want ML-powered denoising and other enhancements for your webcam, live captions/transcription for video, OCR allowing you to select and copy text out of any image, object and face recognition for your photo library enabling semantic search? I can agree that local LLMs aren't for everybody—especially the kind of models you can fit on a consumer machine that isn't very high-end—but NPUs aren't really meant for LLMs, anyways, and there are still other kinds of ML tasks.
> It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
Do you insist that your CPU cores must be completely homogeneous? AMD, Intel, Qualcomm and Apple are all making at least some processors where the smaller CPU cores aren't optimized for power efficiency so much as maximizing total multi-core throughput with the available die area. It's a pretty straightforward consequence of Amdahl's Law that only a few of your CPU cores need the absolute highest single-thread performance, and if you have the option of replacing the rest with a significantly larger number of smaller cores that individually have most of the performance of the larger cores, you'll come out ahead.
I provide a data point
* You don't do anything involving realtime image, video, or sound processing?
I don't
* You don't want ML-powered denoising and other enhancements for your webcam,
Maybe, but I don't care. The webcam is good or bad as it stands.
* live captions/transcription for video,
YouTube has them. I don't need it for live calls.
* OCR allowing you to select and copy text out of any image,
Maybe
* object and face recognition for your photo library enabling semantic search?
Maybe but I think that most people have their photo library on a cloud service that does AI in the cloud. My photos are on a SSD attached to a little single board ARM machine at home, so no AI.
What I would like to be able to do is running the latest Sonnet locally.
In general I think that every provider will do their best to centralize AI in their servers, much like Adobe did for their suite and Microsoft did for Office so local AI will be marginal, maybe OCR, maybe not even blurring the room behind my back (the server could do it.)
There are alternatives to Adobe and Office, because I don't care about more than the very basic functionality: running Gimp and Libreoffice on my laptop costs zero. How much would it cost to run Sonnet with the same performances of Claude's free tier? Start with a new machine and add every piece of hardware. I bet that's not a trivial amount of money.
> You don't do anything involving realtime image, video, or sound processing?
Nothing that's not already hardware accelerated by the GPU or trivial to do on CPU.
> You don't want ML-powered denoising and other enhancements for your webcam
Not really.
> live captions/transcription for video
Not really, since they're always bad. Maybe if it's really good, but I haven't seen that yet.
> OCR allowing you to select and copy text out of any image
Yet to see this implemented well, but it would be a nice QOL feature, but not one I'd care all that much about being absent.
> object and face recognition for your photo library enabling semantic search?
Maybe for my old vacation photos, but that's a solid 'eh'. Nice to have, wouldn't care if it wasn't there.
Is everyone a content creator these days?
Besides, most of what you mentioned doesn't run on NPU anyway. They are usually standard GPU workload.
None of what I listed was in any way specific to "content creators". They're not the only ones who participate in video calls or take photos.
And on the platforms that have a NPU with a usable programming model and good vendor support, the NPU absolutely does get used for those tasks. More fragmented platforms like Windows PCs are least likely to make good use of their NPUs, but it's still common to see laptop OEMs shipping the right software components to get some of those tasks running on the NPU. (And Microsoft does still seem to want to promote that; their AI PC branding efforts aren't pure marketing BS.)
The issue is that the consumer strongly associates "AI" with LLMs specifically. The fact that machine learning is used to blur your background in a video call, for example, is irrelevant to the consumer and isn't thought of as AI.
Never wanted to do high quality voice recognition? No need for face/object detection in near instant speed for your photos, embedding based indexing and RAG for your local documents with free text search where synonyms also work? All locally, real-time, with minimal energy use.
That is fine. Most ordinary users can benefit from these very basic use cases which can be accelerated.
Guess people also said this for video encoding acceleration, and now they use it on a daily basis for video conferencing, for example.
Those usecases are at least 5 years if not 10 years out. They require software support which won't come until a significant part of the pc market has the necessary hardware for it. Until then, paying extra for the hardware is foolish.
This will only come if Windows 12 requires a TPU and most of the old hardware is decommissioned.
Those use cases are already actively implemented, or being implemented in current software. The 5-10 year estimate is wildly wrong.
Also similar to GPU + CPU on the same die, yet here we are. In a sense, AI is already in every x86 CPU for many years, and you already benefit from using it locally (branch prediction in modern processors is ML-based).
> Also similar to GPU + CPU on the same die, yet here we are.
I think the overall trend is now moving somewhat away from having the CPU and GPU on one die. Intel's been splitting things up into several chiplets for most of their recent generations of processors, AMD's desktop processors have been putting the iGPU on a different die than the CPU cores for both of the generations that have an iGPU, their high-end mobile part does the same, even NVIDIA has done it that way.
Where we still see monolithic SoCs as a single die is mostly smaller, low-power parts used in devices that wouldn't have the power budget for a discrete GPU. But as this article shows, sometimes those mobile parts get packaged for a desktop socket to fill a hole in the product line without designing an entirely new piece of silicon.
Your comment is almost completely irrelevant to what the parent is saying. "AI would change everything we do" has nothing to do with "This new chip along with bloat from Windows enables new workflows for you". If you have been paying attention, you'd know that NPUs from these new CPUs barely made any difference from a consumer's perspective.
This is irrelevant when the question is whether marketing your CPU with "AI" will help sales.
Toilets also changed everything we do and are helpful in unobtrusive ways, but that won't make the "Ryzen Crapper" a customer favorite.
So I’ve got a lot warmer to believing that AI can be a better programmer than most programmers these days. That is a low bar :). The current approach to AI can definitely change how effective a programmer is: but then it is up to the market to decide if we need so many programmers. The talk about how each company is going to keep all the existing programmers and just expect productivity multipliers is just what execs are currently telling programmers; that might change when the same is execs are talking to shareholders etc.
But does this extrapolate to the current way of doing AI being in normal life in a good way that ends up being popular? The way Microsoft etc is trying to put AI in everything is kinda saying no it isn’t actually what users want.
I’d like voice control in my PC or phone. That’s a use for these NPUs. But I imagine it is like AR- what we all want until it arrives and it’s meh.
When I interview people for a job I'm not looking to hire an average programmer, though.
You want to hire an above-average programmer, I assume. Do you also pay above-average salaries?
Doubt it, and competition does the same so not like they are going intel coz of that
I'd actually love to have an NPU that isn't useless on my 285k.
Or hoping to get ahead of the Windows 12 requirements.
Indeed, I was buying a laptop for my wife, and she was viscerally against "Ryzen AI": I don't want a CPU with builtin AI to spy on my screen all the time!
They can just buy a regular Ryzen 9000 series CPU, then. Maybe add a real graphics card if they're into gaming.
Some of these supporting ECC is of some interest. Though fast udimm ecc ram is going to be extremely expensive
Do we expect special AI processors to diverge from GPUs? Like, processors that can do parallel neural network computations but cannot draw graphics?
That is already the case with datacenter "GPUs". A A100, MI300 or Intel PVC/Gaudi does not have useful graphics performance nor capabilities. Coprocessors ala NPU/VPU are also on the rise again for CPUs.
Great now I’m envisioning a rich guy using an A100 as his desktop GPU just to show off. Which begs the question if that’s even possible.
It has no video output.
I believe some cards at least you can make the motherboard display ports the output
This is kind of true back in the day though. Uninformed people would buy Quadro cards because they were the most expensive GPU on Newegg only to realize this thing sucks for gaming.
I'm expecting in the not so distant future we'll even have LLMs baked into an ASIC, in our PCs.
That’s already the norm no?
Pretty much every hardware vendor has an NPU
Yes, this has already been the case for years on mobile devices, CoPilot+ PC design requires this approach as well.
Additionally, GPUs are going back to the early days, by becoming general purpose parallel compute devices, where you can use the old software rendering techniques, now hardware accelerated.
We kinda already have it with NPU/TPUs, tho they are usually attached to CPUs (and for some reason come with near zero proper documentation).
I can see separate cards for datacenter use but for consumers they will probably come on same SOC as CPU
Yes.
Even the latest NVIDIA Blackwell GPUs are general purpose, albeit with negligible "graphics" capabilites. They can run fairly arbitrary C/C++ code with only some limitations, and the area of the chip dedicated to matrix products (the "tensor units") is relatively small: less than 20% of the area!
Conversely, the Google TPUs dedicate a large area of each chip to pure tensor ops, hence the name.
This is partly why Google's Gemini is 4x cheaper than OpenAI's GPT5 models to serve.
Jensen Huang has said in recent interviews that he stands by the decision to keep the NVIDIA GPUs more general purpose, because this makes them flexible and able to be adapted to future AI designs, not just the current architectures.
That may or may not pan out.
I strongly suspect that the winning chip architecture will have about 80% of its area dedicated to tensor units, very little onboard cache, and model weights streamed in from High Bandwidth Flash (HBF). This would be dramatically lower power and cost compared to the current hardware that's typically used.
Something to consider is that as the size of matrices scales up in a model, the compute needed to perform matrix multiplications goes up as the cube of their size, but the other miscellaneous operations such as softmax, relu, etc.. scale up linearly with the size of the vectors being multiplied.
Hence, as models scale into the trillions of parameters, the matrix multiplications ("tensor" ops) dominate everything else.
The 100 class Nvidia chips are targeted at training. With Nvidia buying Groq it will further move in that direction.
I'm not following the whole LLM space, but
> the compute needed to perform matrix multiplications goes up as the cube of their size,
are they really not using even Strassen multiplication?
I'm not aware of any major BLAS library that uses Strassen's algorithm. There's a few reasons for this; one of the big ones is Strassen is much worse numerical performance than traditional matrix multiplication. Another big one is that at very large dense matrices--which are using various flavors of parallel algorithms--Strassen vastly increases the communication overhead. Not to mention that the largest matrices are probably using sparse matrix arithmetic anyways, which is a whole different set of algorithms.
AFAIK the best practical matrix multiplication algorithms scale as roughly N^2.7 which is close enough to N^3 to not matter for the point that I'm trying to make.
Interestingly, this tops off at 8 cores. Where are the 16 core versions that already exist on laptops?
These are mobile chips shoehorned into AM5. They aren't very good e.g. for gaming purposes. https://videocardz.com/newz/amd-ryzen-ai-400-does-not-suppor...
I guess it depends on what games you play. I have an AI Max 395 (Framework Desktop) and it runs every game in my library flawlessly. I'm sure if I played this year's most resource-intensive games it might stutter, but I don't. For me, it's an amazing low power minpc doing triple duty as a gaming PC, development box, and running my self-hosted services for the rest of the house.
Yeah. I have one. People buying the Framework Desktop mainboard aren't buying it just for gaming. There are better and far cheaper options for gaming. What this does is everything, though. Good enough for 1440p gaming. 16c/32t powerful CPU, it can run LLM's. SFF main pc that can do everything in a tiny space is a win.
Presumably that's why the subheading is:
>First wave of Ryzen AI desktop CPUs targets business PCs rather than DIYers.
8 Core/16 Thread, boosting up to 5.1GHz with iGPU would be pretty neat for a Plex Server or Proxmox Server with a few VMs.
As far as I can find, Plex does not support AMD iGPU for transcoding. Jellyfin will work, but support seems rather spotty. For other AI/ML work, it seems like ROCm is up and coming, but support - e.g. for Frigate object detection - is still a work in progress, especially for newer chips.
Jellyfin supports it, but the resulting quality is noticeably poor compared to Intel QuickSync or software transcoding. Perhaps the newer chips are better, but if you're building a media server from scratch you'd probably build around an Intel CPU or ARC GPU anyway.
I have an AM4 AMD iGPU I use with Jellyfin; it works fine.
Is it actually using the iGPU, or just "brute forcing" it?
I've put it in quotes as the effort required from these chips for streaming transcoding is so low these days that brute force makes it sound like more effort than it really is.
>I've put it in quotes as the effort required from these chips for streaming transcoding is so low these days
What's your source for this? Transcoding without acceleration is incredibly expensive, especially for 4K content, and especially for 4K HDR content.
Even a single 4K HDR -> 1080p transcode takes a huge amount of resources.
The Asustor Lockerstor4 Gen3 has a Quad-Core Ryzen Embedded V3C14 and cannot transcode 4K content.
Meanwhile, an old Kaby Lake Intel chip does so just fine but only because its QSV can handle h265.
Thats interesting. My 5 year old Ryzen laptop can transcode 4k faster than realtime, which is what I mean mean by "these chips". Modern Ryzen, which is what the subject is about.
Quick Sync is invaluable for low powered processors, my old Intel embedded Wyze can do several streams.
"faster than realtime" doesn't mean much if it's in a device that's supposed to do more than just transcoding (such as serving a web app) or if you need multiple transcodes, etc.
Even on modern chips, transcoding is quite expensive.
People who are running Plex generally are running on servers also serving files, web apps, and who knows what else. These devices are often running 24/7, so both overall cost and power efficiency are big concerns. I wouldn't want to rely on my server being at high CPU usage most of the time - for power, heat, and overall reliability concerns.
My proxmox server with a few VMs works perfectly fine with much less compute.
My homelab setup runs out of memory much faster than it does CPU cores.
Depends what you’re doing on the VMs, I run one as a desktop PC so have 4/6 cores and all the GPU access is important.
Maybe also Immich, for face and object recognition.
PC desktop chips have 128 bit busses to right ? From what I know theoretical maximum memory bandwidth of chips is less than 100gbps - which is less than like base M5/M4 chips.
So no matter how much compute you stuff in there it's going to be shit for AI ?
PC architecture is not adapting to AI workloads at all, and no signs of that changing in years to come. I would not be surprised if your phone was more capable of running AI models than an average desktop - especially given gpu pricing.
Apple has their high-bandwidth chips, the rest of the commercial desktop market is effectively running Windows, and Microsoft has no incentive to move towards local AI, their ideal case is that you use their cloud-based services and pay for them forever (you being enterprise clients with thousands of PCs).
It doesn't sound as impressive as I wanted :-(
I wanted a better strix halo (which has 128GB unified RAM and 40cu on the 8080s (or something) iGPU).
This looks like normal Ryzen mobile chips + but with fewer cus.
Putting Strix Halo into the AM5 socket would make no sense. Half the memory controllers would be orphaned and the GPU would be severely bandwidth-starved (assuming that the memory controller on Strix Halo actually supports DDR5 and not just LPDDR5).
They could put it on threadripper socket, very similar memory bandwidth
And waste all those PCI Express lanes?
Yeah the next generation of Strix Halo is what would get me excited. I think right now TSMC has no capacity, so maybe we have to wait another year. Kinda ironic that all CPU/RAM capacity is being sold to LLM companies, and as a result we can't get the hardware needed for good local LLMs.
> all CPU/RAM capacity is being sold to LLM companies, and as a result we can't get the hardware needed for good local LLMs.
yeah... Ironic I guess. It's as if they've realised that it's only a matter of time until we get a "good enough" FOSS model that runs on consumer hardware. The fact that such a thing would demolish their entire business of getting VC hyped while giving out their service for a loss surely got lost to them. Surely they and Nvidia have not realised that the only thing that could stop this is to make good hardware unreachable for anything smaller than a massive corp
Mark my words: in less than one year, we'll probably get something akin to Opus 4.6 FOSS. China is putting as much money into that as they can because they know this would crash the US economy, which is in the green only thanks to big tech pumping up AI. China wants Trump either gone or neutered as soon as possible, which they know they can do by making Republicans as unelectable as possible - something that will probably do if the economy crashes and a recession happens
I can use this in OpenWeb on Unraid? Save me buying a pascal card?
50 tops ain’t all that much.
Since this is for desktop, the NPU is irrelevant. Consumer-grade NPUs are not made for high performance. They are optimized for low power consumption, which makes sense when you are trying to run basic AI tasks on a laptop without turning it into a frying pan. On consumer-grade gear, the GPU will outperform the NPU, and since a desktop is not constrained by battery and can dissipate far more heat, the NPU is mostly irrelevant (aside from using it to develop software for laptop NPUs).
"AI" branding applied to subpar products hoping to boost sales.
The Ryzen AI line is actually great if deployed to an entire org as the bottom tier, as it garuantees every device has a 50 TOPs NPU. We deploy local software at $STARTUP and this makes deployment to a Windows corp more predictable.
So happy I got my 9800X3D + 64GB RAM this fall instead of now lmaooo
Well, for me personally it is a meh until RAM prices go down. Suddenly, decent PC has turned from a tool accessible to average Joe to a luxury item
Narrator: The RAM prices did not, in fact, go down.
[dead]
Looks like someone has run out of money.
> This makes them AMD’s first desktop chips to qualify for Microsoft’s Copilot+ PC label, which enables a handful of unique Windows 11 features like Recall and Click to Do.
Microsoft: "Friendship ended with Intel, now AMD is my best friend"
Actually it is Qualcom, as they keep trying to push for ARM, but due to the way PC ecosystem has been going since the IBM PC clones started, no one is rushing out to adopt ARM.