I was wondering how much financial value they put in one truck. Presumably the value density of this stuff is incredibly high (although not really to anyone else, so security probably isn't a huge issue). At what point do you split the shipment between multiple trucks to avoid huge losses if there's a crash?
When I worked on satellites we had some electronics which used a radiation hardened FPGA which cost something like $500k per chip. Carrying that around was nerve wracking.
I had a cousin that worked private security, and for a few years (I think it was early 2000's) his main gig was to work as a team of armed security, and follow trucks from the two Intel campuses in Oregon to the airport in unmarked vehicles, get on the cargo plane with the cargo, and fly to the location where they did the packaging of the wafers (I want to say it was the Philippine Islands) and then fly back. Lots of overtime.
I once had to keep a $20M (technically; a million download codes for the same $20 movie for a promotion) thumbdrive in my home safe because it arrived too late to put in the safety deposit box like it was supposed to be.
I had to remind myself a few times the practical value to anyone was about $22.
I worked at a [Not to be named Country]'s monetary authority. Part of everyone's duty was to receive shipments of newly printed cash from De La Rue and get it into the vault before it was distributed to banks. Seeing millions of dollars packaged in cardboard boxes being moved from the street to the vault with men posted up in the bushes with machine guns was something I will never forget.
It's funny how rote and regularized "playing" with money gets after a while though.
I used to work for a financial institution small enough to sometimes need someone from the back office to help fill in as a warm body for dual-control when we received cash shipments.
It only took a couple sessions in a vault to get entirely bored with having a half million dollars in front of you.
My first serious software was for a sizable goldsmith factory. They keep an incredibly low profile visible from outside, and they get really good at it. Once inside, the amount of gold that's moving around is jaw dropping. After a while, most of the time it feels like aluminum, or any other material. All you see is cuts to be made, welding to be done, carvings, sandblast, etc... It's work and responsibility and everyone is just trying to get it done and make it the problem of the next person.
Enterprisey client chain-of-custody requirements. It was couriered to us, and the courier was late. Cue panicked phone calls to get an exemption to the contractual requirement for overnight storage at the bank.
My buddies company shuffles chips around the west coast, some deliveries are a few million dollars. They shove the chips into the truck, then shove a giant movable wall, leaving about an inch or two gap around the perimeter.
The truck has multiple drivers and private security.
It's treated the same way as any other generic shipment of cargo, really. It's fine throwing $20 million in one truck.
Security is an issue, though, these are treated as a high value load and typically would be followed by unmarked, plain clothes armed security guards tailing around the truck. (That's how Apple often does warehouse deliveries of trucks full of iPhones. It wouldn't surprise me if there was going to be some rolling lead soon following some trucks full of Duos...)
They probably have insurance which means it doesn’t matter if it is all in one car or not. And even if they didn’t Samsung is big enough where a loss of a couple million dollars of components isn’t gonna bankrupt them.
Ha - I still remember when I worked on the IBEX-Lo Sensor, we had to hand carry it to Switzerland. One lucky member of our team got to fly first class with it because it wouldn't fit in a coach seat!
Almost 10 years ago at my university in Chile. My professor of Computer Networks invited a PhD Student that was working on the ALMA Observatory [0] doing some research on signal processing.
What stayed with me from that talk was that all the data that was captured from those massive antennas was packaged in boxes, loaded in a cargo van and travel 1.600 kilometers to Santiago to be uploaded into the systems.
I wanted to mention AWS Snowmobile [0], but after looking it up I found that apparently it was discontinued [1]. It seems that nowadays there are more economical/efficient alternatives (I guess theoretically you could still beat these alternatives in certain metrics, e.g. by using more trucks at once to get more bandwidth).
I’ve often wondered about die thinning. I mean just looking at sd cards and micro sd etc, it was clear that there had to be a thinning step (or many steps) and it seemed amazing to me that such a complex step could be economically viable. And yet it clearly is. All a matter of being able to operate at scale I guess.
It’s not a step I’ve seen discussed very much, so it was kind of neat to see it discussed front and center in a lay article like this.
Not dissimilar to lens or mirror making, the process of grinding to a precise flatness. The key step here is bonding it to a glass carrier, presumably because otherwise it gets too thin to support itself.
See also the "reduced kerf diamond wire saw": technology critical to cheap solar panels is the ability to saw the boule into thinner and thinner pieces with less waste at each cut, like a big ham slicer.
Yeah, the scale is kind of mind-bending. You're taking something that's already extremely fragile, grinding most of its thickness away, handling it through several more process steps, and somehow doing that cheaply enough for mass-market products.
The critical back end step for HBM is thermal compression bonding - where they actually compress 13 very thin die separated by solder balls. Cracking is a major issue.
We are getting so many technologies like this that are so insanely specialized and require such deep knowledge of physical processes that some event like a 1918 flu that kills 10% of the population could take out enough people that we'd be unable to make things like this for years as we fill in the missing knowledge workers.
It's just crazy how far down the scale we're tickling atoms these days.
Not really at all. Between those dates and now the world has become fully global and concentration of knowledge has massively occurred. The past world was much more highly distributed, far less technical, and far more redundant.
On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.
The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.
The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
The only thing the West is achieving here is that sooner or later China will be able to compete on their own terms rather than ours. It may buy some time but the end result is very predictable.
As other comments pointed out, that was always China's plan, the value here is really that when the Chinese overlord annillate the world's industrial competition, they'll have the moral high ground of being out of self-reliance and hostility by western monopolists.
Edit:
And, that, my fellow keyboard high-minded thinker but powerless corp peons, was also part of the China's plan all along.
China's plan to be self-sufficient in semiconductors was a failure before western sanctions, because all the Chinese chipmakers found it too easy to buy western chipmaking equipment or to fab through TSCM. No Chinese company wanted to buy from other Chinese companies. Nobody was incentivized to develop self-sufficiency. "Because the government wants it" was not enough incentive.
Western sanctions have succeeded in achieving China's self-sufficiency goals where their own government failed.
Eh. Industrial policy in a big country is always the output of politics between stakeholders, anthropomorphizing it to a "guy with a plan", especially an "overlord" looking to "annihilate" is probably getting it wrong.
A friend of a friend is married to a member of the national academy of sciences in China. I heard at dinner a few years ago that he had been lobbying for more onshoring of chip production for years before the Nvidia restrictions, and always heard "you're right, maybe next year we'll be able to fund that". Then the restrictions came in and everything changed.
And they are more about sounding good than about reality. They also affect a much smaller share of the economy than they used to -- which is very good! The original five year plans caused nothing but misery and starvation.
I’m pointing out it’s disenguous to say ‘it’s not like there was a plan’, when there was explicitly a plan with that title on it. Especially if the plan didn’t work out well.
It's disingenuous to characterize "don't anthropomorphize the political process" as saying no planning documents exist as part of that process.
And, to the point I was responding to originally, we can check those documents! They don't speak of "annihilating" other countries' industrial capacities.
>They don't speak of "annihilating" other countries' industrial capacities.
Why would they have to?
When China talked about their 5 and 10 year plans on solar panels there was no need to talk about putting the rest of the competitors out of business. That was a highly probable end goal of what they were doing, no need to state it.
It would be like Walmart stating "We're going to put out massive loss leaders in sporting goods to put X out of business". Instead they'll talk about growing market share and long term customer loyalty.
More of, China is smart enough to use the benefits of both capitalism and authoritarianism to grow their country while western governments are busy stealing the furniture and fixtures off the walls.
I agree with 100% of that comment but I'm struggling to understand the beef?
Of course they're trying to take care of themselves while the US implodes, how can you blame them for that? The whole world has a serious oil problem right now, because of US actions, and of course each country is dealing with it in their own way. Why is that a problem?
You do realize that a lot of people have had problems with America after WWII and using their economic and military power to dictate what the rest of the world does, right?
"Problem" is a relative term for those dealing with the consequences of it. It is one if you are America means a whole lot less economic freedom than you had in the past. If this is a good thing or not depends on your position.
If your beef is that they are succeeding or at least failing less while America steps on it's dick repeatedly, maybe this isn't their problem?
If you're fired up by nationalism but your nation is sucking, getting mad at the Chinese isn't the answer. And that's before we even get into relative levels of activism, Chinese political ideology is much more biased towards "neutral, just trade" than American politics.
Oh boy, how to get a bunch of seething haters more seething.
Near as I can tell, they fundamentally believe that the US actually can’t do better (despite protestations to the contrary), at least not with them there doing there thing, which is why they get so manipulative and shitty. They are unwilling to make other decisions than the ones leading to these outcomes.
Yes people, it is actually possible to do better. It requires making better choices, however.
False dichotomy. the problem is far deeper than that, and wanting to simplify the problem to those two extremes belies a belief that the world could and should be able to be simplified into two power dynamic. It is not. The world, and the powers that fight within in, are far more dynamic than a simple communist or capitalist spectrum, and attempts to simplify the world along those lines is only going to hurt you.
That just means your stakeholders argue twice: once when drafting the five year plan and then again, later, saying their policy position is more aligned with the spirit of the plan.
> [...] that was always China's plan, [...] they'll have the moral high ground
It's a possibility. It is also possible that they will steal yesterday's technology and still pretend to have moral high ground.
I do not wish China and Chinees wrong but most probably communism will work as it always do - by faking thing and bringing self-harm to their own citizens.
In free markets, everyone steals everything they can, and stops everyone else stealing from them as much as they can. This is normal and expected. The ideology says that it will even out and converge to a fair situation.
If you're paying money for something you could get for free, you're a sucker. Ordinarily the person who is selling it makes sure you can't get it for free.
Our LLMs were trained on the widespread theft of copyright content, with clearly defined and well tested law. But they made capitalists too much money so it became defacto legal.
Next you'll be telling me fairy tales like "laws are real".
> In free markets, everyone steals everything they can, and stops everyone else stealing from them as much as they can. This is normal and expected. The ideology says that it will even out and converge to a fair situation.
This is all nonsense. Capitalism isn't an ideology, it's just a slight formalisation of what naturally arises. People are free to choose; they can own things; they can make agreements; they must honour the agreements or pay a penalty. Capitalism in no way is about stealing.
It’s the same in communism/socialism except people are prevented from stopping everyone else from stealing their stuff, in the name of the ‘greater good’, and if they try to get more than everyone else get guilt tripped into giving it up under the ‘to everyone what they need, from everyone what they can’.
Which incentivizes pretending you need a ton, and can give nothing. You think being useful at work in Capitalism is bad? Wait until you are accidentally useful in a communist/socialist economy.
There is usually less pretending under capitalism, but people are people, so it all the same stories still play out.
People are shitty, if given the chance and influenced in specific ways. Capitalism or not. We’re dealing with the current bullshit in the west because somehow people bought the line that everyone is a good person - and we totally can’t actually see or do anything if we see someone doing something bad! - and we need to always follow ‘the rules’ even when someone bad clearly isn’t.
But none of them had a government-mandated one-child policy (maximum, not minimum) for over three decades. That's a pretty major difference between China and neighboring countries.
And yet all of them have an even lower fertility rate than China.
On the one hand, you're proving my point. On the other hand, if you think that China's headed for a demographic disaster, so are all of those countries, and your thesis of blaming the CCP for everything... Falls a bit flat.
China has extremely low fertitily rate given that they haven't yet reached the status of a developed country (which Taiwan, SK and Singapore have reached).
Developed countries tend to have lower fertility rates (i.e. less kids per family) than undeveloped ones. Comfortably off, well educated people tend to put thought into whether to have kids or not. In poor countries people tend to have large families so they can send the kids out to work.
Having a high fertility rate isn't "high status" and a sign of a country's progress, but rather the exact opposite.
Of course this is separate from this issue of having a shrinking population, which is a sign that your citizens are not optimistic about the future, not a positive thing.
Still, a stupid governmental policy cost them at least 100 million people compared to the expected spontaneous development. Probably more. That is the sort of punishment that omnipotent governments sometimes visit upon their folk.
I don't believe that even the CCP itself, in hindsight, considers the one-child policy to have been a good decision. At the very least it should have been ended some 15 years earlier.
Well, I've been reading on The Economist how China is seriously struggling with youth unemployment right now. Lots of fresh university graduates are sitting unemployed, or doing manual labor. In my home country (Finland) we have had way below replacement birth rates for a long while now, yet youth unemployment is at all time high.
The problem with low birth rates is assumed to be that there are less workers to take care of the pensioners. But right now it seems like even the few young workers available often aren't needed anymore, thanks to increasing automation of industries. I predict youth unemployment will be made even worse by AI soon enough. So, why do we need children, if capitalist economies don't need workers? Unemployed people do not pay much taxes to sustain the non-working age population.
I believe low birth rates might become a blessing soon enough. It just means less people for governments to take care of when robots and AI take away most jobs.
No, a remarkably farsighted policy that prevented PRC from having to deal with additional 200m-400m subsistent farmers. If anything it wasn't done sooner/aggressive enough. Like you can literally wipe away 200-400m from PRC button quantile and PRC would be net better off due to increase per capita resources and wealth. This hindsight will be historically obvious in 20-30 years when you see the outcome of India with 1.6B people with 1.2B poor, vs PRC with 1.4B people with 400m poor. The drag of having disproportionate, bluntly, worthless bodies is not linear for governance, and it's not because India hasn't tried aggressive family planning of their own, they simply failed unlike PRC. Who very much regards family planning as largely successful demographic management system that enabled proper allocation of resources away from generating peasants to building industry and the largest pool of high end human talent in recorded history, where frankly any tertiary dividend is worth 50 farmers in terms of productivity.
China is headed for a demographic disaster, though. High schools are incredibly packed, middle schools, not so much and primary schools and kindergartens are kind of empty.
It's hard to say that it's stupidity (at least not country-specific stupidity) - it seems that globally, presumably due to the state of the world along many axis, many people are choosing to have fewer children, and it's happening on a large enough scale to have major consequences for things like the size of your tax base, trying to keep your economy growing, demographic concerns over an aging population with no-one to look after or pay for them etc, etc.
Different countries are handling this in different ways, with encouraging immigration being the major one. Japan is facing a major lack of workers, with many businesses having to shut down as a result, but have decided to tackle it in a different way relying on advanced robotics as an alternative to immigration.
There are very few developed countries NOT having this problem. Most of the population growth is in poorer parts of the world like Africa.
Well, if there are >0 developed countries that have solved this (Israel, the GCC, dare I say fairly well-off MENA countries like Jordan, Algeria, Morocco, etc., and to a minor degree Botswana) then this proves wrong the hypothesis that modern hyper-development must inherently mean population collapse.
Going purely by the evolution theory and natural selection, having so few children that there is a serious risk a population will disappear in a century or two, means losing. Needless to say, this is stupid. From the entire American continent, to Europe, India more recently, Thailand, China, the Koreas, Japan.
Africans, Arabs, and some non-Arab Muslim populations will carry the torch. Much respect for not falling into the trap.
Algeria and Morocco hardly seem "fairly well-off" countries, considering how many of their citizens prefer moving to the low-birth rate European countries you loathe. High birth rate of Israel is also unsustainable from economic perspective, as it relies mostly on orthodox Jews who on average are much poorer, less educated, and contribute less to the Israeli economy.
I'm not saying fast fall in birth rates isn't a problem, it obviously is. However, nearly all countries that don't have this problem offer much worse quality of life than those which do.
Young southern europeans are in turn migrating north. Are Spain, Italy, Portugal not well off?
There is a saying. More is better. Morocco has issues with tourists and diaspora pricing out locals from houses to food even, but Algeria is an honestly good country all around. With its issues of course. But there is more to life than earning $$$.
Westerners forget the tiny detail that these countries are extremely young unlike them, hence why it feels like there are so "many" people.
Edit: oops, I glossed over the part about Israel. Trust me, it's still a billion times better than nothing, which is the case for the entire West.
Edit 2: Oh my oh my, I missed this too. I "loathe" European countries? Do I? I mildly not respect European populations for wiping themselves out. I cannot loathe land.
> this proves wrong the hypothesis that modern hyper-development must inherently mean population collapse
I don't know the exact correlation but even in Morocco, total fertility rate is declining very rapidly, down from 6 children in 1980s to now around 1.7 in urban areas. UAE (Dubai) has similar TFR to Belgium or Portugal.
> then this proves wrong the hypothesis that modern hyper-development must inherently mean population collapse.
Honestly you're reading the data completely wrong. You cannot look at it at the country level. In Israel for example the populations that are producing the most population are highly religious and less educated than those that are producing less population.
It's more about memes. Religion is a meme driven breeding cult. This is why said religions tend to have very long term staying power. The particular problem here is these religions come at a cost of having to ignore a lot of reality to get to that point. Because they are faith based anything that presents a direct challenge to that faith must be setup as an enemy or it may risk the religion. Yes, religions can incorporate new information but it tends to be a slow process as it needs to involve slowly changing peoples mind to avoid instability.
What this looks like in the long term is large populations of highly religious that have a fundamental disconnect from the technology that's keeping them alive. A new Late Bronze Age collapse.
So? How does the population being religious disprove the fact that they are developed and having children? Not to mention that I have listed around ~10 countries after Israel.
Uh... Meme-based? This is such an unserious argument I don't even know what I can possibly say to that.
How are they disconnected with the technology keeping them alive? All people, in GCC countries especially, are working to make 1st world development happen.
Implying that religious Jews or Muslims are stuck in the Bronze Age is ignoring reality. Most European development itself came from men of science who were Christian, I thought this apparent conflict of ideas was something taught in schools.
First, you seems to think a meme is something unserious. Read Richard Dawkins The Selfish Gene so you know what meme actually stands for rather than the thing you see on Reddit.
GCC countries are resource economies mostly. They must weave a very narrow path to success to non-oil based means of income or the entire region collapses. I will say they do realize this is a problem and are using oil money to diversify, but resource curses are very hard to break and collapse can happen very quickly.
"My grandfather rode a camel, my father rode a camel, I drive a Mercedes, my son drives a Land Rover, his son will drive a Land Rover, but his son's son will ride a camel." --Sheikh Rashid bin Saeed Al Maktoum
>Most European development itself came from men of science who were Christian
This is just stating that that in 1700 you were a Christian or you got hung (hell, pre-independence America was founded by people that thought that too few people were being hung). As we see when you take out the hanging or social shame part of the religion people tend to flee them as they are oppressive structures. You could say them being men of science either had no bearing to their religion, or was in spite of their religion.
I am saying, for example, that the Abrahamic religions are bronze age (ok, sorry, Iron age) social structures. I don't think studied scholars are going to disagree on that point very much. People saw how people worked and could be sheppereded in societies without formal education and turned it into a series of stories that were effective in modifying peoples behaviors. Kings later came along and added their stories so they could further control the population. Over time a semi-coherent story about how people worked was formed below a set of rather incoherent stories regarding things that didn't happen that people are supposed to accept with faith.
Notably, it is hilarious how ‘modern western society’ seems to believe that people somehow work differently than they always have, while clearly mimicking the same rough structures they are deriding.
(Comic book superheros, cough cough. The references to Canon in what are essentially liturgical disagreements - I mean power scaling arguments - are particularly hilarious.)
Not me of course, I’m different and unique. Just like everyone else.
The official one child policy didn't actually have much of an impact. Birth rates had already fallen a lot, despite the official "lots-a-children" policy.
The one child policy is gone now but birth rates haven't really moved upwards.
It's debatable and impossible to disentangle from living standards going up, but a lot of the economy is built on assumptions of one child, from apartment sizes to personal transit (ebike can do 2 adults + 1 small child).
Of course it was devasting for some families. That is never the same thing as saying that it affected China's birth rate much (which it didn't).
There were forced abortions (sometimes very late), there was forced contraception, forced sterilization (I think), and lots of punishments for people who broke the policy.
There were also others who got around it fairly easily by paying a bribe and by not registering their "surplus" child/children (especially if they were girls). That of course caused some problems down the road.
That still doesn't mean that it affected China's birth rate much. It was already low (and had fallen steadily) when the policy was instituted. It also had plenty of exceptions for rural China and ethnic minorities.
So China has similar family sizes as the US that had no one child policy.
It's difficult to near impossible here to determine which counterfactuals would actually hold true. Based on other world wide trends it's very likely that Chinese family sizes now would only be slightly higher than if the policy didn't exist. In the modern world there are just a huge number of different factors working together to decrease family sizes.
What the West is doing is forcing China into a situation which is impossible to sustain when their economic dividend runs out. Especially considering these investments are fuelled by an extremely high level of debt.
And it’s not a given that they’ll catch up. I think it’s fair to say that their development of jet engines is on track to be obsolete before they’re competitive.
They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.
i dont think it will - they still have a chokehold on battery production (which implies dominance in the EV sector), and they also control the vast majority of all rare earths. The chinese gov't can subsidize chip research/development with these other sectors that are dominating.
The west is the one that need to watch out for their own economic dividend running out from the past century.
"Economic dividend" really means "Demographic dividend". It's when your fertility rate decreases but your total population isn't decreasing yet, so you get more people to work more as they spend less time as parents. This of course is a self perpetuating cycle and ends in demographic collapse. China's population is now decreasing. The West's population is still growing. China is in the most literal sense dying. It's what you get for having a one child policy for 50 years. But hey, for a beautiful moment in time they really created a lot of shareholder value.
What makes you think they don't have a domestic market?
The Chinese domestic EV market is the largest in the world.
Chinese factories are full of Chinese robotics.
Companies like Huawei are struggling to meet the domestic demand for AI accelerators, while growing volume very fast year over year.
The Chinese domestic AI market is massive, but we just don't hear much about it outside of the models that are also sold here. Can you even name the models that ByteDance (China's Meta) makes?
As far as exports go, China is a low cost producer for many things. If the US doesn't want to buy high quality EV's costing 1/2 the price of a Tesla, then there are plenty of other countries happy to do so. The US is only 5% of the global population - there are lots of other consumers out there.
> They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.
Is this really true? Can't they just keep exporting "forever"? What requires them to find a domestic audience? I can think of some other countries that have done quite well mostly without finding a domestic audience (eg, Switzerland comes to mind, and frankly Europe might largely be in the same bucket).
Europe (by which I assume you mean EU or possibly Euro area) has a fairly balanced balance of trade overall.
And a teeny-tiny country like Switzerland can't be usefully compared to something as big as China - the rest of the world can easily consume whatever the Swiss produce, but there are limits to how much Chinese can produce until consumption will have to shift to domestic consumption because the world doesn't have infinite ability to pay for its demand.
That said don't expect any big change anytime soon - there are very strong political incentives to keep the status quo with China focusing a bit too much on exports.
> It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
As someone who is myself 'old af' by tech standards (I'll be 53 very soon) the main problem is that the policymakers are all 70+ years old.
Their concept of what China even is is forever stuck in pre-2004 thinking from back when they still had significant neuroplasticity.
Vote these fucking fossils out, kids, before it is entirely too late.
(And, yeah, before someone inevitably brings it up, some exceptions-that-prove-the-rule older policymakers/politicians do avoid this trap of getting stuck in the past, but most don't)
Over the last 20 years the economy has become dysfunctional. It no longer really resembles a free market; monopolies have established barriers to entry everywhere. And the biggest investors are awash with helicopter money that's been doled out for favors by the political class.
So those investors have tons of cash to burn and surprisingly few opportunities. Even within Silicon Valley/VC there are surprisingly few who seem to really understand the fundamental economics of software. Or perhaps those economics just aren't that important when you have billions of dollars on hand and cash is obviously not going to get you a return. Any whisper of possible exponential growth is worth throwing money at. Crypto? Why not. AI? Why not. Datacenters? Why not. Tulips? Why not.
This is by all definitions an empire in decline. Everything is broken or fake. Everyone is afraid to do what needs to be done. Power forbids it. So we're all just waiting for the other shoe to drop. Our secret police aren't as bad as the late stage USSR's yet, but hold Uncle Sam's beer...
(That's not a recommendation to try and time the nadir, by the way, as it could easily be 50 years away.)
I feel like my point is being over extrapolated a bit. All I was saying is that people in 2020 believed AI companies had much more of a moat then they actually do - that other companies would struggle to obtain enough data (y’know, downloading the internet to train models wasn’t legal back then) or resources to compete
It started way before the last 20 years, the past 40 years have created the conditions for this to happen.
Overfinancialisation of the economy while removing safeguards to keep markets functional, like anti-trust enforcement, are the main forces behind this erosion. Through finance the focus of companies is completely shifted away from producing good products and services, and into how to extract more paper wealth from existing structures to the detriment of products.
Not enforcing anti-trust and letting behemoths to form which cannot be competed against since with their amount of capital they can either buy their competition outright or just price dump for long enough to make competition non-viable.
It's the failure of neoliberalism, and that agenda has been pushed into Western countries since the 1980s-1990s, it made enormous wealth for the few at the top while eroding whole societies, economically and socially, it's all dysfunctional.
I feel like my point is being over extrapolated a bit. All I was saying is that people in 2020 believed AI companies had much more of a moat then they actually do - that other companies would struggle to obtain enough data (y’know, downloading the internet to train models wasn’t legal back then) or resources to compete
I replied to the extrapolation, not to your comment directly, I think your point might be more fitting under that comment than mine, I just drilled down from the points above mine.
The initial growth of the internet was a boom bust cycle, and as you see the internet is still here. This is more about the economic paradigm that is being used to grow a technology versus the economic paradigm the technologies rate of growth can support over time.
Just to give some rough numbers in the past 5 years the amount of GPU compute installed in FLOPS is somewhere over 5 times all the CPU flops that have ever existed.
This has nothing to do with AI being good or bad or being able to produce things and economic value. It is by far the largest and fastest growth of any technology ever and we have zero clue about the economic stability of this grand experiment we're performing.
The internet is a much more appropriate comparison than crypto or tulips, which bring no or negligible value.
Some differences to the market at that time seems to be that during the dotcom bubble, many of the companies had little to no revenue.
Leading AI labs already generate enormous revenue. The investments into capacity are needed to address the current demand.
The situation seems somewhat less speculative.
That said, I cannot predict how AI capabilities will develop and how demand will respond.
Should capabilities plateau hard and soon, maybe the demand will not be there for the compute investments.
If it does not, and instead AI applications in robotics, science, and self driving expand, chances seem reasonably good that the demand will be there, no?
And as for the economic stability, much of the investment comes from existing giants like Microsoft, Alphabet, Amazon, and Meta, who have the necessary cash flow.
These companies are less likely to collapse than some of the ones during the dotcom bubble.
Eh, I'd say it's closer to something like internet + tulips.
It's the total amount of money in the economy that's been invested toward a potential outcome. AI represents the largest amount of money, and largest fractional part of the economy invested ever.
Because of this AI could be the biggest economic boon ever, yet still not recover the full amount invested. This will have deep economic impacts that affect everyone and everything. At this point AI must achieve all its stated economic impacts or there will still be a huge economic crash that kills off any company that is over invested and cannot make a profit.
Worse, the many of the perceived economic impacts of AI are not for humans like you or I, but the huge companies you listed. Even if they economically win, everyone else made out of meat could still lose.
They say history doesn't repeat, but it does rhyme. This, at least to me, sounds like a mixtape of "internet" + "tulips" + "1920s financial world leading to global political instability".
Every potential outcome I see occurring pushes us closer to further instability, even if the economics on it work out on paper.
Out of this number how much of this goes to payroll that goes to humans versus how much of this goes to non-human infrastructure costs. The numbers suggest this is anywhere from 60 to 90%, with 70% being a reasonable average figure. Just under a trillion dollars paid directly to humans would disappear if AI somehow captured all of this market.
Moreso the AI industry needs to capture these gains very fast or they will be crushed by interest payments on the massive debts they've accrued.
>ven though Amodei hopes to cure most major disease in the next 5-10 years
The average development time for medicine is 10 to 15 years before a single dollar is earned from said medicine. AI will shave very little off that as human testing and our stupidly complex bodies introduce all kinds of problems. Furthermore running head long into blindly using AI medicine is how you produce X risks from super intelligence AI.
The speed at which AI has to develop in which to get profitability is the biggest risk, a potentially catastrophic risk at that.
You can theorize all you like, and many people like yourself are expecting to see GDP growth pick up as a result of AI, maybe if only because of the boom in datacenter construction etc (regardless of whether AI ends up actually boosting the economy), BUT ...
The reality is that so far there has been no sign of GDP growth picking up. It is basically flat at 2.5% +/- over the last few years.
In a similar vein one might have expected that the internet (think of all the e-commerce and efficiencies!) might have shifted GDP growth into a higher gear, but it did not, although in that case there was at least a significant boost in the 1996-2000 "dot com" era when the build out was happening (then to be followed by the crash and all the unused dark fiber etc).
So, maybe the hoped-for AI boom will be just as much of a dud (as it appears to be so far) as the internet boom. New day, different tools, same growth.
It's perhaps odd that we're not even seeing datacenter/etc build out register on GDP, but perhaps the scale of it is not as large as the internet build out?
>but perhaps the scale of it is not as large as the internet build out?
In the past 5 years 5 times as much GPU compute capacity has been installed as total CPU capacity that has ever existed. When it comes to capital intensity the growth of AI has been one of, if not the largest capital expenditure for a class of project in this short of time frame. Things like the Manhattan project which was absolutely monstrous expense at the time is dwarfed by this.
Of course measuring GDP growth is difficult when you have an idiot in chief trying to destroy the global economy with the dumbest set of actions contrived by a human ever. Disentangling this factors will take a lot of work on someones part.
No longer strangling their own economy would be a good start. Take a look at the magnificance that was the ESRS. A 300 A4 pages long list of yearly reporting duties. And the best thing is: Just because they're no longer nicely summarized in one place, doesn't mean the underlying duties went away.
Well Soviet Union tried to be self-sufficient, but the truth is such strategy loses to global free trade. Whether we want it or not, global trade is what keeps the world from WW3 for now, not the nuclear MAD.
Yep, exactly. US is in the active process of shooting itself in the foot. My point is that China is very dependent on resources imports, so there's no way for them (and any other country) to be self-sufficient or they will stagnate.
And China is very dependent on imports of almost everything. Maybe some rare elements, but e.g. oil is a big part of it (they agressively diversify off it, but it's still a large piece). Global interdependency is a good thing.
Sanctions can absolutely create the incentive to build a domestic supply chain, but incentive doesn't automatically translate into catching up on the same timeline
people have been claiming Nvidia will get competition too for years but you guys seem to be forgetting that the current leaders are not asleep at the wheel in the meantime.
When China eventually catches up on conventional lithography, the goalposts will shift to silicon photonics and optical logic, a transition DARPA laid the publicly visible groundwork for decades ago through programs like POEM and PIPES while keeping the mature implementations strictly classified until needed.
Not quite, it's got quite a bit more complicated with agentic use cases.
Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result.
Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound.
Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
This is why I was careful to specify tokens per second not time to first token which is the prefill step you’re talking about. Clearly to run these models well you need both but as I said adding more chips or compute units can give you more latency where as overall memory bandwidth (throughput) is limited by access to fast memory.
I can easily use close to 100 million input tokens a day. A few million output tokens but at the end of the day maybe a thousand or so lines of code get written.
According to SemiAnalysis, both inference and post-training (RLVR) is mostly memory bandwidth bound. Only pre-training is compute bound, but it now only takes a small share of overall data center capacity.
The "abysmal" yield using DUV is an overblown statement by western commentators who don't look closer at the development. Certainly yield is worse than with EUV, but after multiple iterations of development, yield has become pretty good, within economically acceptable bounds, still making scaling possible. Volume is still ramping up. A lot of capacity will come online in 2027. The removal of western middlemen such as Mediatek actually improved the economics. Further yield improvements are still coming. Volume is already so high that domestically made phones and NPUs are making a real impact, yet are also selling like hot cakes.
It depends on what you are trying to make, but there are workarounds for most things, and it becomes more of a cost and volume issue than a showstopper.
If your yield is low and you still want the volume, then run more wafers, but then you need access to more DUV machines and more wafers. If your defect density is too high, then design smaller chips with a higher chance of avoiding defects, which is what Hauwei are doing - split processor into multiple chiplets (NVidia do this too, to raise yields and reduce cost).
HBM3E usually has 5-6000 vias, but doing this with multi-pattern DUV without defects is tough, so CXMT currently drop that to 3000, at a cost of some loss of thermal and voltage stability.
HBM:processor production ratios aren't what Huawei would like, so they mitigate this at system level by putting an optical memory bus on the GPU chip and sharing memory across the system.
Not everything is a huge LLM - smaller models like recommendation systems don't have so many parameters and need so much memory, so why waste HBM on them? ByteDance use their SeedChip acelerator for this, currently made by TSMC (using an older sanctions-approved 28mm process), which instead etches dense RRAM on die beside the processor.
There is also a time dynamic to this, with China still using pre-sanctions equipment and chips as their domestic alternatives ramp up to replace them. One interesting part of this is HBM ... HBM is very demanding to make, and everyone had problems with it, with initially only SK Hynix being successful. Samsung and Micron took a year or so to catch up, and during this time Samsung had made a ton of HBM that didn't meet NVidia's specifications, so ended up, pre-sanctions, selling it to China, where it has acted as a stockpile to carry them over as CXMT's domestic capacity ramps up.
China seems to be doing fine. Of course they would like sanctions lifted, moreso for HBM than anything else, but all that sanctions have really achieved is accelerating their semiconductor independence. They are still building 1T+ SOTA models, standing up 100K clusters of domestic AI accelerators, etc, etc.
The two challenges there are firstly - that analog design has been a separate electrical engineering school for most of a century, so there are few who could design it - and secondly - that every single chip will have subtle variations in its computations, necessitating some sort of model finetuning per chip. Possibly the chip could be characterised at the factory, and ship with the characterisation data burned into a controller rom or something, but if that doesn’t pan out the whole thing is likely a non-starter.
If it could be made to work, you could run a fable-grade model in tens of watts.
This made me smile. A true test of human-like intelligence. Maybe they could take some entry level jobs (“I can make a nice Wordpress site”) and build a CV?
China has had a very strong incentive to recreate lithography and fab equipment for years, but a lot of the specialized HBM packaging chain was probably much lower priority until relatively recently
This competition will be great for us consumers for a while. Until China undercuts everybody else for long enough to essentially monopolize the market. This pattern has repeated itself across many markets already.
It's not that there's a blocker. It's that it takes roughly 3x the manufacturing capacity to produce an HBM package at the same storage capacity as DRAM. We are sacrificing total bytes for bandwidth.
I don't fully understand the source of the "total bytes" constraint, but a major factor may be because HBM4 / HBM4E can only make use of the footprint directly above the processor/logic die (or in direct vicinity of its interconnect), while traditional DRAM can be placed further away where there's lots of real estate on the motherboard.
I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?
These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per square millimeter of motherboard, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
Thanks for that, just went down an interesting rabbit hole. Many of us were hoping this re-tooling would eventually trickle some fast RAM down to DRAM-exhausted PCs, but given it would require a rearchitecture of the motherboard it's unlikely.
HBM4 has over 2048 signals to the processor’s PHY with tight signal integrity requirements that require the HBM stack to be < 0.5 mm from the processor die. That’s why HBM integration is done with interposers (soldered on the package). So, it’d be the CPU package that integrates it. Motherboard is too far away.
Kind of seems like we should be making chips with both. Big HBM stack on top as a sort of huge L5 cache like thing. And then a bunch of DRAM type sockets (like LPCAMM) around the exterior.
For chips with integrated CPU+GPU+NPU, it could be worth it tech-wise. The GPU and NPU can eat HBM bandwidth. For general purpose CPU code, the HBM would likely not be worth it. It's high bandwidth, but you trade latency, and general CPU code is branchy. Economics-wise, the HBM stacks alone will cost more than a consumer CPU (or APU).
[edit] The packaging needed to support HBM is also much more expensive too. If demand for current HBM applications tanks and the manufacturing lines need filled, then maybe. Currently, the price point would make it very very niche.
How so? FEOL is pretty much the same, BEOL is almost the same save TSVs, the packaging tech is different and more advanced, but not exactly 1:1 comparable. Do TSVs really occupy 3x the area of DDR IO's?
It doesn't really say that it needs to use 3x more area, but that 3x more gets consumed due to "advanced packaging and manfuacturing complexity". Which doesn't properly explain why it consumes 3x more and could simply mean they have a bad yield and 2/3 produced is garbage.
Classic DRAM stacks up to four wafers on top of each other and then is packaged with BGAs. The manufacturer can check the DRAM chips independently.
Soldering the DRAM onto a PCB is such a reliable process that there is almost zero risk of defects and even if a defect occurs the damage is limited. If the DRAM is soldered onto a DIMM the risk of a defect on the non memory hardware is non-existent. If the DRAM is soldered straight onto an SBC or GPU, then the DRAM can be removed to save the precious SoC or GPU chips.
Meanwhile HBM is the ultimate nightmare scenario. You stack up to 16 DRAM wafers on top of each other. One defect and the whole stack is worthless and that was actually the easy part.
In stage two things get even worse. You now have your accelerator chip and you must place the HBM on that chip. E.g. Blackwell GB300 has eight HBM stacks and the accelerator chip has a bigger area than the HBM. You must get the packaging right eight times in a row or you have wasted not only the DRAM silicon, but also the accelerator silicon because HBM cannot be removed and defects are permanent.
The issue here isn't just the yield of the HBM (which is obviously lower if you have taller stacks) but rather the yield of the combined HBM-based product, which is why doesn't make sense to say it needs more area but it is completely correct to say that HBM leads to more silicon being consumed. Hence it doesn't make sense to talk about yield of the HBM itself, because it is always part of an integrated product.
You make HBM by stacking a whole bunch of dies on top of each other. The signals from the upper dies need to pass through vias in the lower dies to get out - taking up valuable die space in a way which simply isn't needed with regular DDR. Similarly, HBM has a far wider bus, so each individual die has, say, 16 banks of depth 32, rather than 4 banks of depth 128. That's more control area needed per byte of memory.
Those two combined already result in a huge reduction in bytes per mm2, so with the same wafer processing capacity you're producing far less byte of memory. Add to that a complicated chain of HBM-specific packaging steps, and you're now also losing a decent bunch of perfectly-fine dies because rather than putting it into DDR you tried making a HBM sandwich and screwed up.
Even if the memory cells are the same and have an absolutely identical yield, HBM will always end up having a significantly lower output. That's just the cost of stacking, but some people are willing to pay the per-gigabyte price penalty in return for the higher bandwidth.
This might not be far off the mark. You are irreversibly linking the fates of these devices after a certain stage of manufacturing. If something goes wrong at final packaging time, you lose all dies instead of one.
The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.
So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!
These are not normal requirements, other than for a GPU.
There is a world where scale and experience means it's about the same though, is the thing.
And memory is already expensive. It's downright hard to even get it though - you frequently would prefer not what's cheapest, but whatever is in largest scale production.
Scale and experience almost entirely share between HBM and normal memory. And they're both in large-enough scale production to not have a big difference on availability; if you're willing to pay HBM prices you should find even more sellers of DDR.
The only way I see HBM becoming competitive for consumer CPUs is if they solve the yield issues. Or if AI crashes so hard that people are putting those GPUs on fire sale and salvaging mass quantities of HBM off of them.
It seems unlikely as the entire machines with HBM on them are apt to be sold whole on the market and snapped up quickly. With how much demand is in the current market machines 3 years old may not be sold if they can't get newer faster machines fast enough.
This will blow your mind, but it actually is pretty close to being unbounded. :)
Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth.
Nah his point is a bit simpler. Vector units can process a limited amount of data per time unit. Memory can load a limited amount of data per time unit.
If you have infinite memory bandwidth you just move the bottleneck back to compute so both have to grow simultaneously in lockstep.
What you should have said is that CPUs have so much compute headroom for matrix vector multiplication that simply adding more memory bandwidth would make them faster so every improvement in memory bandwidth is welcome.
The "move the bottleneck back to compute" bit is changing rapidly though. The first time a major hardware company ships a PIM chip, you can push for a few orders of magnitude more data through without being compute bound.
I think people might prefer the lower idle power consumption from lpddr over the better bandwidth in hbm in battery powered stuff. That said right now the price is definitely preventing us from finding out.
That hasn't been true since early HBM2 days, before the controllers standardized on power/voltage management and did things like leave them in P0 to ship on time
LPDDR/DDR/GDDR generally still win over HBM when there’s no data being transferred. HBM is primarily better in watts/byte transferred. Consumer electronics spend most of their time idle.
Right, but HBM idle is significantly worse than LPDDR idle or even DDR idle for that matter. That matters a lot precisely because the device is idle most of the time. Your idle power draw dominates.
This is literally a thread where people are saying “but why laptops no HBM”. It’s a perfectly cromulent response that meets the question where it is. It might be mildly a more defensible question for normal plugged in desktops but that would require a massive ecosystem change since people who build those traditionally really like their DIMMs and less buying a CPU that has non-swappable RAM built in, not to mention there’s no sustainable market for the higher cost and low volume product.
If LPDDR6 PIM ever becomes a thing then the advantage of HBM will shrink. I'm not saying LPDDR6 PIM will make HBM obsolete in the datacenter or where it is currently shining, I'm saying that large volumes have their own charm and it is more likely for LPDDR6 PIM to be in your laptop or smartphone or SBC than HBM.
LPDDR6 PIM would primarily help the low end and mid range accelerator market. E.g. embedded models running on SBCs can be up to 1 GiB in size with acceptable performance, small models at 8 GiB become really easy to run at reasonable speeds on a smartphone and PIM enabled laptops or mini PCs make it possible to run 32 GiB models locally without compromise.
Of course this also assumes that the associated accelerators (NPUs) will catch up too, but the general point is that you won't need a 5090 or a 4090 anymore.
Mainly bus width. Afaik HBM is like 1024 bits vs DDRs 64 so you need lots of transfers in parallel to saturate the bus, and CPUs kinda want 64 bytes of data as that's the size of a cache line ASAP. So you need a ton of in flight transfers which isn't a thing CPUs provide, maybe multicore workloads.
Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
Even if the cost were the same of the RAM itself, you'd need much bigger and more expensive CPU to deal with it.
Running 17 chrome tabs doesn't benefit at all from that HBM and all the additional hardware+software complexities that come with it. You want a specialized coprocessor to handle specialized workloads. The GPU exists separately from the CPU for a reason.
In one sense, nothing, in another, everything. It is DRAM, but the bandwidth requirements mean it’s paired to a processor, i.e. no DIMMs. Not 100% sure but things like MacBooks and the Framework tower, where you have fixed RAM for the device lifetime, have ~0 tradeoff.
HBM isn't dramatically better in every dimension. You get huge bandwidth and good energy efficiency per bit transferred, but not necessarily a meaningful latency improvement and capacity expansion becomes tied to the package
If they're talking about production capacity, that is some product of die area and process steps, right? It doesn't have to be 3x die area, just 3x lower factory throughput for the same number of functioning memory bits.
Yup, the only reason Macs have higher memory bandwidth is because they use more memory channels, which gives them a wider bus. Both Intel and AMD only allow more than dual channel memory on server class processors these days.
The “Apple only does X better because they do Y” thing has been a meme for ages. I remember dismissals like “PowerPC is only faster at math because it has more integer units” or something along those lines, and thinking, uh, isn’t that a good thing?
There's a qualitative difference between "they're doing a different thing" and "they're doing the same thing, tuned differently". GP is saying that this is a case of "they just tuned it differently".
This distinction doesn't change what the performance numbers look like today, but it does inform what changes would be necessary for those numbers to look different tomorrow. E.g. Apple Silicon isn't fundamentally orders of magnitude more efficient than x86, they just used smaller features. Newer Intel and AMD chips made on equivalent processes _also_ get similar efficiency gains.
There are AMD and Intel devices on similar process (not talking about A20Pro or M6 which are set to ship later this week), and they do not get the same gains.
And honestly, they have historically had different markets.
When the design is for only one customer, you don't need to generalize things, and those things you generalize to give different customers different options has costs.
AMD will soon be a larger customer for TSMC than Apple (NVIDIA is already there) so Apple's pre-booking new processes is likely to be gone in the near future.
I am just saying it's not magic and x86 is capable of doing the same. Quad channel memory used to be more common in consumer hardware but now it looks like you don't even have the option anymore for desktops. AMD's Strix Halo was the first sign to reversing that (it has quad channel memory!) and hopefully we see more of that in the future.
AMD does market segmentation and limits consumer chips to dual-channel. You need to cough up the dough and get Threadripper for quad-channel. Or even more for Threadripper Pro to get octa-channel.
Kind of but not really. DDR5 splits your typical 64-bit channel into two 32-bit subchannels meaning the bus width is not increased. These subchannels are not always advertised because it's a just a feature of DDR5. Actually adding more channels increases bus width, which is what meaningfully improves memory bandwidth.
Keep in mind that non-Pro threadripper is still only 256 bits wide and Pro is 512. And the memory is 30% slower than with an M5. So an M5 Ultra has 3x the memory bandwidth of the best threadripper.
Moving memory around is a bottleneck. Non-unified memory usually means copying over the PCIe bus which is way slower than RAM (and way way slower than VRAM). Actually unified memory means you don't need to copy anything at all though which is the absolute best case for performance.
It isn't really, as long as you don't care about power consumption, physical constraints and money. Basically desktops <2025 (and hopefully >2027).
DDR is optimized for latency and stability at the cost of bandwidth whilst GDDR is optimized for bandwidth at the cost of latency and stability. GDDR is pushed so hard these days that a small percentage of errors is expected and corrected because this is still faster than running it slower but more accurate.
GDDR7 often has 10-20x the total bandwidth but 3x the latency of DDR5. Graphical workloads want as much bandwidth as possible but care relatively little for latency. Conversely, applications love low latency but don't really see any performance benefit from higher bandwidth.
So basicallyt you have workloads that are diametrically opposed and running unified memory forces you to compromise.
Is GDDR7 that's ECC then just the super well binned stuff (plus the ECC parts added) ? I've been wondering why we don't see more cards using it as they perform pretty damn well. Take the Nvidia 6000 Pro Blackwell for example. The compute is insanely fast assuming you can fit what you need in 96GB of ECC GDDR7
The price is that you essentially glue CPU and GPU together, which limits total compute, from a size and thermal perspective.
This is really not a limit because of unified memory -- in principle, PCIe GPUs could read/write main memory without the CPU. But it's a limit for /fast/ unified memory, because fast means close.
So unified memory is great as long as the integrated GPU is strong enough. Then it has two advantages:
a) probably faster transfer CPU<->GPU (but that's an implementation choice for the non-unified case
b) If you either need a lot of memory for the CPU or the GPU, but not for both at the same time, you pay for memory only once.
In theory perhaps but the benefit is not as relevant with a weak iGPU. In practice all PC's with performance ambitions had a dGPU until Strix Halo and Panther Lake.
No I just failed to check before I posted, I thought it was HBM. I don't have any interest in Apple hardware so wasn't properly informed, and have been duly crucified.
More channels or soldered memory? Channels are basically RAID 0 so it depends what you're measuring. Soldering memory down was the only way to use LPDDR5X so if you wanted the best memory you had to solder it down. LPCAMM2 exists though so newer devices can use that instead of soldering them down, but not all devices would be able to fit the required LPCAMM2 slots.
That's tiny! But it still depends a lot on form factor. LPDDR5X is used in phones and making memory removable would have its compromises. You may even have compromises in laptops. Look at how tiny a MacBook Air's mainboard is and you'll see the RAM modules on the same package as the SoC. SOCAMM2 is too large for that but a variant with only two modules could possibly work.
The normal non-tech-savvy person already does this. They simply don't know that their ram is upgradeable or something else breaks first, before having to touch ram.
what is this too much RAM thing you mention? I thought the only valid RAM situation you could find yourself is not enough RAM. Too much? That's just fantasy
HBM4 and HBM4E DRAM are NOT the DDR4/5 that consumer markets need. Capacity allocation is leaning more to data center grade HBMs so less to produce dedicated DDR4/5 DRAMs. Supply demand will further drive up the consumer DRAM price! Note, the article mentions NO of new fabs is being constructed (all semiconductor manufacturers know that constructing more fabs means the boom/burst cycle will eventually kill them, so no one create more fabs) Perhaps the federal government need to step in here - the market doesn't fit the issue.
No, completely wrong. Supply and demand will definitely fix this.
More HBM4 will drive the price of it down. That will make it less profitable to produce. That will mean firms shift some production back to more profitable "consumer" RAM. Increased supply will lead to lower prices.
What specifically did that say that you think is wrong?
Supply is currently moving the wrong way. Supply and demand for consumer DRAM will get even worse, even if it will "definitely fix this" in the long run.
And the fact that production is moving the wrong way suggests that in the short term supply and demand is actually doing more harm than good. HBM is so profitable it's radioactive.
I don't know where you got the idea in another comment that all memory will get "very cheap quite soon". They're not increasing total memory production. (much/yet)
China is about to massively compete. And doesn't Micron and someone else have new fabs in NY coming online ? I can't imagine they're going to exclusively make HBM.
Ram always goes through these boom and bust cycles. They're going to increase supply but not so much that they're eventually left with huge overproduction as usual.
Of course if Google researchers come up with more ram reduction tricks for newer AI architectures that could make a huge difference as well
I don't expect China to be a particularly large competitor in the next year or two affected by the decision in the article, but they'll help a lot in the long run.
Micron's NY fab had a press release two months ago that they started pouring the first concrete, so that's far from coming online.
If AI demand sticks around and buildouts stay cautious it could take much much longer than usual to build up enough capacity to push prices back down below $3/GB.
More efficient AI memory use risks even more demand for memory because you're now able to get even more performance per dollar; the benefit of extra memory has to be sufficiently far into diminishing returns for demand to drop overall.
> What specifically did that say that you think is wrong?
You said:
> the market doesn't fit the issue.
I presume you meant "fix" not "fit". You are wrong that the market will not fix the issue. There are literally millions of observed cases across many markets of increased demand (or decreased supply) leading to higher prices leading to increased supply leading to lower prices, which is why it's a fundamental law of economics.
I feel like everyone here forgets that the DRAM industry is operated by one big cartel. The rules of supply and demand don't apply here. Now that prices are up, they will keep it up, cutting supply as needed in order to do so. Every big DRAM manufacturer in the industry has been charged with price fixing, among other things: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing
There are so few players in this space they can coordinate and collude pretty easily.
DRAM isn't nearly as hard to get into as compute. The price fixing stuff is going to bite them in the ass someday soon by massively increasing the incentives for new players to enter the market and compete.
Also they're talking about the short to medium term. On that time scale they're not wrong. The market is so far from fixing it right now, it's going in the opposite direction. The market is chasing HBM money which is outbidding everything else.
> The price of RAM will fall.
And if that might take 5+ years, that's not much comfort.
> higher prices leading to increased supply leading to lower prices
So far higher prices are causing barely any supply increases. (Note that the article we're commenting on is about a supply decrease.) Companies are barely trying to expand, and even if they do start trying hard it'll take a very long time.
Yes. Non-HBM Samsung supply going from 60% to 20% will cause prices to increase, which will cause increased production of non-HBM, and then prices will fall (below where they currently are).
Depends on what proportion of Samsung's output is current HBM4 and HBM4E DRAM.
Everything in their statement can be true and it be a bad thing for consumers of non-HBM RAM.
"HBM capacity to expand to 250,000 wafers a month"
So let's say their current HBM capacity is 100k wafers/month (pure speculation/random number for illustration), and their total RAM capacity (including HBM( is 300k wafers/month, then non-HBM capacity reduces from 200k to 50k.
And it's worse than that, Micron said that converting capacity to HBM was at a 3:1 ratio, so that 250k HBM could be up to 750k in non-HBM. Fortunately, some of the increase is due to improved processes.
Keep in mind that total manufacturing capacity is increasing as well. Perhaps not enough to fully offset, but it's incorrect to assume that supply capacity is flat.
Micron has been working on expanding in Boise since around 2023. They are predicting that the first new chips will start rolling out around 2027.
I don't think we'll see any relief until at least 2028 or even later. And even then, probably not without antitrust laws being enforced. Because you know memory manufacturers won't just simply drop prices, they have a long history of price fixing.
Question is what does the demand curve for regular ram look like also? If regular production is on a 10% increase curve but a 40% demand curve than we've been at the samw place.
Part of the implication is that factories that could be producing consumer-facing DRAM like DDR5 would be retooled to produce HBM instead, leading to even less total consumer RAM production.
SK Hynix's new Cheongju M15x fab was originally intended to be for DDR, but mid-construction they shifted it to HBM. Similarly, the Icheon M10F packaging line has been retooled to HBM, allowing it to assist in processing the output of other fabs.
And this was news early last year, so who knows how far it has progressed since then.
If Samsung is limited in the number of wafers they can process per month and they use more of those to produce HBM they necessarily have less of them left to make other products, like consumer DRAM.
I have read an interview with Acer's CEO claiming that the shortage is already over.
He alleged memory maker's public announcements of the shortage lasting throughout the end of 2027 are attempts to covertly signal the others on pricing strategy.
With additional production capacity ramping up this year and next, this seems somewhat plausible.
If manufacturer's inventories are filling since they advanced orders when prices were rising, there could be an oversupply of memory when they don't place new orders at the higher current price.
This could quickly result in a price war while capacity expansion and yield improvements create irreversible oversupply.
It matters so that average people have a chance to own their means of production, and that the current hardware-grab by entrenched capital doesn't get to charge such high rent.
Frontier AI labs are customers just like the average Joe down the street. The only difference is who can bid more. Average Joe probably doesn't have millions or wont buy millions of chips.
There are no privileged consumers that have "dedicated" amount of production, no matter the demand.
What point are you making by saying "they're just like the average Joe" followed immediately by saying they have so much more money that they can outbid the average Joe at any price point.
If someone can dump 100x the money to get something I want, they're not "just like me".
I mean... You can buy a stick of RAM with Samsung chips on it and use it in your own machine to run a local model vs. paying for an Anthropic subscription. I assume that's what they're referring to, not owning Samsung stock.
The solution would seem to be preventing that "Buyer (AI) has crazy money". They already have borrowed billions and are only generating losses so the money should run out. It just needs to be made sure that nobody (taxpayers) will rescue them.
During the Covid global chip shortage Intel announced new factories to address the production bottleneck, before the factories even started to be constructed, the shortage was long over and the projects were eventually canceled.
What if the AI companies start selling the hardware with their models?
I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.
That's never going to happen. By the time you can run current frontier models on your $10k desktop the frontier will have massively advanced and people will want those models instead.
I’m not so sure about that. Already AI vendors are back to cutting prices to try and keep customers from cutting back on their usage. My own employer is working hard at pivoting to much smaller fine-tuned models for established use cases, and seeing model performance improvement in addition to large inference cost reductions. Being able to run them locally hasn’t exactly been a disaster for devex, either.
It may turn out that demand for SOTA frontier models isn’t so limitless after all.
Every product follows demand curves. At a price of 0 you could find infinite usage. This has nearly zero relation to how much it costs to provide the product.
Infinite demand isn't a thing. Even if they'd offer free compute forever (not likely possible) I and many others would still use local models that we have full control over and that do not harvest our personal data.
Except of course it relates. All else being equal, we will prefer $X COGS over $2X COGS because that helps us with both profit margins and price competition.
Framing it in terms of the price people might be willing to pay for a single product in isolation frames the point I was making, which was about price competition, right out of the picture.
Maybe I'd be willing to pay $10 for product A if I had other options. But if there's a product B for $3 that's not quite as nice but still ticks all my boxes, then product instantly becomes a lot less attractive.
This + your ROI in $10k device will be always lower than busy datacenter, its literally math. They sell free compute to others when you dont use it, you will never sell at that level or even you magically sell home compute, you will not compete at price
That greater efficiency only benefits the LLM SaaS providers as long as the hardware manufacturers, probably especially the VRAM manufacturers, remain supply constrained, since the high-efficiency users are the ones who can pay top dollar. But the hardware guys' dream is presumably to get parts into millions of laptops which remain on standby for 19 hours a day, not to bargain with SaaS providers who obsessively optimise their memory consumption.
We're not seeing the progress in those "frontier models" that we have previously seen. There's certainly still gas left in tank tank, but we're way into the diminishing returns by now.
Cloud inference still beats hardware investments by orders of magnitude of course, but that's only if your data doesn't really matter to you.
I’ll grant that for specialized applications like coding agents and mathematics, but even there I suspect that most the real gains are actually taking place in the harness.
But I suspect returns may have already diminished into negative territory for at least some other use cases. One of my least favorite job responsibilities in this brave new era is figuring out how to avoid performance and behavior regressions when an older model were using for some application reaches end of life. It’s getting uncommon for me to look at our benchmark results and say, “Oh, good, it does better on one of the newer models!”
That is true, but the eventual realization that more machines doing more coin flips in parallel does not mean "more work gets done" might.
LLMs are amazing tech, but they're terrible without oversight. More agents faster just makes reality collapse on them quicker.
But yeah, you're right, temporarily, this will still push demand. But the topic was about "diminishing returns" as in "tech getting better". Not as in "customer spending".
It's kind of weird because more machines working together does mean more work gets done. Coin flips and weighted coin flips are totally different things. Any biases weights towards reality push you closer to reality when you use them.
New models keep being able to use more and more agents on longer time frames. Your hypothesis doesn't look like what we're measuring.
But high demand for LLM time isn't sufficient to keep customers at the frontier LLM SaaS providers. That demand can be satisfied locally or at non-frontier outlets, absent hardware shortages at least. The Tier 1 providers (and the would-be Tier 1s) presumably need to open up a much bigger lead in model quality, one that doesn't simply get distilled away this time, and/or continue to be protected by ongoing (or worsening!) hardware shortages. (And that's overlooking the revenue shortfalls which OpenAI and Anthropic seem to be facing already.)
I had actually been thinking more about all the non-LLM functionality that go into the harnesses. I'm not going to name names and I haven't done any rigorous testing, but my general impression is that choice of harness matters more than choice of model. In terms of basic task completion success specifically, not code aesthetics.
A perfect harness will not extract gold from a dumb model. It's a system that builds on each other, though we've not probed that frontier much to have a good intuition on what effects what.
One thing that I really want to know - the better models from today vs a year ago - what has changed. They have already pre-trained on all available public data. Scooping up the last percentage of archaic texts which were never digitized is not going to move the needle.
Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?
And you apparently only read the title and ignored the content.
It’s also poor reasoning to you only pick one thing you think supports your view, and ignore vastly more things not supporting it, all from the same useful criteria.
Now if only you’d carefully read the report you chose, and spend equal time looking at ample presented evidence, you’d develop a more accurate understanding.
Well I mean if I wanted to be extra pedantic, I would argue that we've been in that phase since LLMs were first introduced.
Before that, we had 0.
After that, we had more than 1.
A leap as far as that is hard to recreate.
But that wasn't my point. That's just trolling.
The actual point is that LLMs aren't gaining new capabilities anymore. They just get more reliable at the ones they already have; turning what was a coin flip to some higher probability.
That's (intuitively speaking, not strictly mathematically speaking) kinda the mathematical definition of diminishing returns.
It’s a constant tension in computing that has been around since mainframes and clients… Neither is going to disappear. My general feeling is normal people care more about how thin and light something is than their privacy, so if data center powered LLMs will have a strong future.
Edge based storage is very very viable, and very very cheap, people still use Apple / Google Photos (consumers don’t want any friction).
Coorperate people likely just want “the best” and are happy to pay a lot for it to make their workforce more efficient.
It’s hard for me to guess where the data center buildout will go, eventually there will be an oversupply, but we haven’t hit that yet, and at the moment the companies owning these high capacity data centers appear to be making lots of very real money in leases to hyper scalers, hard to imagine this premium continuing, but we need to see where demand for cloud based LLMs tops out.
It's going to happen very soon, which is why these frontier labs are scrambling to shut down open source language models. There's an existential risk threatening their obscene returns.
I feel like cost competitiveness of local has been going down, if anything, not up. API providers can use hardware more and have scale efficiencies. Do you see any reason this will reverse?
Because everything is converging on a backlog of huge efficiency gains established in research, waiting to be combined. Looped transformers, a whole host of diffusion techniques and new quantization techniques, maturation of ternary distillation and new ways to separate logic from stuff that can be looked up. It would surprise me if most frontier models were actually even that big at that point in terms of active params. I highly doubt it.
But he gave a reason for that. "Before that happens the frontier will move". Why do you think it will happen anyway? Do you think the frontier will not move fast enough that local models are unable to catch up, or do you think people will prefer local models at a point. Or something else?
You don’t need frontier-level performance for every task. That’s why companies hire both junior and senior developers. I suspect a decent percentage of people using Fable would probably be fine with Opus.
Also, the frontier can’t keep advancing at this rate forever. Eventually the low-hanging fruit will all be gone and advances will slow down.
Seemingly the AI bulls are either not sufficiently optimistic, or not sufficiently wealthy(?!), to build new fabs as joint ventures with the manufacturers in which they agree to assume most of the downside risk?
They signed long term contracts for the hardware delivery over years. AI investment is approaching 1 trillion per year and expected to grow to well over 1 trillion per year. [1]
Even over their many years, projects like the Manhattan Project, the Apollo Program, or the U.S. Interstate Highway System never added up to that. [2]
Is your argument that there is insufficient AI spending at present as they aren’t also taking on building their own fabs?
If the AI bubble bursts, the DRAM manufacturers switch back to making conventional DRAM and pent-up business and consumer demand results in a surge of purchases, cushioning the fall considerably.
Look into the time frames a bit more, this isn’t a switch they flip. If the bubble bursts today it will be years before those downstream effects add up.
Not at all, I’m going to relax with a cooling drink and watch the consequences of this unbelievably destructive and wasteful venture implode. I also adore the idea that it's on the customer to be "fair" to a company that dumped us a group in favor of chasing B2B money.
I think they're talking about the past: memory has often been boom/bust. Memory manufacturers lose money during gluts, and did so long before AI. People benefited from cheap prices at that time.
In fact as a business opportunity it did just that because the massive investment in the mid-late 1990's was without anything like a connection to profitability or sustainable demand. It took years after that crash for the concept of e-commerce to begin both a recovery and evolution into what we see today.
I expect something similar for AI. It's useful tech... just not very profitable tech, and certainly not to the tune of trillions of dollars worth of public demand. The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises, but the tech will survive and thrive.
>The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises
The hype will die but there are many reasons why super intelligence and robots are a separate entity from said hype.
Look, back a few decades and tell someone our gdp would be in the trillions and it's likely they'd have a hard time believing you. A huge portion of our products that we use day to day would be complete science fiction to them.
All we're negotiating at this point is the timescale it will take.
These things aren't really unpredictable. Work long enough on a robot with more dexterity and you will get just that. The timetable on when it happens is a bit more up in the air, economic downturns and wars can have huge impacts on it.
But in my mind robots and superintelligence are inevitable unless there is a massive setback in humanity before then. Nature already did it once. We're not inventing something totally new. Add in every new invention and bit of intelligence we build on and actualize pushes us that much closer to the goal. Information technology allows this to speed up even further with easy sharing of information and testing.
Superintelligence is not like faster than light travel. We have many frameworks that show us FTL is impossible. I don't believe there is a single widely accepted framework that shows there is some limit to intelligence and we're near it. Intelligence isn't even a singular thing. My calculator is a super intelligent adder compared to me. It would seem highly improbable that somehow nature random walked into making brains the most efficient general intelligence device in all dimensions and scales.
Samsung is already building another fab (which was originally suspended due to low memory prices years ago). Hopefully when it goes live in a few years it's not all dedicated to HBM for AI.
Right now, every company that can is ramping up their RAM production as much as humanly possible. There is such outrageous and growing demand, that it's like printing money if you can manufacture more RAM chips. And that massive financial incentive is driving huge investment in manufacturing capacity.
But someday, that demand will stop rising.
At some point, most people will have enough RAM. And they just won't need to buy more.
The inventories of RAM will become overfull, in warehouses and stores. And suddenly the manufacturers, middlemen, and even retailers will all find the price they can sell at keeps dropping.
But you can't shut down a huge manufacturing facility overnight. You can't shut down a years-in-the-making retail distribution system overnight, either. You can't even shut down a facility that is halfway finished with construction. They'll keep producing and selling RAM at lower and lower profit margins, until the day comes they finally accept they're losing money with each sale.
And when that happens, we'll all have more RAM than we know what to do with. It'll be cheaper than we have all ever experienced in our lifetimes.
I don't expect this to happen soon. But demand cannot keep exponentially rising forever. There will come a point, probably in the next 5 years, when demand at least levels off, even as production capacity is still rising.
And it will be interesting to see what's possible at that point. We'll be able to create software that is not doable right now, because it will no longer be constrained by memory. And even cheap laptops will massive amounts.
Yeah, that's kind of the beauty of capitalism. It can lead to short-term shortages but never structural ones. Any demand gets immediately fulfilled.
I grew up in a country where there was a structural shortage of say, shoes and cars. You didn't really even get to choose which shoe you wanted to get, when there was a new shipment, folks went to the "department store" and got whatever size they could get their hands on and later trading it for the right one. To get any car, you would submit an application and 5-8 years down the line you would get a letter that your car would now be ready to be picked up, in a random color.
Thanks for sharing that. I am grateful every day that I was born in such an abundant environment. I have a few friends who did not, like you, and it always reminds me of how lucky I have been.
This bizarro take, HBM is a triumph of non market, industrial policy. HBM is product of autocratic chaebol industrial strategy + markets (which =/= capitalism) at work. The reality is SKHynix corporate autocracy kept nonprofitable HBM going for 10+ years until it turned into unobtanium by chance, any normal capitalist+market system would have strangled HBM dead years before - Murican brand capitalism - Micron couldn't have done what SK did. HBM is literally inefficient Chaebol department store holding stock of useless misallocated shit 10 years ahead, and through sheer serenity (arguably in face of SK mismanagement) became the trendiest product around, and now positioned to reap rewards from market signals. Meanwhile memory companies that operated off capitalist impulses (intel, micro, gf) years head and can't respond to signal at all.
When you commoditize intelligence, you dramatically lower the cost to create. The amount of electronics being created is... incredible. We've really seem to have hit the asymtote part of the growth and i don't really see it coming back.
Everything is moving to edge compute. From smart rings and watches, to robotics and drones, iphones and humanoids, and no end in sight to the scaling improvements in AI... It does seem that you can continue to get more emergent behavior the more compute you expend in pre/ post training. There's no reason we can't scale up to a million data centers in space, as long as you get dramatically improved intellegence.
Yes, only a small number of companies produce memory because of capital requirements and complexity. When prices go up both of those things become less of a problem for investors who can see better return for lower risk on that high capital investment. They will then invest in production to capture those returns. This is literally the exact reason Samsung are doubling supply of this particular type of memory.
Memory factories cost billions to tens of billions, "investors" aren't going to spend that kind of money if they are not reasonably certain that they are going to make a profit, and the market is very uncertain right now with everybody bracing for a bubble burst at any moment. It's not the first time there has been such a shortage in the computer parts market, and the oligopoly, be it RAM or hard drives manufacturers, has always managed to keep prices high and production low enough to avoid a price crash after recovery.
I'm sure you can make it work with HBM chips on adapter boards for PC. And there should be lots of HBM chips from scavengers taking apart bankrupt data centers.
HBM dies are designed to be physically mounted on top of other dies in the same package, they can have up to 2048 data pins (compared to 8-64 on consumer dies which are in their own packaging) - to do what you propose you'd need an interposer chip of some type which would push the price back up
To me, the most idiotic thing is that the world currently cannot build power plants fast enough to even use all the HBM that is being produced. That is, a large portion of HBM and cards are sitting in storage, waiting for the data centers and the power plants to run them. The FOMO in the AI world isn't just in the stock market
About 2 years ago I bought some SDRAM or something like that, DDR4 or DDR5, don't recall offhand. A few months ago I looked at the price today, and it was over
3x as high. That's just insane. Governments need to do something against this abuse system that AI amplified here.
> Governments need to do something against this abuse system that AI amplified here.
Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.
No one has all the cards and no one controls all the levers.
Governments have sued DRAM companies for price fixing and forming cartels before: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing. They can absolutely step in and if there are evidence of price fixing or anti-competitive behaviours, can impose fines. The choice of whether to do that though is political.
And on that page govts have tried and failed at it also. Thinking the current prices are the result of fixing and not insane demand is surely going to lose in court. It’s trivial to show massive demand, and thus production line changes, as the driving forces here.
If you look over all such lawsuits, they extremely rarely succeed.
Is there any allegations of price fixing in this case? The demand just skyrocketed. Other than legislation like the defense act in the US I don’t know of any governments that have jurisdiction over these ram makers that mandate companies produce more of a product.
No one has all the cards and no one controls all the levers.
I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
I don’t really know much about how this stuff works, but I have a feeling we wouldn’t even have to do anything punitive. We might just need to find and close whatever weird loophole allows the folks who are participating in this gold rush to feel confident enough that they’ve externalized their risks to be willing to engage in speculative data center buildout projects on such a grand scale in the first place.
Apple used to do the same thing with TSMC’s newer process nodes, and nobody really complained because people were getting superior products. I happen to like US antitrust enforcement that considers consumer benefit primarily.
The funny thing is, we're mostly in violent agreement, going by your other comments. This is a weird point to disagree on... so weird, in fact, that it's easy to believe you are associated with OpenAI. Hence my follow-up comment.
What Altman did, and the way he did it, was not OK. Unless you believe that humanity's adoption of AI-based intellectual work is best served by concentrating an enormous amount of power in one company at the expense of everyone else, turning the market into a literal zero-sum game. Do you?
The same major memory makers got prosecuted for cartel behavior by the DoJ a little over 20 years ago[0].
One might say "well what specific law have they broken" and I don't know and I don't know if they have, but at the very least the power of the State _could_ be used to force behavioral changes in some ways or another.
Well, thats kinda how strategic reserves work - so that say food manufacturers just don't dump all the cheap food to sell crack instead & everybody starves to death.
No. The other industries can bid for access to the resource, and under most circumstances, whether they are willing to bid higher tracks importance. Sometimes there are cases where something has significant positive externalities and so people are not willing to bid high enough, and then we can talk about some kind of government intervention (typically a subsidy that attempts to track the value of the externalities) but I don't see that applying here.
If others are willing to pay more, that a signal that more value is created if the resource goes to them rather than to the more price sensitive customers.
This logic falls apart completely when gamblers allocate one trillion to bidding up the price, denying access to the resource to people who are responsibly spending their own money. The ideal world of the imaginary perfect free market never takes into consideration the messiness of the real world, like the fact that people will spend extreme sums of money irrationally. It can take years for this effect to correct, damaging the market severely in the meantime.
Alternatively, the money invested can be rational because it prices out competitors and establishes a monopoly, after which point the monopolist earns their absurd investment back with complete control of the market. This is also bad.
You can reach any conclusion you want if you assume others are behaving irrationally. But why should I assume they are wrong instead of you being wrong?
This has zero to do with the company that is making the product and all to do with the company buying them.
The problem you have here is hundreds of different companies are doing the "gambling" in a non collusionary manner so it's going to take decades in court to prove it. Are you saying governments should do an authoritarian take over of RAM allotment?
Yeah the 300 already felt like a splurge - it's the max. 96 was less than 200 I felt kinda stupid not just going with that. Imagine 96 of anything now for less than 200
It's not just normal consumer memory. Everything with a chip in it is getting more expensive because of the AI circlejerk. When Bitcoin screwed over consumers, at least normal memory prices weren't affected (though that time a shitcoin based around storing large amounts of data made the rounds, hard drive prices did take a brief hit).
Everything is affected by the endless thirst for RAM by AI companies and the accompanying hyperscalers. I'm honestly disappointed governments aren't doing more to keep the price of consumer goods in check.
I guess the AI companies telling the world how good they are at accidental cybercrime and fraud is creating enough FUD to let the AI industry run wild.
The article features a truck that the chips get loaded onto. This quote from Tanenbaum immediately jumped into mind:
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
I was wondering how much financial value they put in one truck. Presumably the value density of this stuff is incredibly high (although not really to anyone else, so security probably isn't a huge issue). At what point do you split the shipment between multiple trucks to avoid huge losses if there's a crash?
When I worked on satellites we had some electronics which used a radiation hardened FPGA which cost something like $500k per chip. Carrying that around was nerve wracking.
I had a cousin that worked private security, and for a few years (I think it was early 2000's) his main gig was to work as a team of armed security, and follow trucks from the two Intel campuses in Oregon to the airport in unmarked vehicles, get on the cargo plane with the cargo, and fly to the location where they did the packaging of the wafers (I want to say it was the Philippine Islands) and then fly back. Lots of overtime.
I once had to keep a $20M (technically; a million download codes for the same $20 movie for a promotion) thumbdrive in my home safe because it arrived too late to put in the safety deposit box like it was supposed to be.
I had to remind myself a few times the practical value to anyone was about $22.
I worked at a [Not to be named Country]'s monetary authority. Part of everyone's duty was to receive shipments of newly printed cash from De La Rue and get it into the vault before it was distributed to banks. Seeing millions of dollars packaged in cardboard boxes being moved from the street to the vault with men posted up in the bushes with machine guns was something I will never forget.
It's funny how rote and regularized "playing" with money gets after a while though.
I used to work for a financial institution small enough to sometimes need someone from the back office to help fill in as a warm body for dual-control when we received cash shipments.
It only took a couple sessions in a vault to get entirely bored with having a half million dollars in front of you.
My first serious software was for a sizable goldsmith factory. They keep an incredibly low profile visible from outside, and they get really good at it. Once inside, the amount of gold that's moving around is jaw dropping. After a while, most of the time it feels like aluminum, or any other material. All you see is cuts to be made, welding to be done, carvings, sandblast, etc... It's work and responsibility and everyone is just trying to get it done and make it the problem of the next person.
My dad was a goldsmith! (Who eventually ended up doing software for jewelers!)
I definitely know what those small brown boxes which ship UPS ground look like. Very normal.
Why did that have to be transported in a thumb drive? It can't have been more than a few hundred megabytes at the most, right?
Enterprisey client chain-of-custody requirements. It was couriered to us, and the courier was late. Cue panicked phone calls to get an exemption to the contractual requirement for overnight storage at the bank.
My buddies company shuffles chips around the west coast, some deliveries are a few million dollars. They shove the chips into the truck, then shove a giant movable wall, leaving about an inch or two gap around the perimeter.
The truck has multiple drivers and private security.
It's treated the same way as any other generic shipment of cargo, really. It's fine throwing $20 million in one truck.
Security is an issue, though, these are treated as a high value load and typically would be followed by unmarked, plain clothes armed security guards tailing around the truck. (That's how Apple often does warehouse deliveries of trucks full of iPhones. It wouldn't surprise me if there was going to be some rolling lead soon following some trucks full of Duos...)
They are getting hijacked/stolen and disappearing.
https://www.wired.com/story/the-worst-ive-ever-seen-cargo-th...
Likely ending up on a container to China, export bans be damned.
They probably have insurance which means it doesn’t matter if it is all in one car or not. And even if they didn’t Samsung is big enough where a loss of a couple million dollars of components isn’t gonna bankrupt them.
Ha - I still remember when I worked on the IBEX-Lo Sensor, we had to hand carry it to Switzerland. One lucky member of our team got to fly first class with it because it wouldn't fit in a coach seat!
Almost 10 years ago at my university in Chile. My professor of Computer Networks invited a PhD Student that was working on the ALMA Observatory [0] doing some research on signal processing.
What stayed with me from that talk was that all the data that was captured from those massive antennas was packaged in boxes, loaded in a cargo van and travel 1.600 kilometers to Santiago to be uploaded into the systems.
[0] https://www.almaobservatory.org
Amazon did this initially as well for large data pools when launching AWS S3 etc.: Sending you a truck/container to put all the disks in:
https://www.supplychain247.com/article/how_do_you_get_all_of...
I wanted to mention AWS Snowmobile [0], but after looking it up I found that apparently it was discontinued [1]. It seems that nowadays there are more economical/efficient alternatives (I guess theoretically you could still beat these alternatives in certain metrics, e.g. by using more trucks at once to get more bandwidth).
[0] https://aws.amazon.com/blogs/aws/aws-snowmobile-move-exabyte...
[1] https://www.cnbc.com/2024/04/17/aws-stops-selling-snowmobile...
I forget how much it is exactly now but I saw a short video on the bandwidth of moving modern cold storage down the highway and its actually nuts
Just hope nothing changes the latency is terrible.
I’ve often wondered about die thinning. I mean just looking at sd cards and micro sd etc, it was clear that there had to be a thinning step (or many steps) and it seemed amazing to me that such a complex step could be economically viable. And yet it clearly is. All a matter of being able to operate at scale I guess.
It’s not a step I’ve seen discussed very much, so it was kind of neat to see it discussed front and center in a lay article like this.
Planarization has always been part of the process: https://jeez-semicon.com/blog/Planarization-in-Semiconductor...
https://resources.pcb.cadence.com/blog/2023-the-planarizatio...
Not dissimilar to lens or mirror making, the process of grinding to a precise flatness. The key step here is bonding it to a glass carrier, presumably because otherwise it gets too thin to support itself.
See also the "reduced kerf diamond wire saw": technology critical to cheap solar panels is the ability to saw the boule into thinner and thinner pieces with less waste at each cut, like a big ham slicer.
Yeah, the scale is kind of mind-bending. You're taking something that's already extremely fragile, grinding most of its thickness away, handling it through several more process steps, and somehow doing that cheaply enough for mass-market products.
The critical back end step for HBM is thermal compression bonding - where they actually compress 13 very thin die separated by solder balls. Cracking is a major issue.
We are getting so many technologies like this that are so insanely specialized and require such deep knowledge of physical processes that some event like a 1918 flu that kills 10% of the population could take out enough people that we'd be unable to make things like this for years as we fill in the missing knowledge workers.
It's just crazy how far down the scale we're tickling atoms these days.
Maybe Covid killed a smaller but nonzero amount of technologies?
Surely we'd have had the same issue during the Spanish flu? Any examples?
Not really at all. Between those dates and now the world has become fully global and concentration of knowledge has massively occurred. The past world was much more highly distributed, far less technical, and far more redundant.
Automation, automation, automation.
Zero chance you could do it if a human had to touch the die/chip directly during those steps.
But clean room + vacuum pick and place machine? A matter of fine tuning, and it should work just fine.
The skill is in the fine tuning, probably.
And the not screwing with it while it’s still working hah.
Another hard-won lesson, no doubt.
On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.
The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.
The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
The only thing the West is achieving here is that sooner or later China will be able to compete on their own terms rather than ours. It may buy some time but the end result is very predictable.
As other comments pointed out, that was always China's plan, the value here is really that when the Chinese overlord annillate the world's industrial competition, they'll have the moral high ground of being out of self-reliance and hostility by western monopolists.
Edit:
And, that, my fellow keyboard high-minded thinker but powerless corp peons, was also part of the China's plan all along.
China's plan to be self-sufficient in semiconductors was a failure before western sanctions, because all the Chinese chipmakers found it too easy to buy western chipmaking equipment or to fab through TSCM. No Chinese company wanted to buy from other Chinese companies. Nobody was incentivized to develop self-sufficiency. "Because the government wants it" was not enough incentive.
Western sanctions have succeeded in achieving China's self-sufficiency goals where their own government failed.
Eh. Industrial policy in a big country is always the output of politics between stakeholders, anthropomorphizing it to a "guy with a plan", especially an "overlord" looking to "annihilate" is probably getting it wrong.
A friend of a friend is married to a member of the national academy of sciences in China. I heard at dinner a few years ago that he had been lobbying for more onshoring of chip production for years before the Nvidia restrictions, and always heard "you're right, maybe next year we'll be able to fund that". Then the restrictions came in and everything changed.
There are literal 5 year plans, eh?
And they are more about sounding good than about reality. They also affect a much smaller share of the economy than they used to -- which is very good! The original five year plans caused nothing but misery and starvation.
I’m pointing out it’s disenguous to say ‘it’s not like there was a plan’, when there was explicitly a plan with that title on it. Especially if the plan didn’t work out well.
It's disingenuous to characterize "don't anthropomorphize the political process" as saying no planning documents exist as part of that process.
And, to the point I was responding to originally, we can check those documents! They don't speak of "annihilating" other countries' industrial capacities.
>They don't speak of "annihilating" other countries' industrial capacities.
Why would they have to?
When China talked about their 5 and 10 year plans on solar panels there was no need to talk about putting the rest of the competitors out of business. That was a highly probable end goal of what they were doing, no need to state it.
It would be like Walmart stating "We're going to put out massive loss leaders in sporting goods to put X out of business". Instead they'll talk about growing market share and long term customer loyalty.
I'm sorry, I'm not following. Is the problem here that China is too capitalist or too communist?
Yes.
More of, China is smart enough to use the benefits of both capitalism and authoritarianism to grow their country while western governments are busy stealing the furniture and fixtures off the walls.
I agree with 100% of that comment but I'm struggling to understand the beef?
Of course they're trying to take care of themselves while the US implodes, how can you blame them for that? The whole world has a serious oil problem right now, because of US actions, and of course each country is dealing with it in their own way. Why is that a problem?
You do realize that a lot of people have had problems with America after WWII and using their economic and military power to dictate what the rest of the world does, right?
"Problem" is a relative term for those dealing with the consequences of it. It is one if you are America means a whole lot less economic freedom than you had in the past. If this is a good thing or not depends on your position.
If your beef is that they are succeeding or at least failing less while America steps on it's dick repeatedly, maybe this isn't their problem?
If you're fired up by nationalism but your nation is sucking, getting mad at the Chinese isn't the answer. And that's before we even get into relative levels of activism, Chinese political ideology is much more biased towards "neutral, just trade" than American politics.
Oh boy, how to get a bunch of seething haters more seething.
Near as I can tell, they fundamentally believe that the US actually can’t do better (despite protestations to the contrary), at least not with them there doing there thing, which is why they get so manipulative and shitty. They are unwilling to make other decisions than the ones leading to these outcomes.
Yes people, it is actually possible to do better. It requires making better choices, however.
False dichotomy. the problem is far deeper than that, and wanting to simplify the problem to those two extremes belies a belief that the world could and should be able to be simplified into two power dynamic. It is not. The world, and the powers that fight within in, are far more dynamic than a simple communist or capitalist spectrum, and attempts to simplify the world along those lines is only going to hurt you.
That just means your stakeholders argue twice: once when drafting the five year plan and then again, later, saying their policy position is more aligned with the spirit of the plan.
> [...] that was always China's plan, [...] they'll have the moral high ground
It's a possibility. It is also possible that they will steal yesterday's technology and still pretend to have moral high ground.
I do not wish China and Chinees wrong but most probably communism will work as it always do - by faking thing and bringing self-harm to their own citizens.
In free markets, everyone steals everything they can, and stops everyone else stealing from them as much as they can. This is normal and expected. The ideology says that it will even out and converge to a fair situation.
And by stealing you mean exchange? Or I somehow misunderstood what you consider a market to be - a place for pickpockets?
If you're paying money for something you could get for free, you're a sucker. Ordinarily the person who is selling it makes sure you can't get it for free.
Our LLMs were trained on the widespread theft of copyright content, with clearly defined and well tested law. But they made capitalists too much money so it became defacto legal.
Next you'll be telling me fairy tales like "laws are real".
> In free markets, everyone steals everything they can, and stops everyone else stealing from them as much as they can. This is normal and expected. The ideology says that it will even out and converge to a fair situation.
This is all nonsense. Capitalism isn't an ideology, it's just a slight formalisation of what naturally arises. People are free to choose; they can own things; they can make agreements; they must honour the agreements or pay a penalty. Capitalism in no way is about stealing.
https://www.britannica.com/money/capitalism
Sorry, I'm not sure where in that link you're getting that from. Do you have a quote from there?
It’s the same in communism/socialism except people are prevented from stopping everyone else from stealing their stuff, in the name of the ‘greater good’, and if they try to get more than everyone else get guilt tripped into giving it up under the ‘to everyone what they need, from everyone what they can’.
Which incentivizes pretending you need a ton, and can give nothing. You think being useful at work in Capitalism is bad? Wait until you are accidentally useful in a communist/socialist economy.
There is usually less pretending under capitalism, but people are people, so it all the same stories still play out.
People are shitty, if given the chance and influenced in specific ways. Capitalism or not. We’re dealing with the current bullshit in the west because somehow people bought the line that everyone is a good person - and we totally can’t actually see or do anything if we see someone doing something bad! - and we need to always follow ‘the rules’ even when someone bad clearly isn’t.
Until we find our balls, good luck.
communism derangement syndrome - rent free (ironically)
Ask anyone who lived in the Soviet Union how that was like
[dead]
Oef, still kicking the “communism” horse. It’s dead Jim, especially in China it’s dead and buried
Was it also the Chinese plan all along to annihilate their own birh rate?
Was the Great Leap Forward part of the plan too?
> Was it also the Chinese plan all along to annihilate their own birh rate?
No, that's generally what happens when countries stop being poor. Taiwan, Singapore, Korea, all have similar fertility rates.
But none of them had a government-mandated one-child policy (maximum, not minimum) for over three decades. That's a pretty major difference between China and neighboring countries.
And yet all of them have an even lower fertility rate than China.
On the one hand, you're proving my point. On the other hand, if you think that China's headed for a demographic disaster, so are all of those countries, and your thesis of blaming the CCP for everything... Falls a bit flat.
China has extremely low fertitily rate given that they haven't yet reached the status of a developed country (which Taiwan, SK and Singapore have reached).
Huh?
Developed countries tend to have lower fertility rates (i.e. less kids per family) than undeveloped ones. Comfortably off, well educated people tend to put thought into whether to have kids or not. In poor countries people tend to have large families so they can send the kids out to work.
Having a high fertility rate isn't "high status" and a sign of a country's progress, but rather the exact opposite.
Of course this is separate from this issue of having a shrinking population, which is a sign that your citizens are not optimistic about the future, not a positive thing.
Still, a stupid governmental policy cost them at least 100 million people compared to the expected spontaneous development. Probably more. That is the sort of punishment that omnipotent governments sometimes visit upon their folk.
I don't believe that even the CCP itself, in hindsight, considers the one-child policy to have been a good decision. At the very least it should have been ended some 15 years earlier.
Well, I've been reading on The Economist how China is seriously struggling with youth unemployment right now. Lots of fresh university graduates are sitting unemployed, or doing manual labor. In my home country (Finland) we have had way below replacement birth rates for a long while now, yet youth unemployment is at all time high.
The problem with low birth rates is assumed to be that there are less workers to take care of the pensioners. But right now it seems like even the few young workers available often aren't needed anymore, thanks to increasing automation of industries. I predict youth unemployment will be made even worse by AI soon enough. So, why do we need children, if capitalist economies don't need workers? Unemployed people do not pay much taxes to sustain the non-working age population.
I believe low birth rates might become a blessing soon enough. It just means less people for governments to take care of when robots and AI take away most jobs.
No, a remarkably farsighted policy that prevented PRC from having to deal with additional 200m-400m subsistent farmers. If anything it wasn't done sooner/aggressive enough. Like you can literally wipe away 200-400m from PRC button quantile and PRC would be net better off due to increase per capita resources and wealth. This hindsight will be historically obvious in 20-30 years when you see the outcome of India with 1.6B people with 1.2B poor, vs PRC with 1.4B people with 400m poor. The drag of having disproportionate, bluntly, worthless bodies is not linear for governance, and it's not because India hasn't tried aggressive family planning of their own, they simply failed unlike PRC. Who very much regards family planning as largely successful demographic management system that enabled proper allocation of resources away from generating peasants to building industry and the largest pool of high end human talent in recorded history, where frankly any tertiary dividend is worth 50 farmers in terms of productivity.
China is headed for a demographic disaster, though. High schools are incredibly packed, middle schools, not so much and primary schools and kindergartens are kind of empty.
Everyone is, not just China.
Everyone stupid enough to let that happen is.
There is a handful of developed countries where this is not an issue.
It's hard to say that it's stupidity (at least not country-specific stupidity) - it seems that globally, presumably due to the state of the world along many axis, many people are choosing to have fewer children, and it's happening on a large enough scale to have major consequences for things like the size of your tax base, trying to keep your economy growing, demographic concerns over an aging population with no-one to look after or pay for them etc, etc.
Different countries are handling this in different ways, with encouraging immigration being the major one. Japan is facing a major lack of workers, with many businesses having to shut down as a result, but have decided to tackle it in a different way relying on advanced robotics as an alternative to immigration.
There are very few developed countries NOT having this problem. Most of the population growth is in poorer parts of the world like Africa.
Well, if there are >0 developed countries that have solved this (Israel, the GCC, dare I say fairly well-off MENA countries like Jordan, Algeria, Morocco, etc., and to a minor degree Botswana) then this proves wrong the hypothesis that modern hyper-development must inherently mean population collapse.
Going purely by the evolution theory and natural selection, having so few children that there is a serious risk a population will disappear in a century or two, means losing. Needless to say, this is stupid. From the entire American continent, to Europe, India more recently, Thailand, China, the Koreas, Japan.
Africans, Arabs, and some non-Arab Muslim populations will carry the torch. Much respect for not falling into the trap.
Algeria and Morocco hardly seem "fairly well-off" countries, considering how many of their citizens prefer moving to the low-birth rate European countries you loathe. High birth rate of Israel is also unsustainable from economic perspective, as it relies mostly on orthodox Jews who on average are much poorer, less educated, and contribute less to the Israeli economy.
I'm not saying fast fall in birth rates isn't a problem, it obviously is. However, nearly all countries that don't have this problem offer much worse quality of life than those which do.
Young southern europeans are in turn migrating north. Are Spain, Italy, Portugal not well off?
There is a saying. More is better. Morocco has issues with tourists and diaspora pricing out locals from houses to food even, but Algeria is an honestly good country all around. With its issues of course. But there is more to life than earning $$$.
Westerners forget the tiny detail that these countries are extremely young unlike them, hence why it feels like there are so "many" people.
Edit: oops, I glossed over the part about Israel. Trust me, it's still a billion times better than nothing, which is the case for the entire West.
Edit 2: Oh my oh my, I missed this too. I "loathe" European countries? Do I? I mildly not respect European populations for wiping themselves out. I cannot loathe land.
[dead]
> this proves wrong the hypothesis that modern hyper-development must inherently mean population collapse
I don't know the exact correlation but even in Morocco, total fertility rate is declining very rapidly, down from 6 children in 1980s to now around 1.7 in urban areas. UAE (Dubai) has similar TFR to Belgium or Portugal.
> then this proves wrong the hypothesis that modern hyper-development must inherently mean population collapse.
Honestly you're reading the data completely wrong. You cannot look at it at the country level. In Israel for example the populations that are producing the most population are highly religious and less educated than those that are producing less population.
It's more about memes. Religion is a meme driven breeding cult. This is why said religions tend to have very long term staying power. The particular problem here is these religions come at a cost of having to ignore a lot of reality to get to that point. Because they are faith based anything that presents a direct challenge to that faith must be setup as an enemy or it may risk the religion. Yes, religions can incorporate new information but it tends to be a slow process as it needs to involve slowly changing peoples mind to avoid instability.
What this looks like in the long term is large populations of highly religious that have a fundamental disconnect from the technology that's keeping them alive. A new Late Bronze Age collapse.
So? How does the population being religious disprove the fact that they are developed and having children? Not to mention that I have listed around ~10 countries after Israel.
Uh... Meme-based? This is such an unserious argument I don't even know what I can possibly say to that.
How are they disconnected with the technology keeping them alive? All people, in GCC countries especially, are working to make 1st world development happen.
Implying that religious Jews or Muslims are stuck in the Bronze Age is ignoring reality. Most European development itself came from men of science who were Christian, I thought this apparent conflict of ideas was something taught in schools.
First, you seems to think a meme is something unserious. Read Richard Dawkins The Selfish Gene so you know what meme actually stands for rather than the thing you see on Reddit.
GCC countries are resource economies mostly. They must weave a very narrow path to success to non-oil based means of income or the entire region collapses. I will say they do realize this is a problem and are using oil money to diversify, but resource curses are very hard to break and collapse can happen very quickly.
"My grandfather rode a camel, my father rode a camel, I drive a Mercedes, my son drives a Land Rover, his son will drive a Land Rover, but his son's son will ride a camel." --Sheikh Rashid bin Saeed Al Maktoum
>Most European development itself came from men of science who were Christian
This is just stating that that in 1700 you were a Christian or you got hung (hell, pre-independence America was founded by people that thought that too few people were being hung). As we see when you take out the hanging or social shame part of the religion people tend to flee them as they are oppressive structures. You could say them being men of science either had no bearing to their religion, or was in spite of their religion.
I am saying, for example, that the Abrahamic religions are bronze age (ok, sorry, Iron age) social structures. I don't think studied scholars are going to disagree on that point very much. People saw how people worked and could be sheppereded in societies without formal education and turned it into a series of stories that were effective in modifying peoples behaviors. Kings later came along and added their stories so they could further control the population. Over time a semi-coherent story about how people worked was formed below a set of rather incoherent stories regarding things that didn't happen that people are supposed to accept with faith.
Notably, it is hilarious how ‘modern western society’ seems to believe that people somehow work differently than they always have, while clearly mimicking the same rough structures they are deriding.
(Comic book superheros, cough cough. The references to Canon in what are essentially liturgical disagreements - I mean power scaling arguments - are particularly hilarious.)
Not me of course, I’m different and unique. Just like everyone else.
Don't worry, we've got Elon Musk trying to single-handedly repopulate America.
And if that fails, we've got "Idiocracy" to save the human species as the dumb-asses pump out babies as fast as you like.
Population collapse? Who, besides Zimbabwe?
Yes, they literally named it the ‘one child policy’. It was just overly successful.
The official one child policy didn't actually have much of an impact. Birth rates had already fallen a lot, despite the official "lots-a-children" policy.
The one child policy is gone now but birth rates haven't really moved upwards.
It's debatable and impossible to disentangle from living standards going up, but a lot of the economy is built on assumptions of one child, from apartment sizes to personal transit (ebike can do 2 adults + 1 small child).
Not what I heard from friends who were subjected to it, and their parents.
It was insanely disruptive.
Of course it was devasting for some families. That is never the same thing as saying that it affected China's birth rate much (which it didn't).
There were forced abortions (sometimes very late), there was forced contraception, forced sterilization (I think), and lots of punishments for people who broke the policy.
There were also others who got around it fairly easily by paying a bribe and by not registering their "surplus" child/children (especially if they were girls). That of course caused some problems down the road.
That still doesn't mean that it affected China's birth rate much. It was already low (and had fallen steadily) when the policy was instituted. It also had plenty of exceptions for rural China and ethnic minorities.
Uh huh [https://en.wikipedia.org/wiki/One-child_policy]
Of course it had an impact, if nothing else, a cultural one. It basically eradicated the concept of large families.
So China has similar family sizes as the US that had no one child policy.
It's difficult to near impossible here to determine which counterfactuals would actually hold true. Based on other world wide trends it's very likely that Chinese family sizes now would only be slightly higher than if the policy didn't exist. In the modern world there are just a huge number of different factors working together to decrease family sizes.
What the West is doing is forcing China into a situation which is impossible to sustain when their economic dividend runs out. Especially considering these investments are fuelled by an extremely high level of debt.
And it’s not a given that they’ll catch up. I think it’s fair to say that their development of jet engines is on track to be obsolete before they’re competitive.
They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.
> their economic dividend runs out
i dont think it will - they still have a chokehold on battery production (which implies dominance in the EV sector), and they also control the vast majority of all rare earths. The chinese gov't can subsidize chip research/development with these other sectors that are dominating.
The west is the one that need to watch out for their own economic dividend running out from the past century.
"Economic dividend" really means "Demographic dividend". It's when your fertility rate decreases but your total population isn't decreasing yet, so you get more people to work more as they spend less time as parents. This of course is a self perpetuating cycle and ends in demographic collapse. China's population is now decreasing. The West's population is still growing. China is in the most literal sense dying. It's what you get for having a one child policy for 50 years. But hey, for a beautiful moment in time they really created a lot of shareholder value.
another thing: chinese gov can just some of its wipe its populace easily (the gov controls media 100%)
but... not the western govs
What makes you think they don't have a domestic market?
The Chinese domestic EV market is the largest in the world.
Chinese factories are full of Chinese robotics.
Companies like Huawei are struggling to meet the domestic demand for AI accelerators, while growing volume very fast year over year.
The Chinese domestic AI market is massive, but we just don't hear much about it outside of the models that are also sold here. Can you even name the models that ByteDance (China's Meta) makes?
As far as exports go, China is a low cost producer for many things. If the US doesn't want to buy high quality EV's costing 1/2 the price of a Tesla, then there are plenty of other countries happy to do so. The US is only 5% of the global population - there are lots of other consumers out there.
> They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.
Is this really true? Can't they just keep exporting "forever"? What requires them to find a domestic audience? I can think of some other countries that have done quite well mostly without finding a domestic audience (eg, Switzerland comes to mind, and frankly Europe might largely be in the same bucket).
https://www.imf.org/en/publications/fandd/issues/2023/06/sup...
Totally non-biased of course.
Nothing exists in the world that is not biased. The key to knowledge is understanding what those biases are.
Europe (by which I assume you mean EU or possibly Euro area) has a fairly balanced balance of trade overall.
And a teeny-tiny country like Switzerland can't be usefully compared to something as big as China - the rest of the world can easily consume whatever the Swiss produce, but there are limits to how much Chinese can produce until consumption will have to shift to domestic consumption because the world doesn't have infinite ability to pay for its demand.
That said don't expect any big change anytime soon - there are very strong political incentives to keep the status quo with China focusing a bit too much on exports.
It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
They expect to retire with a decent amount of money before things go down hill.
> It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
As someone who is myself 'old af' by tech standards (I'll be 53 very soon) the main problem is that the policymakers are all 70+ years old.
Their concept of what China even is is forever stuck in pre-2004 thinking from back when they still had significant neuroplasticity.
Vote these fucking fossils out, kids, before it is entirely too late.
(And, yeah, before someone inevitably brings it up, some exceptions-that-prove-the-rule older policymakers/politicians do avoid this trap of getting stuck in the past, but most don't)
They probably weren't expecting AI foundation models / model research to be quite as fungible as they are
Yes.
Over the last 20 years the economy has become dysfunctional. It no longer really resembles a free market; monopolies have established barriers to entry everywhere. And the biggest investors are awash with helicopter money that's been doled out for favors by the political class.
So those investors have tons of cash to burn and surprisingly few opportunities. Even within Silicon Valley/VC there are surprisingly few who seem to really understand the fundamental economics of software. Or perhaps those economics just aren't that important when you have billions of dollars on hand and cash is obviously not going to get you a return. Any whisper of possible exponential growth is worth throwing money at. Crypto? Why not. AI? Why not. Datacenters? Why not. Tulips? Why not.
This is by all definitions an empire in decline. Everything is broken or fake. Everyone is afraid to do what needs to be done. Power forbids it. So we're all just waiting for the other shoe to drop. Our secret police aren't as bad as the late stage USSR's yet, but hold Uncle Sam's beer...
(That's not a recommendation to try and time the nadir, by the way, as it could easily be 50 years away.)
I feel like my point is being over extrapolated a bit. All I was saying is that people in 2020 believed AI companies had much more of a moat then they actually do - that other companies would struggle to obtain enough data (y’know, downloading the internet to train models wasn’t legal back then) or resources to compete
It started way before the last 20 years, the past 40 years have created the conditions for this to happen.
Overfinancialisation of the economy while removing safeguards to keep markets functional, like anti-trust enforcement, are the main forces behind this erosion. Through finance the focus of companies is completely shifted away from producing good products and services, and into how to extract more paper wealth from existing structures to the detriment of products.
Not enforcing anti-trust and letting behemoths to form which cannot be competed against since with their amount of capital they can either buy their competition outright or just price dump for long enough to make competition non-viable.
It's the failure of neoliberalism, and that agenda has been pushed into Western countries since the 1980s-1990s, it made enormous wealth for the few at the top while eroding whole societies, economically and socially, it's all dysfunctional.
I feel like my point is being over extrapolated a bit. All I was saying is that people in 2020 believed AI companies had much more of a moat then they actually do - that other companies would struggle to obtain enough data (y’know, downloading the internet to train models wasn’t legal back then) or resources to compete
I replied to the extrapolation, not to your comment directly, I think your point might be more fitting under that comment than mine, I just drilled down from the points above mine.
Fair
Datacenters and AI obviously work and come with a credible value proposition and path to profitability.
Comparing this to crypto or turnips makes me doubt the merit of any other opinions your comment offers.
The initial growth of the internet was a boom bust cycle, and as you see the internet is still here. This is more about the economic paradigm that is being used to grow a technology versus the economic paradigm the technologies rate of growth can support over time.
Just to give some rough numbers in the past 5 years the amount of GPU compute installed in FLOPS is somewhere over 5 times all the CPU flops that have ever existed.
This has nothing to do with AI being good or bad or being able to produce things and economic value. It is by far the largest and fastest growth of any technology ever and we have zero clue about the economic stability of this grand experiment we're performing.
The internet is a much more appropriate comparison than crypto or tulips, which bring no or negligible value.
Some differences to the market at that time seems to be that during the dotcom bubble, many of the companies had little to no revenue.
Leading AI labs already generate enormous revenue. The investments into capacity are needed to address the current demand.
The situation seems somewhat less speculative.
That said, I cannot predict how AI capabilities will develop and how demand will respond.
Should capabilities plateau hard and soon, maybe the demand will not be there for the compute investments.
If it does not, and instead AI applications in robotics, science, and self driving expand, chances seem reasonably good that the demand will be there, no?
And as for the economic stability, much of the investment comes from existing giants like Microsoft, Alphabet, Amazon, and Meta, who have the necessary cash flow.
These companies are less likely to collapse than some of the ones during the dotcom bubble.
Eh, I'd say it's closer to something like internet + tulips.
It's the total amount of money in the economy that's been invested toward a potential outcome. AI represents the largest amount of money, and largest fractional part of the economy invested ever.
Because of this AI could be the biggest economic boon ever, yet still not recover the full amount invested. This will have deep economic impacts that affect everyone and everything. At this point AI must achieve all its stated economic impacts or there will still be a huge economic crash that kills off any company that is over invested and cannot make a profit.
Worse, the many of the perceived economic impacts of AI are not for humans like you or I, but the huge companies you listed. Even if they economically win, everyone else made out of meat could still lose.
They say history doesn't repeat, but it does rhyme. This, at least to me, sounds like a mixtape of "internet" + "tulips" + "1920s financial world leading to global political instability".
Every potential outcome I see occurring pushes us closer to further instability, even if the economics on it work out on paper.
> At this point AI must achieve all its stated economic impacts or there will still be a huge economic crash
I had a look at the numbers.
The investment into data centers is estimated around $800B/year currently.
OpenAI + Anthropic combined had ARR of >$100B in July.
Global labor income is around $65T/year, the US GDP $32T, the global one $126T.
Global software spending: $1.4T/year.
I am no financial analyst, but I don't think all the stated AI impacts have to be met just to recover the investments.
A 1% productivity gain on global labor income represents $660B/year. At 5% we would look at $3.3T/year.
They don't need to cure cancer to justify the investment, even though Amodei hopes to cure most major disease in the next 5-10 years.
Lets take this number for example
>Global software spending: $1.4T/year.
Out of this number how much of this goes to payroll that goes to humans versus how much of this goes to non-human infrastructure costs. The numbers suggest this is anywhere from 60 to 90%, with 70% being a reasonable average figure. Just under a trillion dollars paid directly to humans would disappear if AI somehow captured all of this market.
Moreso the AI industry needs to capture these gains very fast or they will be crushed by interest payments on the massive debts they've accrued.
>ven though Amodei hopes to cure most major disease in the next 5-10 years
The average development time for medicine is 10 to 15 years before a single dollar is earned from said medicine. AI will shave very little off that as human testing and our stupidly complex bodies introduce all kinds of problems. Furthermore running head long into blindly using AI medicine is how you produce X risks from super intelligence AI.
The speed at which AI has to develop in which to get profitability is the biggest risk, a potentially catastrophic risk at that.
You can theorize all you like, and many people like yourself are expecting to see GDP growth pick up as a result of AI, maybe if only because of the boom in datacenter construction etc (regardless of whether AI ends up actually boosting the economy), BUT ...
The reality is that so far there has been no sign of GDP growth picking up. It is basically flat at 2.5% +/- over the last few years.
In a similar vein one might have expected that the internet (think of all the e-commerce and efficiencies!) might have shifted GDP growth into a higher gear, but it did not, although in that case there was at least a significant boost in the 1996-2000 "dot com" era when the build out was happening (then to be followed by the crash and all the unused dark fiber etc).
So, maybe the hoped-for AI boom will be just as much of a dud (as it appears to be so far) as the internet boom. New day, different tools, same growth.
It's perhaps odd that we're not even seeing datacenter/etc build out register on GDP, but perhaps the scale of it is not as large as the internet build out?
It's not really odd when you look at the numbers in my comment, where the datacenter investments are 0.8T vs 126T GDP.
That's 0.6%. Subtract last year's investment and then we're talking yearly growth.
Turns out it's not the earth shattering capital investment the comment before mine was making it out to be.
And about productivity gains, AI became good at software development in spring this year.
I'm not sure what productivity miracles reflecting in GDP growth you already expect to see by now.
>but perhaps the scale of it is not as large as the internet build out?
In the past 5 years 5 times as much GPU compute capacity has been installed as total CPU capacity that has ever existed. When it comes to capital intensity the growth of AI has been one of, if not the largest capital expenditure for a class of project in this short of time frame. Things like the Manhattan project which was absolutely monstrous expense at the time is dwarfed by this.
Of course measuring GDP growth is difficult when you have an idiot in chief trying to destroy the global economy with the dumbest set of actions contrived by a human ever. Disentangling this factors will take a lot of work on someones part.
they expect the Chinese to bend their knee and beg for the sweet, sweet chip
they were sold an idea the AGI is right around the corner so even a slight delay might be critical to secure the "win"
if you sell them ASML equipment your market gets destroyed tomorrow instead of 10 years later so what options do they have?
No longer strangling their own economy would be a good start. Take a look at the magnificance that was the ESRS. A 300 A4 pages long list of yearly reporting duties. And the best thing is: Just because they're no longer nicely summarized in one place, doesn't mean the underlying duties went away.
Well Soviet Union tried to be self-sufficient, but the truth is such strategy loses to global free trade. Whether we want it or not, global trade is what keeps the world from WW3 for now, not the nuclear MAD.
You understand that the US has alienated almost all of its trading partners and is aggressively attempting to kill free global trade, right?
Whereas China is developing a powerful network of rapidly developing trading partners like Africa and the Middle East?
Yep, exactly. US is in the active process of shooting itself in the foot. My point is that China is very dependent on resources imports, so there's no way for them (and any other country) to be self-sufficient or they will stagnate.
The Soviet Union never was the manufacturing power behind a very significant fraction of the worlds products.
And China is very dependent on imports of almost everything. Maybe some rare elements, but e.g. oil is a big part of it (they agressively diversify off it, but it's still a large piece). Global interdependency is a good thing.
Few countries rely more economically on global trade than China though.
Sanctions can absolutely create the incentive to build a domestic supply chain, but incentive doesn't automatically translate into catching up on the same timeline
They always had the option to do that
But not the necessity, so it wasn't getting funded with the same urgency as now.
But if it was always an option and they didnt do it then they didnt think it was in their best interest
people have been claiming Nvidia will get competition too for years but you guys seem to be forgetting that the current leaders are not asleep at the wheel in the meantime.
When China eventually catches up on conventional lithography, the goalposts will shift to silicon photonics and optical logic, a transition DARPA laid the publicly visible groundwork for decades ago through programs like POEM and PIPES while keeping the mature implementations strictly classified until needed.
[dead]
I think eventually The west will want to kneecap China, so I guess this is the best time to do so.
Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.
Not quite, it's got quite a bit more complicated with agentic use cases.
Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result.
Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound.
Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
This is why I was careful to specify tokens per second not time to first token which is the prefill step you’re talking about. Clearly to run these models well you need both but as I said adding more chips or compute units can give you more latency where as overall memory bandwidth (throughput) is limited by access to fast memory.
I can easily use close to 100 million input tokens a day. A few million output tokens but at the end of the day maybe a thousand or so lines of code get written.
According to SemiAnalysis, both inference and post-training (RLVR) is mostly memory bandwidth bound. Only pre-training is compute bound, but it now only takes a small share of overall data center capacity.
https://x.com/EugeneNg/status/2099315982959616369
[dead]
The "abysmal" yield using DUV is an overblown statement by western commentators who don't look closer at the development. Certainly yield is worse than with EUV, but after multiple iterations of development, yield has become pretty good, within economically acceptable bounds, still making scaling possible. Volume is still ramping up. A lot of capacity will come online in 2027. The removal of western middlemen such as Mediatek actually improved the economics. Further yield improvements are still coming. Volume is already so high that domestically made phones and NPUs are making a real impact, yet are also selling like hot cakes.
It depends on what you are trying to make, but there are workarounds for most things, and it becomes more of a cost and volume issue than a showstopper.
If your yield is low and you still want the volume, then run more wafers, but then you need access to more DUV machines and more wafers. If your defect density is too high, then design smaller chips with a higher chance of avoiding defects, which is what Hauwei are doing - split processor into multiple chiplets (NVidia do this too, to raise yields and reduce cost).
HBM3E usually has 5-6000 vias, but doing this with multi-pattern DUV without defects is tough, so CXMT currently drop that to 3000, at a cost of some loss of thermal and voltage stability.
HBM:processor production ratios aren't what Huawei would like, so they mitigate this at system level by putting an optical memory bus on the GPU chip and sharing memory across the system.
Not everything is a huge LLM - smaller models like recommendation systems don't have so many parameters and need so much memory, so why waste HBM on them? ByteDance use their SeedChip acelerator for this, currently made by TSMC (using an older sanctions-approved 28mm process), which instead etches dense RRAM on die beside the processor.
There is also a time dynamic to this, with China still using pre-sanctions equipment and chips as their domestic alternatives ramp up to replace them. One interesting part of this is HBM ... HBM is very demanding to make, and everyone had problems with it, with initially only SK Hynix being successful. Samsung and Micron took a year or so to catch up, and during this time Samsung had made a ton of HBM that didn't meet NVidia's specifications, so ended up, pre-sanctions, selling it to China, where it has acted as a stockpile to carry them over as CXMT's domestic capacity ramps up.
China seems to be doing fine. Of course they would like sanctions lifted, moreso for HBM than anything else, but all that sanctions have really achieved is accelerating their semiconductor independence. They are still building 1T+ SOTA models, standing up 100K clusters of domestic AI accelerators, etc, etc.
China should invest in an analog inference chip. It's a hail mary but why not.
The two challenges there are firstly - that analog design has been a separate electrical engineering school for most of a century, so there are few who could design it - and secondly - that every single chip will have subtle variations in its computations, necessitating some sort of model finetuning per chip. Possibly the chip could be characterised at the factory, and ship with the characterisation data burned into a controller rom or something, but if that doesn’t pan out the whole thing is likely a non-starter.
If it could be made to work, you could run a fable-grade model in tens of watts.
Or we treat them like humans and make each one do job interviews to work out what tasks they’re suited for.
This made me smile. A true test of human-like intelligence. Maybe they could take some entry level jobs (“I can make a nice Wordpress site”) and build a CV?
It's the hidden sociopath that's able to mirror a functional entity that I'd be a bit worried about in these cases. Damn thing would make CEO quick.
awesome explanation. never thought about the complexity of doing analog and having precision issues because it's not discrete anymore and repeatable
They can probably afford to do both.
I'd assume that they have - but will keep mum 'till they have a major breakthrough or large-scale operational deployment to announce.
That would be like skipping land line phones for mobile...
Developing countries have done exactly that in many cases.
And skipped DSL for fiber, and skipped credit cards for banking apps.
Or on DRAM on die compute
Huawei uses their own non-standard HBM called HiZQ probably not produced by CXMT.
China has had a very strong incentive to recreate lithography and fab equipment for years, but a lot of the specialized HBM packaging chain was probably much lower priority until relatively recently
This competition will be great for us consumers for a while. Until China undercuts everybody else for long enough to essentially monopolize the market. This pattern has repeated itself across many markets already.
I read HBM yields are 25-30% (vs 80-90%) making them 3 to 5 times as expensive. They are 4-5 years behind, that probably means 1-2 in Chinese time.
Does that include yield from packaging?
The 3 to 5 times more expensive does.
What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
It's not that there's a blocker. It's that it takes roughly 3x the manufacturing capacity to produce an HBM package at the same storage capacity as DRAM. We are sacrificing total bytes for bandwidth.
I don't fully understand the source of the "total bytes" constraint, but a major factor may be because HBM4 / HBM4E can only make use of the footprint directly above the processor/logic die (or in direct vicinity of its interconnect), while traditional DRAM can be placed further away where there's lots of real estate on the motherboard.
I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?
These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per square millimeter of motherboard, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
Thanks for that, just went down an interesting rabbit hole. Many of us were hoping this re-tooling would eventually trickle some fast RAM down to DRAM-exhausted PCs, but given it would require a rearchitecture of the motherboard it's unlikely.
HBM4 has over 2048 signals to the processor’s PHY with tight signal integrity requirements that require the HBM stack to be < 0.5 mm from the processor die. That’s why HBM integration is done with interposers (soldered on the package). So, it’d be the CPU package that integrates it. Motherboard is too far away.
Kind of seems like we should be making chips with both. Big HBM stack on top as a sort of huge L5 cache like thing. And then a bunch of DRAM type sockets (like LPCAMM) around the exterior.
For chips with integrated CPU+GPU+NPU, it could be worth it tech-wise. The GPU and NPU can eat HBM bandwidth. For general purpose CPU code, the HBM would likely not be worth it. It's high bandwidth, but you trade latency, and general CPU code is branchy. Economics-wise, the HBM stacks alone will cost more than a consumer CPU (or APU).
[edit] The packaging needed to support HBM is also much more expensive too. If demand for current HBM applications tanks and the manufacturing lines need filled, then maybe. Currently, the price point would make it very very niche.
HBM also trades bandwidth for latency, and your regular computing is much more sensitive to latency than bandwidth.
How so? FEOL is pretty much the same, BEOL is almost the same save TSVs, the packaging tech is different and more advanced, but not exactly 1:1 comparable. Do TSVs really occupy 3x the area of DDR IO's?
See remark on the slide 11: https://www.servethehome.com/micron-evolving-memory-architec... That presentation is by Micron.
It doesn't really say that it needs to use 3x more area, but that 3x more gets consumed due to "advanced packaging and manfuacturing complexity". Which doesn't properly explain why it consumes 3x more and could simply mean they have a bad yield and 2/3 produced is garbage.
Classic DRAM stacks up to four wafers on top of each other and then is packaged with BGAs. The manufacturer can check the DRAM chips independently.
Soldering the DRAM onto a PCB is such a reliable process that there is almost zero risk of defects and even if a defect occurs the damage is limited. If the DRAM is soldered onto a DIMM the risk of a defect on the non memory hardware is non-existent. If the DRAM is soldered straight onto an SBC or GPU, then the DRAM can be removed to save the precious SoC or GPU chips.
Meanwhile HBM is the ultimate nightmare scenario. You stack up to 16 DRAM wafers on top of each other. One defect and the whole stack is worthless and that was actually the easy part.
In stage two things get even worse. You now have your accelerator chip and you must place the HBM on that chip. E.g. Blackwell GB300 has eight HBM stacks and the accelerator chip has a bigger area than the HBM. You must get the packaging right eight times in a row or you have wasted not only the DRAM silicon, but also the accelerator silicon because HBM cannot be removed and defects are permanent.
The issue here isn't just the yield of the HBM (which is obviously lower if you have taller stacks) but rather the yield of the combined HBM-based product, which is why doesn't make sense to say it needs more area but it is completely correct to say that HBM leads to more silicon being consumed. Hence it doesn't make sense to talk about yield of the HBM itself, because it is always part of an integrated product.
You make HBM by stacking a whole bunch of dies on top of each other. The signals from the upper dies need to pass through vias in the lower dies to get out - taking up valuable die space in a way which simply isn't needed with regular DDR. Similarly, HBM has a far wider bus, so each individual die has, say, 16 banks of depth 32, rather than 4 banks of depth 128. That's more control area needed per byte of memory.
Those two combined already result in a huge reduction in bytes per mm2, so with the same wafer processing capacity you're producing far less byte of memory. Add to that a complicated chain of HBM-specific packaging steps, and you're now also losing a decent bunch of perfectly-fine dies because rather than putting it into DDR you tried making a HBM sandwich and screwed up.
Even if the memory cells are the same and have an absolutely identical yield, HBM will always end up having a significantly lower output. That's just the cost of stacking, but some people are willing to pay the per-gigabyte price penalty in return for the higher bandwidth.
> 2/3 produced is garbage.
This might not be far off the mark. You are irreversibly linking the fates of these devices after a certain stage of manufacturing. If something goes wrong at final packaging time, you lose all dies instead of one.
Yea, that's the question. Yield situation can improve. Area overhead would not improve short of a completely new and incompatible tech.
direct link: https://www.servethehome.com/micron-evolving-memory-architec...
Yuck. Time to build an xSPI/HyperRAM workstation (if only these had multi-bank chips).
Sadly the $ per byte of xSPI and HyperRAM quite high
Yes, but enough to run a text editor, or even a mechanical CAD. Not an LLM, but that's the point!
3x is a reasonable figure. They are not literally that large, though.
>> What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
HBM is meant to be integrated into the same package as the CPU, so no more DIMM sockets. It also has higher latency apparently.
One of the comments mentioned that they have a bus size of 2048, which may be why the latency is higher.
> These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5.
Why would you want/need to?
The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.
So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!
These are not normal requirements, other than for a GPU.
> Compare that to the 128 bit wide bus of a modern CPU.
Or 256-512 bits on medium to high end consumer CPUs if you're apple.
At least DDR6 is probably widening things 50%.
Right but if HBM memory is all that people want to produce, then building a CPU which can use it use it would be useful on it's own merits.
But in reality we also already have unified memory architecture systems, integrated graphics etc.
People want to produce hbm because it’s more expensive and more profitable than regular memory.
There is a world where scale and experience means it's about the same though, is the thing.
And memory is already expensive. It's downright hard to even get it though - you frequently would prefer not what's cheapest, but whatever is in largest scale production.
Scale and experience almost entirely share between HBM and normal memory. And they're both in large-enough scale production to not have a big difference on availability; if you're willing to pay HBM prices you should find even more sellers of DDR.
The only way I see HBM becoming competitive for consumer CPUs is if they solve the yield issues. Or if AI crashes so hard that people are putting those GPUs on fire sale and salvaging mass quantities of HBM off of them.
I was thinking this, but I doubt that the HBM is so easy to repurpose.
I’m assuming that it’s mounted in the same unit as the GPU, not in a handy-dandy socket.
It could be cracked open and extracted no doubt but if it’s glued in that’s going to be nigh on impossible to extract.
I had been pinning my hopes on HBM coming onto the second hand markets after there three years or so of use, perhaps I’m wrong.
It seems unlikely as the entire machines with HBM on them are apt to be sold whole on the market and snapped up quickly. With how much demand is in the current market machines 3 years old may not be sold if they can't get newer faster machines fast enough.
Intel and IBM have already done this almost this with their wide eDRAM caches.
The advantage in going wide is transferring cache lines rapidly, not the CPU bus interface.
SIMD (and especially the modern matrix extensions) can use as much bandwidth you can throw at it.
Without further clarification, that statement seems impossible. "As much" being unbounded and all. You should expand what you mean.
This will blow your mind, but it actually is pretty close to being unbounded. :)
Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth.
All this with a single instruction.
Nah his point is a bit simpler. Vector units can process a limited amount of data per time unit. Memory can load a limited amount of data per time unit.
If you have infinite memory bandwidth you just move the bottleneck back to compute so both have to grow simultaneously in lockstep.
What you should have said is that CPUs have so much compute headroom for matrix vector multiplication that simply adding more memory bandwidth would make them faster so every improvement in memory bandwidth is welcome.
Agreed.
The "move the bottleneck back to compute" bit is changing rapidly though. The first time a major hardware company ships a PIM chip, you can push for a few orders of magnitude more data through without being compute bound.
Surely something prevents it being 1 quadrillion bits per instruction? Since that’s well within “unbounded”.
Most likely the chip running at the core temperature of the sun.
We'll have to figure out how to read and right to the surface of a black hole to get speeds that high.
I think people might prefer the lower idle power consumption from lpddr over the better bandwidth in hbm in battery powered stuff. That said right now the price is definitely preventing us from finding out.
Idle yes, but HBM energy consumption / memory operations seems to be a bit better than DRAM.
Only if you’re running at 100%. Consumers generally do not.
That hasn't been true since early HBM2 days, before the controllers standardized on power/voltage management and did things like leave them in P0 to ship on time
LPDDR/DDR/GDDR generally still win over HBM when there’s no data being transferred. HBM is primarily better in watts/byte transferred. Consumer electronics spend most of their time idle.
Racing to idle is a very common power optimization technique
Right, but HBM idle is significantly worse than LPDDR idle or even DDR idle for that matter. That matters a lot precisely because the device is idle most of the time. Your idle power draw dominates.
Comparing HBM to LPDDR is pretty silly. That's like a bath tub vs a urinal
This is literally a thread where people are saying “but why laptops no HBM”. It’s a perfectly cromulent response that meets the question where it is. It might be mildly a more defensible question for normal plugged in desktops but that would require a massive ecosystem change since people who build those traditionally really like their DIMMs and less buying a CPU that has non-swappable RAM built in, not to mention there’s no sustainable market for the higher cost and low volume product.
No, it really isn't. You're barking up the wrong trees for incorrect reasons
If LPDDR6 PIM ever becomes a thing then the advantage of HBM will shrink. I'm not saying LPDDR6 PIM will make HBM obsolete in the datacenter or where it is currently shining, I'm saying that large volumes have their own charm and it is more likely for LPDDR6 PIM to be in your laptop or smartphone or SBC than HBM.
LPDDR6 PIM would primarily help the low end and mid range accelerator market. E.g. embedded models running on SBCs can be up to 1 GiB in size with acceptable performance, small models at 8 GiB become really easy to run at reasonable speeds on a smartphone and PIM enabled laptops or mini PCs make it possible to run 32 GiB models locally without compromise.
Of course this also assumes that the associated accelerators (NPUs) will catch up too, but the general point is that you won't need a 5090 or a 4090 anymore.
Mainly bus width. Afaik HBM is like 1024 bits vs DDRs 64 so you need lots of transfers in parallel to saturate the bus, and CPUs kinda want 64 bytes of data as that's the size of a cache line ASAP. So you need a ton of in flight transfers which isn't a thing CPUs provide, maybe multicore workloads.
Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
Even if the cost were the same of the RAM itself, you'd need much bigger and more expensive CPU to deal with it.
Running 17 chrome tabs doesn't benefit at all from that HBM and all the additional hardware+software complexities that come with it. You want a specialized coprocessor to handle specialized workloads. The GPU exists separately from the CPU for a reason.
In one sense, nothing, in another, everything. It is DRAM, but the bandwidth requirements mean it’s paired to a processor, i.e. no DIMMs. Not 100% sure but things like MacBooks and the Framework tower, where you have fixed RAM for the device lifetime, have ~0 tradeoff.
HBM isn't dramatically better in every dimension. You get huge bandwidth and good energy efficiency per bit transferred, but not necessarily a meaningful latency improvement and capacity expansion becomes tied to the package
HBM is a stack of DRAMs, so there is no “instead”.
The vias to enable stacking is a significant amount of the die area.
If they're talking about production capacity, that is some product of die area and process steps, right? It doesn't have to be 3x die area, just 3x lower factory throughput for the same number of functioning memory bits.
HBM is less dense at a water scale than DDR because of all the vias, but each memory but is as dense or denser.
You make HBM instead of DDR and the number of bits you're making goes down.
It's really that simple.
Nothing except CPU manufacturer choices. Mac laptops use it and they're consumer products.
People will have to get used to buying a fixed amount of RAM with their CPU but thats unlikely to be a problem.
Apple's "unified memory" marketing is so strong that even tech literate people seem to have this misconception!
They have managed to pull this sort of thing off many many times. https://en.wikipedia.org/wiki/Reality_distortion_field
Yup, the only reason Macs have higher memory bandwidth is because they use more memory channels, which gives them a wider bus. Both Intel and AMD only allow more than dual channel memory on server class processors these days.
The “Apple only does X better because they do Y” thing has been a meme for ages. I remember dismissals like “PowerPC is only faster at math because it has more integer units” or something along those lines, and thinking, uh, isn’t that a good thing?
There's a qualitative difference between "they're doing a different thing" and "they're doing the same thing, tuned differently". GP is saying that this is a case of "they just tuned it differently".
This distinction doesn't change what the performance numbers look like today, but it does inform what changes would be necessary for those numbers to look different tomorrow. E.g. Apple Silicon isn't fundamentally orders of magnitude more efficient than x86, they just used smaller features. Newer Intel and AMD chips made on equivalent processes _also_ get similar efficiency gains.
There are AMD and Intel devices on similar process (not talking about A20Pro or M6 which are set to ship later this week), and they do not get the same gains.
And honestly, they have historically had different markets.
When the design is for only one customer, you don't need to generalize things, and those things you generalize to give different customers different options has costs.
AMD will soon be a larger customer for TSMC than Apple (NVIDIA is already there) so Apple's pre-booking new processes is likely to be gone in the near future.
Depends on the expense trade off.
If I get 10% more performance for 50% more cost it really depends on one's needs, for example.
The point is that "unified architecture" is a buzzword unrelated to what actually makes things fast.
It does make some things faster because you can share memory between CPU, GPU, etc. without copying. But not everything.
"You're only better than me at sports because you practice more and try harder!"
I am just saying it's not magic and x86 is capable of doing the same. Quad channel memory used to be more common in consumer hardware but now it looks like you don't even have the option anymore for desktops. AMD's Strix Halo was the first sign to reversing that (it has quad channel memory!) and hopefully we see more of that in the future.
AMD does market segmentation and limits consumer chips to dual-channel. You need to cough up the dough and get Threadripper for quad-channel. Or even more for Threadripper Pro to get octa-channel.
As I mentioned above AMD's Strix Halo has quad channel memory and is consumer level. But yes, other than that everything is segmented away.
Afaik steam deck is quad channel, despite it being a pretty low end chip (with a decent GPU though)
Kind of but not really. DDR5 splits your typical 64-bit channel into two 32-bit subchannels meaning the bus width is not increased. These subchannels are not always advertised because it's a just a feature of DDR5. Actually adding more channels increases bus width, which is what meaningfully improves memory bandwidth.
Threadripper have that extra bandwidth and M5 still is faster
Can you link a specific benchmark?
Keep in mind that non-Pro threadripper is still only 256 bits wide and Pro is 512. And the memory is 30% slower than with an M5. So an M5 Ultra has 3x the memory bandwidth of the best threadripper.
Don't some AMD full server CPUs have 12 memory channels ?
Yes, some do. So similar bandwidth to a Max but with lots of slower cores. Half as much as an Ultra.
Except strix halo.
The memory bandwidth is a small thing compared to the massive win you get by not having to move data between two memory pools at all.
Depends on your workload. And AMD has supported unified memory long before Apple Silicon existed anyway.
Often you don't move memory around as a programmer, but that's exactly what happens in the background.
It's the address space that's unified, not always the physical hardware.
The data movement (when needed) is handled transparently in the background by page faults and other tricks.
Moving memory around is a bottleneck. Non-unified memory usually means copying over the PCIe bus which is way slower than RAM (and way way slower than VRAM). Actually unified memory means you don't need to copy anything at all though which is the absolute best case for performance.
Having unified memory is a real advantage though, it's not a reality distortion.
It isn't really, as long as you don't care about power consumption, physical constraints and money. Basically desktops <2025 (and hopefully >2027).
DDR is optimized for latency and stability at the cost of bandwidth whilst GDDR is optimized for bandwidth at the cost of latency and stability. GDDR is pushed so hard these days that a small percentage of errors is expected and corrected because this is still faster than running it slower but more accurate.
GDDR7 often has 10-20x the total bandwidth but 3x the latency of DDR5. Graphical workloads want as much bandwidth as possible but care relatively little for latency. Conversely, applications love low latency but don't really see any performance benefit from higher bandwidth.
So basicallyt you have workloads that are diametrically opposed and running unified memory forces you to compromise.
Is GDDR7 that's ECC then just the super well binned stuff (plus the ECC parts added) ? I've been wondering why we don't see more cards using it as they perform pretty damn well. Take the Nvidia 6000 Pro Blackwell for example. The compute is insanely fast assuming you can fit what you need in 96GB of ECC GDDR7
The price is that you essentially glue CPU and GPU together, which limits total compute, from a size and thermal perspective.
This is really not a limit because of unified memory -- in principle, PCIe GPUs could read/write main memory without the CPU. But it's a limit for /fast/ unified memory, because fast means close.
So unified memory is great as long as the integrated GPU is strong enough. Then it has two advantages: a) probably faster transfer CPU<->GPU (but that's an implementation choice for the non-unified case b) If you either need a lot of memory for the CPU or the GPU, but not for both at the same time, you pay for memory only once.
It is a real advantage.
The reality distortion is that people seem to believe it's HBM, or somehow it gives you extraordinary amounts of vram. Neither are really true.
Agreed. That's why it's a good thing all computers made in the past 15 years have unified memory, not just macs.
https://x.com/Lina_Hoshino/status/1820947147312820497
In theory perhaps but the benefit is not as relevant with a weak iGPU. In practice all PC's with performance ambitions had a dGPU until Strix Halo and Panther Lake.
Is that Marcan's vtuber persona (Asahi Lina) that changed name since Marcan doesn't work on Asahi Linux anymore?
No I just failed to check before I posted, I thought it was HBM. I don't have any interest in Apple hardware so wasn't properly informed, and have been duly crucified.
MacBooks use regular soldered lpddr5(x) RAM. Same RAM as every other laptop manufacturer, they just use more lanes to achieve a higher bandwidth.
Perhaps more noteworthy for general home and business computing, doesn’t it also allow for lower latency?
More channels or soldered memory? Channels are basically RAID 0 so it depends what you're measuring. Soldering memory down was the only way to use LPDDR5X so if you wanted the best memory you had to solder it down. LPCAMM2 exists though so newer devices can use that instead of soldering them down, but not all devices would be able to fit the required LPCAMM2 slots.
SOCAMM2 allows for removable ram in nearly 0 added space.
That's tiny! But it still depends a lot on form factor. LPDDR5X is used in phones and making memory removable would have its compromises. You may even have compromises in laptops. Look at how tiny a MacBook Air's mainboard is and you'll see the RAM modules on the same package as the SoC. SOCAMM2 is too large for that but a variant with only two modules could possibly work.
I bet it could fit. It just do a combo of soldered ram and non soldered like many laptops used to.
> Mac laptops use it and they're consumer products
No, Mac laptops use LPDDR, currently LPDDR5X.
The normal non-tech-savvy person already does this. They simply don't know that their ram is upgradeable or something else breaks first, before having to touch ram.
what is this too much RAM thing you mention? I thought the only valid RAM situation you could find yourself is not enough RAM. Too much? That's just fantasy
pretty sure its lpddr not hbm.
A shame that this should if anything, lead to consumer DRAM prices getting even worse.
Why would more RAM supply lead to higher consumer prices?
HBM4 and HBM4E DRAM are NOT the DDR4/5 that consumer markets need. Capacity allocation is leaning more to data center grade HBMs so less to produce dedicated DDR4/5 DRAMs. Supply demand will further drive up the consumer DRAM price! Note, the article mentions NO of new fabs is being constructed (all semiconductor manufacturers know that constructing more fabs means the boom/burst cycle will eventually kill them, so no one create more fabs) Perhaps the federal government need to step in here - the market doesn't fit the issue.
Plus it takes 3x the wafer capacity for hbm than the same byte capacity in dram, so we’ll likely see the consumer market be decimated here
The consumer market will not be decimated. Memory of all kinds will get very cheap quite soon because of supply and demand and substitution.
https://www.tweaktown.com/news/112105/amd-warns-ddr5-prices-...
HBM can be used for GPU though. I wish it was more common.
Aren’t new DRAM fabs being built in China?
No, completely wrong. Supply and demand will definitely fix this.
More HBM4 will drive the price of it down. That will make it less profitable to produce. That will mean firms shift some production back to more profitable "consumer" RAM. Increased supply will lead to lower prices.
A shortage is always followed by a glut.
What specifically did that say that you think is wrong?
Supply is currently moving the wrong way. Supply and demand for consumer DRAM will get even worse, even if it will "definitely fix this" in the long run.
And the fact that production is moving the wrong way suggests that in the short term supply and demand is actually doing more harm than good. HBM is so profitable it's radioactive.
I don't know where you got the idea in another comment that all memory will get "very cheap quite soon". They're not increasing total memory production. (much/yet)
China is about to massively compete. And doesn't Micron and someone else have new fabs in NY coming online ? I can't imagine they're going to exclusively make HBM.
Ram always goes through these boom and bust cycles. They're going to increase supply but not so much that they're eventually left with huge overproduction as usual.
Of course if Google researchers come up with more ram reduction tricks for newer AI architectures that could make a huge difference as well
I don't expect China to be a particularly large competitor in the next year or two affected by the decision in the article, but they'll help a lot in the long run.
Micron's NY fab had a press release two months ago that they started pouring the first concrete, so that's far from coming online.
If AI demand sticks around and buildouts stay cautious it could take much much longer than usual to build up enough capacity to push prices back down below $3/GB.
More efficient AI memory use risks even more demand for memory because you're now able to get even more performance per dollar; the benefit of extra memory has to be sufficiently far into diminishing returns for demand to drop overall.
> What specifically did that say that you think is wrong?
You said:
> the market doesn't fit the issue.
I presume you meant "fix" not "fit". You are wrong that the market will not fix the issue. There are literally millions of observed cases across many markets of increased demand (or decreased supply) leading to higher prices leading to increased supply leading to lower prices, which is why it's a fundamental law of economics.
The price of RAM will fall.
I feel like everyone here forgets that the DRAM industry is operated by one big cartel. The rules of supply and demand don't apply here. Now that prices are up, they will keep it up, cutting supply as needed in order to do so. Every big DRAM manufacturer in the industry has been charged with price fixing, among other things: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing
There are so few players in this space they can coordinate and collude pretty easily.
DRAM isn't nearly as hard to get into as compute. The price fixing stuff is going to bite them in the ass someday soon by massively increasing the incentives for new players to enter the market and compete.
> You said:
That wasn't me.
Also they're talking about the short to medium term. On that time scale they're not wrong. The market is so far from fixing it right now, it's going in the opposite direction. The market is chasing HBM money which is outbidding everything else.
> The price of RAM will fall.
And if that might take 5+ years, that's not much comfort.
> higher prices leading to increased supply leading to lower prices
So far higher prices are causing barely any supply increases. (Note that the article we're commenting on is about a supply decrease.) Companies are barely trying to expand, and even if they do start trying hard it'll take a very long time.
> That will mean firms shift some production back to more profitable "consumer" RAM. Increased supply will lead to lower prices.
Long-term, sure. But the article we are discussing is about next year, non-HBM supply going from 60% of their output to 20% (HBM 40%→80%).
> Long-term, sure.
Yes. Non-HBM Samsung supply going from 60% to 20% will cause prices to increase, which will cause increased production of non-HBM, and then prices will fall (below where they currently are).
Might lower the prices of Mac Minis, Studios etc though
No, those use LPDDR5(x), not HBM.
Depends on what proportion of Samsung's output is current HBM4 and HBM4E DRAM.
Everything in their statement can be true and it be a bad thing for consumers of non-HBM RAM.
"HBM capacity to expand to 250,000 wafers a month"
So let's say their current HBM capacity is 100k wafers/month (pure speculation/random number for illustration), and their total RAM capacity (including HBM( is 300k wafers/month, then non-HBM capacity reduces from 200k to 50k.
And it's worse than that, Micron said that converting capacity to HBM was at a 3:1 ratio, so that 250k HBM could be up to 750k in non-HBM. Fortunately, some of the increase is due to improved processes.
Keep in mind that total manufacturing capacity is increasing as well. Perhaps not enough to fully offset, but it's incorrect to assume that supply capacity is flat.
It'll take a lot of time.
Micron has been working on expanding in Boise since around 2023. They are predicting that the first new chips will start rolling out around 2027.
I don't think we'll see any relief until at least 2028 or even later. And even then, probably not without antitrust laws being enforced. Because you know memory manufacturers won't just simply drop prices, they have a long history of price fixing.
Question is what does the demand curve for regular ram look like also? If regular production is on a 10% increase curve but a 40% demand curve than we've been at the samw place.
OP article estimates HBM is going from 40% to 80% of their output.
Part of the implication is that factories that could be producing consumer-facing DRAM like DDR5 would be retooled to produce HBM instead, leading to even less total consumer RAM production.
This is already happening, by the way.
SK Hynix's new Cheongju M15x fab was originally intended to be for DDR, but mid-construction they shifted it to HBM. Similarly, the Icheon M10F packaging line has been retooled to HBM, allowing it to assist in processing the output of other fabs.
And this was news early last year, so who knows how far it has progressed since then.
The crash won't be kind to them & they deserve everything they will got for their past behavior.
Memory allocated for HBM is memory taken away from DDR production.
If Samsung is limited in the number of wafers they can process per month and they use more of those to produce HBM they necessarily have less of them left to make other products, like consumer DRAM.
Gamma squeezed by the AI business, that's why. The more you add the fuel, the further it will pump up.
Presumably DRAM production capacity gets reallocated to HBM, not sure.
Depends on how quickly they are sold.
Maybe this leads to device manufacturers using HBM memory instead.
Will consumers actually get to purchase it though? Or are they just doubling it for frontier AI labs?
I have read an interview with Acer's CEO claiming that the shortage is already over.
He alleged memory maker's public announcements of the shortage lasting throughout the end of 2027 are attempts to covertly signal the others on pricing strategy.
With additional production capacity ramping up this year and next, this seems somewhat plausible.
If manufacturer's inventories are filling since they advanced orders when prices were rising, there could be an oversupply of memory when they don't place new orders at the higher current price.
This could quickly result in a price war while capacity expansion and yield improvements create irreversible oversupply.
What does it matter?
All supply is supply, despite what certain folks seem to believe.
It matters so that average people have a chance to own their means of production, and that the current hardware-grab by entrenched capital doesn't get to charge such high rent.
Frontier AI labs are customers just like the average Joe down the street. The only difference is who can bid more. Average Joe probably doesn't have millions or wont buy millions of chips.
There are no privileged consumers that have "dedicated" amount of production, no matter the demand.
What point are you making by saying "they're just like the average Joe" followed immediately by saying they have so much more money that they can outbid the average Joe at any price point.
If someone can dump 100x the money to get something I want, they're not "just like me".
yeah this
if I'm ordering 1 RAM and demanding some special customization, even mom-and-pop stores will just ignore me
but if I'm buying 100000000000 RAM, things change
You won't ever own it as long as it comes from a company the size of Samsung.
I mean... You can buy a stick of RAM with Samsung chips on it and use it in your own machine to run a local model vs. paying for an Anthropic subscription. I assume that's what they're referring to, not owning Samsung stock.
Your average consumer is not even a citizen of the country where the company that makes this is based, or where the factories that produce it are.
You can own stock, but other than that?
"average people"'s means of production is their brains. which they have all the chances to own but often reject them.
As far as I know, the only place HBM is used in consumer products is GPUs?
HBM hasn't been used in a consumer GPU in some years. I think the RX 7900XT
It's all GDDR6X or 7X
tl;dr; This is existing production being redirected to HBM. Which means consumer market is going to get a lot worse.
And the worst thing is that is the best possible strategy for them. It's essentially win-win for everyone but consumers.
- Buyer (AI) has crazy money, so will pay whatever
- Seller doesn't have to build anything new, as the buyer is willing to pay whatever
- Generate ridiculous profits from crazy money
- No oversupply risk in case of reversal
- Return from producing HBM to DDR5 in a single quarter if reversal does happen.
Hold on to your existing hardware, people, and be on lookout for your local deals.
The solution would seem to be preventing that "Buyer (AI) has crazy money". They already have borrowed billions and are only generating losses so the money should run out. It just needs to be made sure that nobody (taxpayers) will rescue them.
> just needs to be made sure that nobody (taxpayers) will rescue them.
oof.
This should be the top comment.
It absolutely is not a great situation at all. This won't make things better.
Will that be enough for AI's hunger?
During the Covid global chip shortage Intel announced new factories to address the production bottleneck, before the factories even started to be constructed, the shortage was long over and the projects were eventually canceled.
It will be just in time for when AI will run very well on consumer hardware and the need for data centers will collapse.
There will be demand for both
Some demand. The question is if the capacity will meet it, be under or be over.
What if the AI companies start selling the hardware with their models?
I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.
Sound familiar?
More likely they would sell a chip with the model etched in silicon for the dramatically higher token/s.
A chip that can only be accessed through their software. With a monthly subscription fee, and other enshittification surprises.
Most likely those models would be extracted from the hardware in no time.
I suspect no, because most of the tokens would be used for its internal reasoning loop, and a cheaper model could be used to obfuscate the output.
That's never going to happen. By the time you can run current frontier models on your $10k desktop the frontier will have massively advanced and people will want those models instead.
I’m not so sure about that. Already AI vendors are back to cutting prices to try and keep customers from cutting back on their usage. My own employer is working hard at pivoting to much smaller fine-tuned models for established use cases, and seeing model performance improvement in addition to large inference cost reductions. Being able to run them locally hasn’t exactly been a disaster for devex, either.
It may turn out that demand for SOTA frontier models isn’t so limitless after all.
Every product follows demand curves. At a price of 0 you could find infinite usage. This has nearly zero relation to how much it costs to provide the product.
>At a price of 0 you could find infinite usage
Infinite demand isn't a thing. Even if they'd offer free compute forever (not likely possible) I and many others would still use local models that we have full control over and that do not harvest our personal data.
Except of course it relates. All else being equal, we will prefer $X COGS over $2X COGS because that helps us with both profit margins and price competition.
It relates in the sense there's a minimum cost of production without losses, not the actual price people are willing to pay.
Framing it in terms of the price people might be willing to pay for a single product in isolation frames the point I was making, which was about price competition, right out of the picture.
Maybe I'd be willing to pay $10 for product A if I had other options. But if there's a product B for $3 that's not quite as nice but still ticks all my boxes, then product instantly becomes a lot less attractive.
This + your ROI in $10k device will be always lower than busy datacenter, its literally math. They sell free compute to others when you dont use it, you will never sell at that level or even you magically sell home compute, you will not compete at price
That greater efficiency only benefits the LLM SaaS providers as long as the hardware manufacturers, probably especially the VRAM manufacturers, remain supply constrained, since the high-efficiency users are the ones who can pay top dollar. But the hardware guys' dream is presumably to get parts into millions of laptops which remain on standby for 19 hours a day, not to bargain with SaaS providers who obsessively optimise their memory consumption.
I'm not sure if this prediction will hold true.
We're not seeing the progress in those "frontier models" that we have previously seen. There's certainly still gas left in tank tank, but we're way into the diminishing returns by now.
Cloud inference still beats hardware investments by orders of magnitude of course, but that's only if your data doesn't really matter to you.
We are certainly not in the diminishing returns phase for LLM progress. No sign of that yet.
I’ll grant that for specialized applications like coding agents and mathematics, but even there I suspect that most the real gains are actually taking place in the harness.
But I suspect returns may have already diminished into negative territory for at least some other use cases. One of my least favorite job responsibilities in this brave new era is figuring out how to avoid performance and behavior regressions when an older model were using for some application reaches end of life. It’s getting uncommon for me to look at our benchmark results and say, “Oh, good, it does better on one of the newer models!”
>suspect that most the real gains are actually taking place in the harness.
Part of the reason harnesses work well is you can run a lot of agents in parallel. That doesn't slow down demand.
That is true, but the eventual realization that more machines doing more coin flips in parallel does not mean "more work gets done" might.
LLMs are amazing tech, but they're terrible without oversight. More agents faster just makes reality collapse on them quicker.
But yeah, you're right, temporarily, this will still push demand. But the topic was about "diminishing returns" as in "tech getting better". Not as in "customer spending".
It's kind of weird because more machines working together does mean more work gets done. Coin flips and weighted coin flips are totally different things. Any biases weights towards reality push you closer to reality when you use them.
New models keep being able to use more and more agents on longer time frames. Your hypothesis doesn't look like what we're measuring.
Who is we?
The people mapping AI capabilities.
Oh cool, so that we includes me! :)
Maybe turn on your light when you use a ruler? Not sure what else to say.
But high demand for LLM time isn't sufficient to keep customers at the frontier LLM SaaS providers. That demand can be satisfied locally or at non-frontier outlets, absent hardware shortages at least. The Tier 1 providers (and the would-be Tier 1s) presumably need to open up a much bigger lead in model quality, one that doesn't simply get distilled away this time, and/or continue to be protected by ongoing (or worsening!) hardware shortages. (And that's overlooking the revenue shortfalls which OpenAI and Anthropic seem to be facing already.)
I had actually been thinking more about all the non-LLM functionality that go into the harnesses. I'm not going to name names and I haven't done any rigorous testing, but my general impression is that choice of harness matters more than choice of model. In terms of basic task completion success specifically, not code aesthetics.
A perfect harness will not extract gold from a dumb model. It's a system that builds on each other, though we've not probed that frontier much to have a good intuition on what effects what.
One thing that I really want to know - the better models from today vs a year ago - what has changed. They have already pre-trained on all available public data. Scooping up the last percentage of archaic texts which were never digitized is not going to move the needle.
Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?
Google scholar has a flood of papers showing LLM diminishing returns on pretty much every facet
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&q=llm...
The first title I see there is "The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs":
https://proceedings.iclr.cc/paper_files/paper/2026/hash/3b4e...
And you apparently only read the title and ignored the content.
It’s also poor reasoning to you only pick one thing you think supports your view, and ignore vastly more things not supporting it, all from the same useful criteria.
Now if only you’d carefully read the report you chose, and spend equal time looking at ample presented evidence, you’d develop a more accurate understanding.
Well I mean if I wanted to be extra pedantic, I would argue that we've been in that phase since LLMs were first introduced.
Before that, we had 0. After that, we had more than 1.
A leap as far as that is hard to recreate.
But that wasn't my point. That's just trolling.
The actual point is that LLMs aren't gaining new capabilities anymore. They just get more reliable at the ones they already have; turning what was a coin flip to some higher probability.
That's (intuitively speaking, not strictly mathematically speaking) kinda the mathematical definition of diminishing returns.
It’s a constant tension in computing that has been around since mainframes and clients… Neither is going to disappear. My general feeling is normal people care more about how thin and light something is than their privacy, so if data center powered LLMs will have a strong future.
Hmm I'm not 100% sure about that, given that edge is very viable, and the geopolitical climate has changed quite significantly.
I agree that datacenters are not going to go away, but I have doubts that the buildup that has happened is really going to pay off for most operators.
Edge based storage is very very viable, and very very cheap, people still use Apple / Google Photos (consumers don’t want any friction).
Coorperate people likely just want “the best” and are happy to pay a lot for it to make their workforce more efficient.
It’s hard for me to guess where the data center buildout will go, eventually there will be an oversupply, but we haven’t hit that yet, and at the moment the companies owning these high capacity data centers appear to be making lots of very real money in leases to hyper scalers, hard to imagine this premium continuing, but we need to see where demand for cloud based LLMs tops out.
they really dont want to hear this bro lol
On the contrary I would love it if AI stagnates. I don't want to be out of a job.
But I also don't believe things just because I want them to be true.
I can see that by those reddit-style vote swings, but who are "they", exactly?
Who is so emotionally invested into random comment sections being purely positive about their pet.. uuuuuuuh.. tech?
Very weird.
It's going to happen very soon, which is why these frontier labs are scrambling to shut down open source language models. There's an existential risk threatening their obscene returns.
It is for that reason they are being archived and torrented as a very large middle finger.
I feel like cost competitiveness of local has been going down, if anything, not up. API providers can use hardware more and have scale efficiencies. Do you see any reason this will reverse?
> It's going to happen very soon
Why? You can't just assert it. There are very good reasons to think it won't happen soon, and you've given no reasons to think it will happen soon.
Because everything is converging on a backlog of huge efficiency gains established in research, waiting to be combined. Looped transformers, a whole host of diffusion techniques and new quantization techniques, maturation of ternary distillation and new ways to separate logic from stuff that can be looked up. It would surprise me if most frontier models were actually even that big at that point in terms of active params. I highly doubt it.
You asserted that it was never going to happen first
But he gave a reason for that. "Before that happens the frontier will move". Why do you think it will happen anyway? Do you think the frontier will not move fast enough that local models are unable to catch up, or do you think people will prefer local models at a point. Or something else?
You don’t need frontier-level performance for every task. That’s why companies hire both junior and senior developers. I suspect a decent percentage of people using Fable would probably be fine with Opus.
Also, the frontier can’t keep advancing at this rate forever. Eventually the low-hanging fruit will all be gone and advances will slow down.
Or even more amusing in time for the bubble to burst, I hope the big RAM makers end up holding the whole bag for that. Greedy bastards.
Nearly the entire reason we're in this mess is because RAM/SSD/HDD makers are terrified of AI bubble bursting.
Long-term contracts, not building new fabs - they're in full on hedging mode right now.
Seemingly the AI bulls are either not sufficiently optimistic, or not sufficiently wealthy(?!), to build new fabs as joint ventures with the manufacturers in which they agree to assume most of the downside risk?
They signed long term contracts for the hardware delivery over years. AI investment is approaching 1 trillion per year and expected to grow to well over 1 trillion per year. [1]
Even over their many years, projects like the Manhattan Project, the Apollo Program, or the U.S. Interstate Highway System never added up to that. [2]
Is your argument that there is insufficient AI spending at present as they aren’t also taking on building their own fabs?
[1] https://www.pwc.com/gx/en/news-room/press-releases/2026/glob...
[2] https://www.aljazeera.com/news/2026/2/19/visualising-ai-spen...
If the long term contact are with any AI companies, then good luck getting anything back in bankrupcy proceedings!
If the AI bubble bursts, the DRAM manufacturers switch back to making conventional DRAM and pent-up business and consumer demand results in a surge of purchases, cushioning the fall considerably.
Look into the time frames a bit more, this isn’t a switch they flip. If the bubble bursts today it will be years before those downstream effects add up.
Looking forward to see them burn.
Since you’re not greedy when there is a memory glut I’m sure you’ll be willing to pay extra to make it fair.
Not at all, I’m going to relax with a cooling drink and watch the consequences of this unbelievably destructive and wasteful venture implode. I also adore the idea that it's on the customer to be "fair" to a company that dumped us a group in favor of chasing B2B money.
Their choice, their consequences.
I think they're talking about the past: memory has often been boom/bust. Memory manufacturers lose money during gluts, and did so long before AI. People benefited from cheap prices at that time.
Internet is a fad, it will implode at any moment.
In fact as a business opportunity it did just that because the massive investment in the mid-late 1990's was without anything like a connection to profitability or sustainable demand. It took years after that crash for the concept of e-commerce to begin both a recovery and evolution into what we see today.
I expect something similar for AI. It's useful tech... just not very profitable tech, and certainly not to the tune of trillions of dollars worth of public demand. The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises, but the tech will survive and thrive.
>The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises
The hype will die but there are many reasons why super intelligence and robots are a separate entity from said hype.
Look, back a few decades and tell someone our gdp would be in the trillions and it's likely they'd have a hard time believing you. A huge portion of our products that we use day to day would be complete science fiction to them.
All we're negotiating at this point is the timescale it will take.
Unpredictable things happen routinely, therefore these specific unpredictable things will happen?
These things aren't really unpredictable. Work long enough on a robot with more dexterity and you will get just that. The timetable on when it happens is a bit more up in the air, economic downturns and wars can have huge impacts on it.
But in my mind robots and superintelligence are inevitable unless there is a massive setback in humanity before then. Nature already did it once. We're not inventing something totally new. Add in every new invention and bit of intelligence we build on and actualize pushes us that much closer to the goal. Information technology allows this to speed up even further with easy sharing of information and testing.
Superintelligence is not like faster than light travel. We have many frameworks that show us FTL is impossible. I don't believe there is a single widely accepted framework that shows there is some limit to intelligence and we're near it. Intelligence isn't even a singular thing. My calculator is a super intelligent adder compared to me. It would seem highly improbable that somehow nature random walked into making brains the most efficient general intelligence device in all dimensions and scales.
Samsung is already building another fab (which was originally suspended due to low memory prices years ago). Hopefully when it goes live in a few years it's not all dedicated to HBM for AI.
Not until the industry gets severe indigestion, which doesn't seem that far off.
Not even close.
Right now, every company that can is ramping up their RAM production as much as humanly possible. There is such outrageous and growing demand, that it's like printing money if you can manufacture more RAM chips. And that massive financial incentive is driving huge investment in manufacturing capacity.
But someday, that demand will stop rising.
At some point, most people will have enough RAM. And they just won't need to buy more.
The inventories of RAM will become overfull, in warehouses and stores. And suddenly the manufacturers, middlemen, and even retailers will all find the price they can sell at keeps dropping.
But you can't shut down a huge manufacturing facility overnight. You can't shut down a years-in-the-making retail distribution system overnight, either. You can't even shut down a facility that is halfway finished with construction. They'll keep producing and selling RAM at lower and lower profit margins, until the day comes they finally accept they're losing money with each sale.
And when that happens, we'll all have more RAM than we know what to do with. It'll be cheaper than we have all ever experienced in our lifetimes.
I don't expect this to happen soon. But demand cannot keep exponentially rising forever. There will come a point, probably in the next 5 years, when demand at least levels off, even as production capacity is still rising.
And it will be interesting to see what's possible at that point. We'll be able to create software that is not doable right now, because it will no longer be constrained by memory. And even cheap laptops will massive amounts.
Yeah, that's kind of the beauty of capitalism. It can lead to short-term shortages but never structural ones. Any demand gets immediately fulfilled.
I grew up in a country where there was a structural shortage of say, shoes and cars. You didn't really even get to choose which shoe you wanted to get, when there was a new shipment, folks went to the "department store" and got whatever size they could get their hands on and later trading it for the right one. To get any car, you would submit an application and 5-8 years down the line you would get a letter that your car would now be ready to be picked up, in a random color.
Thanks for sharing that. I am grateful every day that I was born in such an abundant environment. I have a few friends who did not, like you, and it always reminds me of how lucky I have been.
This bizarro take, HBM is a triumph of non market, industrial policy. HBM is product of autocratic chaebol industrial strategy + markets (which =/= capitalism) at work. The reality is SKHynix corporate autocracy kept nonprofitable HBM going for 10+ years until it turned into unobtanium by chance, any normal capitalist+market system would have strangled HBM dead years before - Murican brand capitalism - Micron couldn't have done what SK did. HBM is literally inefficient Chaebol department store holding stock of useless misallocated shit 10 years ahead, and through sheer serenity (arguably in face of SK mismanagement) became the trendiest product around, and now positioned to reap rewards from market signals. Meanwhile memory companies that operated off capitalist impulses (intel, micro, gf) years head and can't respond to signal at all.
When you commoditize intelligence, you dramatically lower the cost to create. The amount of electronics being created is... incredible. We've really seem to have hit the asymtote part of the growth and i don't really see it coming back.
Everything is moving to edge compute. From smart rings and watches, to robotics and drones, iphones and humanoids, and no end in sight to the scaling improvements in AI... It does seem that you can continue to get more emergent behavior the more compute you expend in pre/ post training. There's no reason we can't scale up to a million data centers in space, as long as you get dramatically improved intellegence.
Yes it was great - in 2024 they were practically throwing it in with motherboard purchases.
Well uh they've been making hbm4 for how many months? Yeah I do well hope they are ramping for their sake.
I wonder how many layers theirs is. So many layers (ok, 8, 12, or 16 for different fabs). So much less dram for the rest of us.
A shortage is almost always followed by a glut. Memory of all kinds is going to get very cheap due to supply and demand and substitution effects.
Not when companies have monopolies.
There is no monopoly on computer memory.
Yes, only a small number of companies produce memory because of capital requirements and complexity. When prices go up both of those things become less of a problem for investors who can see better return for lower risk on that high capital investment. They will then invest in production to capture those returns. This is literally the exact reason Samsung are doubling supply of this particular type of memory.
High prices will fix the shortage.
Yeah, its not like companies ever colluded to keep prices high...
https://en.wikipedia.org/wiki/DRAM_industry_price_fixing?use...
> There is no monopoly on computer memory.
Never has been, but what makes you think the major players won't collude to keep prices high?
Memory factories cost billions to tens of billions, "investors" aren't going to spend that kind of money if they are not reasonably certain that they are going to make a profit, and the market is very uncertain right now with everybody bracing for a bubble burst at any moment. It's not the first time there has been such a shortage in the computer parts market, and the oligopoly, be it RAM or hard drives manufacturers, has always managed to keep prices high and production low enough to avoid a price crash after recovery.
> hard drives manufacturers, has always managed to keep prices high
No. Please take a look at the unit price for hard drive storage 10 years ago as a percentage of what it is today.
Good to see Samsung stepping up; maybe GPU prices will finally cool down a bit with more HBM supply hitting the market.
This insane nonsence needs to stop.
You just know that the AI bubble is going to break and they're going to be stuck with tons of HBM4s.
The big problem is that this will not depress the consumer DRAM market and mean we can all afford DRAM again
I'm sure you can make it work with HBM chips on adapter boards for PC. And there should be lots of HBM chips from scavengers taking apart bankrupt data centers.
HBM dies are designed to be physically mounted on top of other dies in the same package, they can have up to 2048 data pins (compared to 8-64 on consumer dies which are in their own packaging) - to do what you propose you'd need an interposer chip of some type which would push the price back up
If there is enough scrap chips & enough demand, I can still see it work.
The AI boom is turning into a massive race for HBM capacity.
Good, maybe this means cloud GPU prices will eventually normalize. Still feeling the pinch from the AI gold rush.
To me, the most idiotic thing is that the world currently cannot build power plants fast enough to even use all the HBM that is being produced. That is, a large portion of HBM and cards are sitting in storage, waiting for the data centers and the power plants to run them. The FOMO in the AI world isn't just in the stock market
This is crazy!
Now make some DDR5.
HBM is DDR5 in a different package.
If it doesn’t fit it’s no use to me.
[dead]
[dead]
[flagged]
Why wouldn't it get posted here?
Because you are not familiar with the site.
The beginning of the bubble pop?
how would you even derive that conclusion from the headline/article?
https://youtu.be/k_gaZjXD5OY?t=200
so you fell for clickbait.
its ram for the types of systems ai needs in datacenters so...no?
About 2 years ago I bought some SDRAM or something like that, DDR4 or DDR5, don't recall offhand. A few months ago I looked at the price today, and it was over 3x as high. That's just insane. Governments need to do something against this abuse system that AI amplified here.
> Governments need to do something against this abuse system that AI amplified here.
Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.
No one has all the cards and no one controls all the levers.
Governments have sued DRAM companies for price fixing and forming cartels before: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing. They can absolutely step in and if there are evidence of price fixing or anti-competitive behaviours, can impose fines. The choice of whether to do that though is political.
And on that page govts have tried and failed at it also. Thinking the current prices are the result of fixing and not insane demand is surely going to lose in court. It’s trivial to show massive demand, and thus production line changes, as the driving forces here.
If you look over all such lawsuits, they extremely rarely succeed.
Is there any allegations of price fixing in this case? The demand just skyrocketed. Other than legislation like the defense act in the US I don’t know of any governments that have jurisdiction over these ram makers that mandate companies produce more of a product.
Especially not for consumer goods.
If they're charging one customer $X and another customer $5X for the same chips... there's got to be some law about this.
I missed that this is happening, is it really? I thought it's 5X for everyone
Supposedly OpenAI is paying 2025 prices.
So you mean people with contracts are having their contracts honoured.... thats just normal business.
Several of these companies almost went under before the AI boom. There isn’t a cartel.
No one has all the cards and no one controls all the levers.
I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
I don’t really know much about how this stuff works, but I have a feeling we wouldn’t even have to do anything punitive. We might just need to find and close whatever weird loophole allows the folks who are participating in this gold rush to feel confident enough that they’ve externalized their risks to be willing to engage in speculative data center buildout projects on such a grand scale in the first place.
Apple used to do the same thing with TSMC’s newer process nodes, and nobody really complained because people were getting superior products. I happen to like US antitrust enforcement that considers consumer benefit primarily.
Your attempts to insult what they are doing doesn't show they did anything wrong, but it does impeach the validity of your own argument.
Hit dog hollers, sounds like.
That's abuser logic.
The funny thing is, we're mostly in violent agreement, going by your other comments. This is a weird point to disagree on... so weird, in fact, that it's easy to believe you are associated with OpenAI. Hence my follow-up comment.
What Altman did, and the way he did it, was not OK. Unless you believe that humanity's adoption of AI-based intellectual work is best served by concentrating an enormous amount of power in one company at the expense of everyone else, turning the market into a literal zero-sum game. Do you?
The same major memory makers got prosecuted for cartel behavior by the DoJ a little over 20 years ago[0].
One might say "well what specific law have they broken" and I don't know and I don't know if they have, but at the very least the power of the State _could_ be used to force behavioral changes in some ways or another.
[0]: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing
The same governments that dumped bilions into to these companies and gave them insane tax breaks.
How much memory does it take for the consumer market not to be totally screwed?
Can that share of production be allocated to consumers, and the AI fights over the rest?
Well, thats kinda how strategic reserves work - so that say food manufacturers just don't dump all the cheap food to sell crack instead & everybody starves to death.
It is funny how regular people are trapped into legalism while companies are seeing things as a two sided fight.
You can’t demand anything with this mentality.
“The current situation is messing up the economy for a very large portion of us” is universally a good enough reason
What abuse? Responding to supply and demand fluctuations isn't abuse, it's the proper operation of the market.
Would if one industry gobbles up an important resource to the point that other consumers lose practical access to it, shouldn't there be limits?
No. The other industries can bid for access to the resource, and under most circumstances, whether they are willing to bid higher tracks importance. Sometimes there are cases where something has significant positive externalities and so people are not willing to bid high enough, and then we can talk about some kind of government intervention (typically a subsidy that attempts to track the value of the externalities) but I don't see that applying here.
If others are willing to pay more, that a signal that more value is created if the resource goes to them rather than to the more price sensitive customers.
This logic falls apart completely when gamblers allocate one trillion to bidding up the price, denying access to the resource to people who are responsibly spending their own money. The ideal world of the imaginary perfect free market never takes into consideration the messiness of the real world, like the fact that people will spend extreme sums of money irrationally. It can take years for this effect to correct, damaging the market severely in the meantime.
Alternatively, the money invested can be rational because it prices out competitors and establishes a monopoly, after which point the monopolist earns their absurd investment back with complete control of the market. This is also bad.
You can reach any conclusion you want if you assume others are behaving irrationally. But why should I assume they are wrong instead of you being wrong?
This has zero to do with the company that is making the product and all to do with the company buying them.
The problem you have here is hundreds of different companies are doing the "gambling" in a non collusionary manner so it's going to take decades in court to prove it. Are you saying governments should do an authoritarian take over of RAM allotment?
And even if the companies are "gambling" (and what company doesn't do that) it doesn't mean the gambling is irrational.
Where is the additional supply resulting from the sustained increase in demand?
Some are building new fabs, example:
https://news.skhynix.com/en/fab-facility-investment-2026/
It's not an overnight thing obviously.
Check the article we're commenting under
No, this article is not about new fabs.
3x? I bought 128GB DDR5 SODIMMs for 300 eur. Now it's 4000 eur.
Damn you were lucky. I bought 96GB DDR5 last year for £600 UKP. They're now retailing for three times that.
Bastards.
Yeah the 300 already felt like a splurge - it's the max. 96 was less than 200 I felt kinda stupid not just going with that. Imagine 96 of anything now for less than 200
I just checked the latest pricings. 48GB DDR5 SODIMM now selling for £2,400.
It's not just normal consumer memory. Everything with a chip in it is getting more expensive because of the AI circlejerk. When Bitcoin screwed over consumers, at least normal memory prices weren't affected (though that time a shitcoin based around storing large amounts of data made the rounds, hard drive prices did take a brief hit).
Everything is affected by the endless thirst for RAM by AI companies and the accompanying hyperscalers. I'm honestly disappointed governments aren't doing more to keep the price of consumer goods in check.
I guess the AI companies telling the world how good they are at accidental cybercrime and fraud is creating enough FUD to let the AI industry run wild.