Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute are fundamentally different, and the only reason people put up with it is because Vulkan and HIP C/C++ are even worse.
Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development and sane low level API to develop against, instead you have to go through their cloud and their full software stack which greatly limits ecosystem growth. The minute that Google figures that out, that's when Nvidia's dominance would be challenged.
The CUDA runtime coming with a gazillion reasonably decent kernels (DNN, BLAS, CUTLASS) and a concurrency system (NCCL) is a big deal; especially in the “early days” very few researchers or development runtimes were even writing their own kernels or dealing with CUDA C++ extensively, they were wrapping the ones NVidia gave them.
I do agree that it’s really not great, and I also have never been a strong believer in the CUDA moat overall; as the need for GPUs moves from research to production (inference), companies are plenty willing to build software from scratch anyway (and we see this with AMD GPUs being in plenty high demand in the datacenter and enthusiast market now).
What I don't quite get is why can't they use AI to translate CUDA programs into more open architectures like AMD ROCm?
AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?
That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more open than potential competition. Could it be that Nvidia has been on top because none of the competition has actually tried kicking them where it hurts?
I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem.
AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability.
Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.
I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all.
Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC.
And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api.
Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources.
Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO.
I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
I think you are pretty precise and your comment is appreciated, so I think I need to clarify my wording as well. I think a lot of CUDA C/C++'s difficulties are self-inflicted because they conflate two things into what they call "kernels": runtime and shaders. Fundamentally, GPU programming for compute is easier than either graphics programming as well as CPU programming because you are limited on what you can do on the GPU, memory allocation is static because the matrix size doesn't really change dynamically during runtime and branching behaviors are generally to be avoided for GPU compute.
It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing.
The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.
Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars.
Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel.
Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.
Microsoft pulled it off with dotnet, sort of, because they approached it like a completely new language, sold people on the benefits of it.
And the people they were selling that to, ( It's free and open souce now! ), were a very different group to the market they left behind on .NET Framework, who are often still struggling to make the transition now.
Had they actually killed off .NET Framework, it would have been a different story, much more like the VB6/VBA to VB.NET transition, which so few people bothered with that VB.NET died out, because if you had to retrain that much, you figured you might as well go to C# or a instead, or indeed a completely different language entirely.
I briefly worked professionally on a VB.NET project, but outside that job I've never met anyone else who can say the same. I've met a few who went straight from VB6 to C# though.
The whole push behind .Net Core was to get .Net running on Linux well and natively and was being lead largely by those of us using .Net for web. We were being murdered by other languages at the time; mono was an option but Microsoft shops usually needed to target something blessed/backed by MS.
Largely the same market (Enterprise) but not the different segment (web as opposed to Windows/WinForms).
I ported about 15 years of projects from various versions of .Net to .Net Core whilst they were developing (and sent feedback to the team - they were asking us to do that) and the process was pretty reasonable. You were only really stuck if you were using something very very Windows specific (certain image processing libraries iirc) and even then it was largely manageable.
The old full-fat framework is, AFAIK, still supported, as there's a whole lot of legacy code which is Windows specific which is still expensive / hard to port over.
Unfortunately "something windows specific" was pretty broad.
Between MSMQ, WCF over named pipes, MSDTC, and MSI installers, there's a lot to replace that is hard to provide the same guarantees or performance with straight replacements, if they even exist.
The end goal, being on modern dotnet, is better, but it's difficult to get there with a phased approach without accepting a temporary worsening, which is often hard to sell.
Both of those were a vendor X electing to stop updating framework A in favor of framework B. That’s a high cost for vendor X’s users, but if their business depends on vendor X, they have no choice. Maybe they can switch to vendor Y at that point, but now you are switching both vendors and frameworks.
Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features.
Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.
Neither of those are a competitor replacing another, they are a same-vendor replacement. This is easier, because the company whose established product you are trying to displace is cooperating (because its you!) not actively resisting.
One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat.
It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI.
The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA.
It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA.
There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task.
I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.
CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you.
Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.
ZLUDA is still alive. AMD sponsored it and the project was briefly halted during a dispute with them, but it’s been making steady progress.
It doesn’t really make sense for AMD themselves or most use cases, though; any compatibility shim just adds problems on top of problems, and for AMD, entrenching a competitors technology even more never really seemed like a great idea.
Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop.
Management likes it because it removes software developers from the loop.
The biggest advantage of tpus is the high bandwidth fiber optic interconnect between them that allows distributed computing on pods with thousands of tpus and the co-design of cooling systems that go with their racks. I do not think that we will see personal tpus any time soon.
I had a hard time understanding why didn’t AMD make a better developer experience for this two years ago, and am now even more baffled that even with all the LLMs they still don’t seem to have moved a single inch, despite this probably being a tens of billions dollars worth feature.
LLMs make the dev environment almost irrelevant. llama.cpp supports AMD with both ROCm and Vulkan and that's nearly all that matters now. TBD how much AMD's AI Halo play will change things if at all, but they got a lot of positive press in launch reviews for having an actually robust software story for once.
Hardware companies are notoriously bad at software. The software they use sucks, the languages they use suck, the internal tooling software they write sucks.
They don't know what good developer experience is, how do you expect them to deliver it to other people?
I’m not entirely sure if local development will lead to Nvidia’s supremacy being challenged.
I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.
TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.
Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.
Eventually the other chips will go through enough iterations and stability will be reached
Are the switching costs of CUDA ecosystem potentially threatened because LLMs are now quite good at transcoding into other languages? In other words, is Nvidia's greatest strength (AI) also potentially its undoing?
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board
Nvidia's sells hardware yet their market cap is about the same as Google's.
How much value could Google get by selling hardware too? Google'd be selling to competitors, so difficult to capture much of the value and would decrease Google's value as an AI company. Maybe a child company?
Genuine question. Given that LLMs are supposed to allow us to rewrite anything, and I am an LLM believer, what I don't understand is: how does CUDA continue to be a moat in a world where LLMs can rewrite entire software development stacks? If NVIDIA is right about AI, isn't this same technology going to erode the software side of this same software moat?
Interesting take on Google's TPUs. What I've previously heard (and still believe) is that Google's decision to only rent out, never sell, their TPUs is a deliberate and savvy strategy for bolstering GCP, which will work provided that TPUs are able to actually compete with other hardware (in practice meaning Nvidia). A few months ago there was some discussion on HN comparing them, and I think the verdict at the time was that their latest-gen TPUs win on compute-per-Joule for LLM-type workloads by quite a margin, which I think is huge for those who want to run LLMs at scale.
In many investment theses - like Nvidia's bet that demand for compute will keep growing - the first order assumption is usually correct. Yes, demand for more compute, chips, infrastructure is huge and each year some additional data centers will be built. Where such investment bets usually fail is in the second-order assumptions: Ie. the expectation of the growth of demand. This is where there's a high chance that the current expectations are likely exaggerated. So: demand is likely to persist for the foreseeable future but not increase every year. And that can upend the whole investment story. That can be enough to make these bonds a huge burden for Nvidia in the end. Not because people stopped buying more compute but because they stopped buying more every year.
What makes this insanely hard to predict is that the compute needed for the same quality output has roughly gone down 90% every 18 months for ~5 years.
1) We don't know how long that trend will continue, but you do know where to look for when it may end (if smaller sized models continue to compress the knowledge effectively of larger models).
2) We don't know when the appetite for higher cost models might go down and by how much if smaller models get "good enough" and price becomes far more important.
It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.
It is also entirely possible that at some size - LLMs pick up some emergent capability that doesn't scale well to smaller sizes - and that there's an incredible boost to demand to get that capability.
I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side.
The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have to switch to something like burning the model weights into silicon to continue to make gains, then the current math on general purpose accelerators might be upside down.
What's really interesting is that if you scale it to higher densities (eg 3nm and stacked die) with ComputeInMemory for fp8 you can reasonably start to fit 30B-70B models. With MoE and multiple stacked die, just like HBM, you could fit an open weight near frontier 1T model (like GLM5.2) at similarly much lower power <10kW and high token rates >2ktps. For running a bunch of agents where fill rate and speed/latency are important it may not matter that you're 6-18mo behind on weights. The process for the chip design could be largely automated, and new silicon pumped out as new weights are available (with a 3-6mo delay).
> efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute
I buy that. Jevon's Paradox, sure.
> and we are not going to run out of economically useful things to do with it anytime soon on the demand side
This I don't buy. Not fully, at least. Whether or not there's demand for LLMs in some particular field is one thing, whether or not there is a sustainable business model to be built out of that demand is another thing entirely.
There is a staggering amount of money pouring into startups looking for novel use cases for LLM-based agents. As usual, 99% of them will fail, but those other 1% are going to have to look harder and harder to find a novel use case that can actually be served profitably.
First of all, there's only so many places where a chatbot is going to sell. But, that also seems to be the only interface anyone can come up with that allows a user to steer an agent through a long-running task reliably. I'd love to be proven wrong here.
Also, if current trends plateau and large datacenters are still needed for complex tasks, that would stimy growth of LLM usage across entire industries.
But, if present trends continue, then local inference will become feasible for most tasks. That would lower the barrier to entry across tons of heavily-regulated and/or cost-sensitive industries. But, widespread local inference will almost certainly come with a painful market correction centered around hyperscalers, which would itself dry up the pool for ventures into new markets.
Replace "chat-bot" with "Human Being" because the models I've been using are significantly better than 90% of the human-chat-bots that I must talk to on the phone while scheduling and coordinating my internet installation for example.
Now for every human replacement, that is 1 unit less of communication and bureaucratic burden (HR, middle management etc) that the org requires.
> the models I've been using are significantly better than 90% of the human-chat-bots that I must talk to on the phone while scheduling and coordinating my internet installation for example.
I guess I don't know what to say except that my experience is the polar opposite of yours.
I moved to a new state at the beginning of the year. Needed a new doctor, needed to schedule apartment tours, needed to talk to my employer about insurance and relocation stuff, etc etc. Lots of chatbots, a handful of humans. Humans consistently did what I needed them to do, the chatbots just didn't. I could list examples but I'd be typing all night.
And yknow what, my one call with Comcast to get my internet set up was downright pleasant. The rep was knowledgeable and a good conversationalist.
Very few consumers are going to spend multiple thousands of dollars to save $10 per month. Companies absolutely will to save hundreds per month per employee, but that's not consumer hardware.
Well if the trend that the comment further up in this thread claimed continues and compute requirements keep dropping exponentially then perhaps in a few years you can have today’s frontier performance on the normal laptop you already have on your desk anyway.
If you can integrate AI accelerators into consumer cards (you can), you can have local AI for "reasonably" cheap. This is Nvidia's long term goal if you listen to what Jensen has to say.
The limitation is entirely on memory right now. Just a few years ago we could of been strapping 80-100GB to cards for under $200 (BoM).
It could be quite a while before we reach that point though. 5+ years easily.
I've been keenly interested in the ability to run local models, but the hardware is just not there. Consumer RAM speeds and capacity will have to significantly increase before local models will be able to perform as well as even the lowest end GPT-5.6 Luna model.
This is on the backdrop of RAM becoming prohibitively expensive. And without the speed and quantity of RAM, it becomes impossible to generate tokens at interactive speeds, regardless of model. There is a fundamental dependency between calculating all of the active params with the given RAM speed.
Even with a model that has been quantized all the way down to Q4, the DGX/RTX Spark chip with 128GB of RAM can only generate ~18 tokens/sec for a MoE model with only 30B active parameters. There haven't been any broadly useful models below 30B active parameters. And that is for a $5000+ piece of hardware that will be one of the best for running on-device models.
I really want to buy instead of rent my AI, but the economics are truly terrible.
It's also hard to predict how much money will be burned going down wrong avenues. The internet was the future, but it took a lot of failed companies to eventually land on a sustainable model that brought us the giants we have today.
Railways were also the future, but that didn't stop a rush to build out (often subsidized) lines that were ultimately uneconomical (either because they were corrupt or the planned settlements never arrived).
If AI is similar, then there's going to be a long slowdown on compute spend until the surplus is worked through. A good historical analogy could be the fiber optic buildouts of the late 1990s. The demand for data never really went down much, but the industry eventually commodified and took down some large companies (Nortel, especially)
I rhink there's a difference with AI because -- it brings true value because you pay for the tokens, you only pay for what you use. That is true value.
Compare to just paying for an internet connection, you have bandwidth but not sure what you can do with it that is valuable.
Let's say you use AI to produce software. There;s no limit as to how high the quality you want your software to have. And how fast you want your project to be complete. There's plenty of room for higher quality, and more performant AI. As AI becomes chepaer people will use more of it, they're not going to say "We have enough AI".
Compare to railroads. Yes you pay for the distance travelled but there's a limit to how much people wwill want to travel, how it will benefit them.
It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.
I doubt it. If LLM inference efficiency is 50x better than today, then there could be 1000x increase in inference volume due and we'll end up needing even more chips.
Jevons paradox should win out for a long time for AI.
When internet connections got faster than 56k modems, we didn't use the same amount of bandwidth but faster. We used more bandwidth doing things like 4k streaming. I see the same in AI inference. If AI inference is that much more efficient, it will just enable more use cases for AI.
Nvidia's great superpower is flexibility. You can easily run models of very different types on the same card; and their hardware is great for R&D.
However, at some point AI may be good enough for most people and then it makes sense to make an ASIC for the model (or group of models); and at that point you don't need Nvidia.
I suppose this scenario will happen in various moments at different levels.
Each of the hyperscalers has put like 250B each in the last year for infra. That means that they need to be writing AI profits to the tune of 20B per year just to keep up with the cost of the cash they burned.
We are not there. But they better figure it out soon. The cash flows dried up, and everyone is taking debt to support the capex. Google for the first time in its public history is cash flow negative. Amazon too.
Plus on top of that 1st and 2nd order can be correct, but then the price is too high, meaning people lose money even if correct about the future, but over pay for it.
They also have to be feeling the heat of the ASIC vendors. AMD just acquired Taalas and they work with Cerebras all the time on special projects. ASICs outgun nVidia's chips by an order of magnitude.
More interesting take on Nvidia's position than I've come across before. One thing to be noted is 1) Nvidia is already making moves in robotics so even if their position in AI (moreso llms) diminished, they certainly have another big avenue arguably harder to just get into (although I'm not sure what efforts Google is doing for the tpu in robotics). Another point is Nvidia is still the main player in the west, that is, China certainly can and will create their own full stack without reliance on US companies. That puts Europe and other countries in an interesting, do you buy Nvidia because it's the only option or for security. That's to say I believe Nvidia's position relied on many different things being true at the same time, and we're moving towards an environment where those things are certainly being contested at (roughly) the same time.
Even in the west, Nvidia's dominance is bound to weaken. There is a notable uptick of articles on HN about people running large models on AMD hardware. And while I don't know official sales figures, I know we have trouble getting our AMD system delivered
AMD's software story is still a lot worse than Nvidia's. But patching up vllm to run one or two models you care about on AMD hardware is a much easier proposition than using them in most other fields of AI.
I think this is a great example of the disconnect people have in these types of conversations.
You can both become a company that supplies 80% of the world with your type of product, and then still have your stock go down in value.
All it takes is over evaluation by the stock market. Then a course correction from unsustained growth on growth (second order). So even if you continually replace YoY 80% of the world's hardware on a rotating business, but you don't increase market share or increase demand (aka growth)... Your business looks stagnant to the stock market, and there isn't really anything you can do about it. The best you can do is track inflation +/- 2%.
And that's why a lot of older established companies were dividend stocks. You don't expect to growth anymore, but that's not where the value is anymore... The value is in the reliable sales that will happen after infinitum because your company controls a majority share of the business... And that's ok! Unfortunately, silicon valley has created a philosophy of 'you gotta expand into new fields or your on the decline' - aka neo-monopolization
Nobody is saying they’re outright failing, but that they’re not going to be printing money the way they have been recently. Think about Intel circa 2010: most of their competitors like POWER or MIPS were marginalized, they owned the desktop and server markets with a bit of competition from AMD well contained, and their biggest desktop competitor (Apple) had just switched. A lot of analyst predictions … did not match what happened next. The same was true of Cisco a decade earlier. Both companies are still there but they don’t set the terms in their market segments.
I’m not predicting Nvidia will MBA themselves to death in the near future but I think there’s a tendency to overstate how profitable companies will stay. The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
The current administration is definitely corrupt enough that you could imagine an anti-competitive deal of some sort but I don’t think there’s a way for even that to change matters because key competitors are well-connected American companies willing to play that game, too.
> The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.
The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architecture for training and inference, they're separate ASICs unlike CUDA cores/ALUs. Veterans of crypto mining will tell you that the ASICs lost in the end, as Nvidia was evolving their hardware faster than the ASIC manufacturers could iterate. When the crypto acceleration landscape diversified away from ETH/BTC into altcoins, Nvidia was still there making money hand-over-fist from mining hardware.
I guess you could argue that robotics, world models or computer vision won't be a trillion-dollar market. But Nvidia is positioned to be the first mover in all of these markets, and none of their competitors are even coming close to the integrated stack that they sell consumers.
> But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.
How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized.
My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.
Money wasn't really the problem. Apple pulled OpenCL together pro-bono, and worked with Khronos to find willing industry stakeholders that would oppose Nvidia. OpenCL needed hardware standardization though, and nobody wanted to design or implement on a scalable GPGPU architecture like CUDA had. AMD and Apple both bet big on raster efficiency, which turned out to be a terrible play when Nvidia was already putting dedicated ray tracing and tensor hardware into their GPUs. They both bet the farm against each other, and only Nvidia won.
Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.
I stand corrected, only Google's post-Ironwood TPUs have the split as well.
Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.
For awhile I've found two things hard to square, that the hardware and software making up current gen AI will bring us to a socioeconomic singularity, and the reality the thing they're mostly trying to emulate is a few pounds of meat and fat running on tens of watts equivalent. On one hand the current AIs are obviously super human in some tasks, get completely dunked on in others by far simpler organisms. My cat can catch a bug out of the air, Fable 5 in Cowork can lack the dexterity to make a slideshow because I had LibreOffice instead of Microsoft Office. Not even close to analogous, but point being they appear to have pretty fundamental differences in how they can interface with the world that the economic thesis seems to gloss over.
Another interesting discrepancy is that people think current GPUs are maybe capable of running AGI but they still can barely manage photorealistic rendering of a single room in realtime, or simulate something like a shirt thrown into a pile of laundry. They can generate a video of it based on millions of existing videos, but not do a real simulation of light and physics in realtime.
Is the human brain capable of doing real simulations of light and physics in realtime? Or does it hallucinate the details and represent some low-resolution mental image? You may be overestimating the capabilities of flesh-based neural networks and underestimating the capabilities of silicon-based neural networks.
An AGI doesn't need that level of detail to do most tasks effectively. To use the OP's example, a cat catching a bug out of the air does not need to run a fluid dynamics simulation of airflow over the bug's wings to be able to catch it. A cheap approximation of the flight path is sufficient. Perhaps some physical tasks will need that level of detail but many will not.
No need. The 'apocalypse' will come purely out of the hubris and greed chasing untold fortunes as opposed to a healthy, considered integration.
The decision to scare the shit out of normal people early on was a short-term decision that will cost everyone dearly (it led non-technical people to believe in things about LLMs that just aren't true—even worse, they've now made difficult business decisions on bad information with seemingly limited consideration of secondary effects).
The amount of short-sightedness alone should give pause, but at least in America, we seem to have collectively lost our minds in worship of the almighty dollar.
Nvidia has been playing a dangerous but profitable game since the Crypto boom.
but now I think they probably have bitten more than they can chew.
Apple already proved with their unified memory - that as long you have the capacity you can run capable models locally - thereby goes demand for inference if everyone is running some model locally.
For training - Chinese models have proved that you don't need the latest & greatest in Nvidia hardware. Same as TPUs.
It seems they had a head start but are now facing stiff competition on all fronts. Software moat, GPU's for gaming, and its distant cousin datacenter compute. They rightfully invested their insane profits into many ventures, and how many of those have turned around into profit?
They are also a robotics AI company with Omniverse. They are also an AI company with Nemotron. They are also a bleeding edge network equipment company after the Mellanox aquisition.
They stand to make a lot of money if they succeed in every venture. Good for Jensen taking risks and driving innovation, I hope they succeed in chewing even 50% of what they bit off.
> They rightfully invested their insane profits into many ventures
This is the inherent downside of such a rapid rise to being the world's most valuable company AND still being considered a growth stock. At their massive scale, the number of new adjacent businesses that have both sufficient size and potential growth is limited.
> I hope they succeed in chewing even 50% of what they bit off.
Anything approaching that is vanishingly unlikely. They're being forced to play the game more like a VC. The question is if a few unicorn winners can offset dozens of losers. The challenge is that, unlike a VC, their bets are much more correlated around AI.
Right, it's clear that nVidia is taking care and trying to position itself so that it can continue making sales if and when inference goes local. And it's in a much better intrinsic position to do that than the LLM SaaS vendors are: nVidia sells shovels to the army, but it also knows how to sell shovels to Walmart. Whether the financial relationships that Huang's got his company into will cause it problems if the market shifts is a different question, though.
I don't know how people can say this with a straight face. Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced, specifically for edge robotics, computer vision and ML.
The absolute fastest desktop Mac GPUs cannot beat an Nvidia laptop GPU in prefill or inference speeds. Apple Silicon is a non-entity for professional datacenter deployment and arguably unusable for frontier models at agentic context sizes. AMD is Nvidia's primary worry, and they're not doing much better in terms of GPGPU SOC compute.
>Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced
You can believe all you want that the dinky little jetson boards were desktop grade when historically the ARM SoC portion of a jetson board couldn't even keep up with broadcom/rockchip SoCs. It's taken until recently for the actual arm compute portion of Nvidia SoC's to be worth a damn at all, and they still fall far behind Apple let alone the rest of the pack like Qualcomm/Samsung.
I don't have to believe. I've run KDE and GNOME on the Tegra boards, you get full-fat CUDA support without sacrificing Vulkan drivers. It's incredible.
You can believe all you want that good single-core performance will corner the edge compute market. It hasn't, Graviton has more buy-in than any Apple Silicon chip ever got.
Every time nvidia takes its fab time and uses it to build anything other than datacenter chips it is losing money due to the massive markups the datacenter products have. Expanding their consumer offering means the datacenter backlog is going down which is very bad for their margins. Consumers will never pay 10-100x what it costs to fab something like datacenter users will.
In many cases it isn't Nvidia paying for the fab time. For instance, the Nintendo Switch 2 is basically a pure-play design and support product for Nvidia, while Nintendo negotiates with Samsung for the actual SOC prices. Their IP philosophy is closer to AMD's than Apple's, Nvidia has long roped in 3rd party manufacturers to mark up, integrate and sell their hardware.
I'd be happy enough if Nvidia decided to launch an update to the now dated (2019) Shield streaming box, which is still the best on the market as to codec support and image quality.
Yes. The more capable SoC PCs become, the weaker the "everything has to run in the datacenter" argument gets. As more powerful SoC PCs from multiple vendors appear over the next few years, that advantage may shrink for many workloads.
How much in value could Nvidia safely drop and over what period of time for it be fine for greater economy? What level of correction would be manageable?
And I am pretty sure that their value will drop in 5 to 10 years.
4.7% of the discounted future earnings pie of 3761 of the largest current publicly traded global companies.
That’s still a lot, but “the global economy” over the horizon where those earnings are not discounted to 0 contains a lot more than what’s listed in VWRA today.
IMO focusing on the hyperscalers is kind of misleading.
Yes, for programmers and tech companies AI is kinda boring now, but AI integration in general is still kind of uncharted territory.
There are so many small companies and individuals just getting started with AI today and I believe a large the customer base (and revenue) is still untapped. Hell, I’m discovering new use cases regularly still and the average mismanaged 30 people whatever SaaS vendor probably didn’t even get started yet.
This is what so many people on HN and the market are constantly missing.
Jason Kottke almost didn't found his blog in 1998, famously quoted as saying: "I thought I was too late, that no one would be interested." Needless to say, the internet was a tiny joke in 1998 compared to what it is now.
We are just barely scratching the surface of what's possible with AI, both in terms of the leading edge and in the 'torso' of the economy (the portion you're describing).
Folks from Silicon Valley working in AI-forward companies have a skewed perception of how many people have adopted this technology so far. Codex recently celebrated hitting 10 million users. This is a great milestone and all, but to put it in context, Microsoft office has a billion users. Sure, many people use Claude Code and or some other harness and the growth is staggering, but the overall scale is tiny compared to software as a whole. Costs of serving and usage are still very high, prohibitively so for many, so we aren't even close to market saturation.
And even at the leading edge, people who do work in those AI-forward companies; models are still slow, require hand holding, and produce suboptimal outcomes sometimes. Imagine the value when instead of needing to prompt it once per 30 mins, you prompt it once per day. Then once per week. Then once per month. Imagine all this running not on 3 trillion parameter models, not on 10 trillion, but 100 trillion. What kind of computer infra will be needed then? Certainly more than we have today.
I've always marveled at how one can pick any year since the internet went mainstream and in that year people thought, "Oh my goodness, the internet is amazing!"
Then, move five years forward from that year and look back. In every case, people think, "Hah! The internet was so simple then!"
So much of the dotcom bust was essentially: "anything internet will work", but even after it was pretty easy to find different ways to publish web pages or migrate to new forms of social media, video, short-video, etc.
It's orders of magnitude harder to automate work, which is the business value proposition of AI (whether you are displacing worker or adding breakthrough capacity), and it's not clear the LLM hyperscalers will get the value-add from that.
On the consumer side, it might be a race to the bottom: same ad profits, but now you need AI to produce it.
I wonder why Google doesn't create an open-source CUDA alternative. Google released Kubernetes to stay relevant/competitive in the cloud wars, they were a distant third. They now have an opportunity to create an open source industry standard.
Or the companies spending trillions of dollars can do a Manhattan Project (Or X-Prize) and let a thousand startups work on it. One will succeed. Between Google, Amazon, FB, Microsoft, Apple, AMD, Qualcomm, Intel (and dozens of other companies) there is enough economic incentive to do it. Also, isn't this what AI is supposed to be extremely good at, CUDA experts can continue to write CUDA (without having to learn anything new), a translation layer will rewrite it. If software can be one-shot from markdown files, this can't be impossible.
Phew. I thought we were just about to leave it, albeit not quite getting to space, just dissolving the financial system so our AI overlords can make some more harvester drones.
Tend to agree with Ben's thesis RE Demis and DeepMind not really being focused on the agentic coding race. That being said, it remains to be seen whether Sergey and Koray can inspire the foot soldiers in the same way that Sama and Dario do. I'm not too optimistic, and that's to say nothing of the fact that Google cannot possibly hope to compete with these other companies on potential employee upside.
> Google cannot possibly hope to compete with these other companies...
It's possible Google has intentionally decided to take a more conservative blended approach than purely competing at the bleeding edge of the frontier. If so, they obviously have no incentive to state it publicly but the recent departures and financials are consistent with the idea. It also makes sense that a company so much bigger, longer-term and (somewhat) more diversified than pure-play frontier labs would play the game to align with their strengths (capital, balance sheet, breadth, etc).
In their position, why not take an 'arms supplier' strategy in the near-term while drafting behind the frontier labs as a fast follower in AI, essentially betting the AI race is more akin to the Indy 500 than a quarter-mile drag race. If they're wrong and it IS more like a drag race, Google is in a better position to absorb and adjust than a frontier lab, for whom current valuations and capex spending requirements nearly require this to be a relatively short, winner-take-all race.
I think the linked semi-analysis piece is about right. Google is more focused on protecting its 4.2 trillion dollar golden goose than pushing cutting edge which leads to a bureaucratic nightmare that researchers simply dont need to engage with when openAI/anthropic are offering even more money. Why waste time dealing with bureaucracy at google when you can be top dog at openAI or anthropic?
Laughable to cite and reiterate the idea that Google is cooked where it comes to SoTA and AI in general when they operate, reliably and successfully for decades, one of the largest computing infrastructures on Earth and will likely continue to usefully serve the 90% of AI requests that don’t involve managing large codebases. Also seems strange to suggest that Google would need to Aquihire a company like Thinking Machines when it could spin up their AI model in a couple of weeks on its own TPUs if it felt like it. Demis likely wants to focus on his specific interest at the junction of biochemistry, neurobiology and computation which is more specific and unique to Demis than building a general purpose Q&A search model.
There's another factor which Ben failed to consider. Which is that NVIDIA doesn't need to rely on demand for their proprietary CUDA stack or their GPUs growing -- they are already selling directly to the consumer, and likely capturing much higher margins. They are moving up stack, not down, where demand for raw compute matters less. With the DGX Spark and Jensen’s statement about “open models”, their next product is likely a strong hint: consumer devices to fulfill the Mac Mini demand craze. They are probably going to start burning LLMs durectly onto sillicon and then selling DeepSeek-in-your-home to individual developers. I bet that would sell even better than Anthropic Max coding plans and is not dependent on hyperscaler funded boom-bust cycles. So Ben’s analysis highlights the risk of their existing business not growing but they are likely planning new businesses.
There’s a massive difference between “going up forever” and “point B is higher than point A.”
The current setup can’t sustain a downturn, even if yes 20 years from now point B is likely to be higher than present.
That’s the danger. Those that are going to get wiped out by the AI bubble burst aren’t wrong about AI being huge long term, they just put themselves in a position to not survive the storms that happen between points A and B.
People and pundits have been dooming and bearing on Nvidia for as long as it's been around. It increased in intensity when gpus were used for large scale crypto mining and it became material to their operations, and continued as AI ("the bubble") started to really take off.
Over those years, my NVDA stock has been by far my biggest winner. I'm now up more than 1500% on it.
> The subsequent bankruptcy of Jay Cooke & Company triggered the Panic of 1873, culminating in endless railroad bankruptcies across the country, a multi-year depression, multi-decade deflation, and, one could argue, the financial conditions that made Europe, four decades later, into a tinder box.
American history education needs some dire reform.
> After the departure of DeepMind CEO Demis Hassabis (technically promoted to chairman, but no longer in charge of day-to-day operations) and Gemini co-lead and former Chief Scientist Jeff Dean, along with a host of other prominent researchers, SemiAnalysis declared that Gemini is Cooked: "For all intents and purposes, we believe DeepMind is no longer a frontier lab"
Counterpoint: xAI pooped out a frontier model based on nothing but capital and one man's desire to push a right-wing political narrative. Google has the talent, and the money, and the experience, they just need some leadership.
I did not mean to be snarky. But still, the proposition that a railroad bankruptcy “led to world war” (direct quote from the article) 40 years later is ridiculous.
We just don’t want denunciatory rhetoric like “is where I stopped reading” or “this guy should re-read what he puts out” or “ridiculous”. If someone is wrong just point out where they’re wrong. Educate us, don’t fulminate.
He was referencing a book that made that case if you "squint". If you read that as actually a serious "this caused world war 1" statement rather than. This looks to have gotten some dominoes rolling that may have contributed to WW1 then that says more about you than it does about the article itself.
Re: "This looks to have gotten some dominoes rolling that may have contributed to WW1 " - people who believe this should read some history about beginning of WW1. I recommend starting with "The Guns of August".
Ben is wrong; demand for compute, aka revenue backlogs, is mythical and will collapse, simply because of two reasons :
1. Circular investment/spending.
2. Too much capital in the system, so returns cannot be hit regardless because the barrier is too high. (Evidence being every capital cycle in history)
I think more interesting take here would be WHEN this will happen. I don't think Ben, or anyone else, thinks we wont have some sort of correction or stabilization in supply/demand (he has said as much) But when will that occur? 2 months? 2 years? 20 years?
well its typically when it becomes clear that the private equity firms taking the risk decide they can not get the returns they need, forcing the backstoppers such as Nvidia to take that burden, and the whole ecosystem collapses.
I guess my point is that until that collapse, returns are very very very good. This can (and probably will) go on for a decent amount of time more, regardless of the inevitable things you point out.
My guess is at least another 2 years, as most people don't use AI yet, or maybe more precisely, AI is not used in the underlying workflows (which are invisible to the consumer) that make up most people's jobs.
Who knows if I am right. My original post was just pointing out that what you say is about timing, not whether it's true or not, because of course its true.
Yes, agreed, it is timing. I think a sign we are getting closer is the new equity issuances, which kind of are leveraging the current environment and the retail excitement.
Ever free newsletter and talking head spouts narratives like this free. If you want something that quantifies and gives actionable information, you must do it yourself or pay for it. What are your below $2000/month sources for good analysis?
The website from this article. It's not free. Ben Thompson releases 1 free article a week but the other 3 articles published each week requires a $15/month subscription. Ben Thompson is also very influential in Silicon Valley and the overall tech/media industry.
I know him, I subscribed for a while. Even his paid content is lacking. I'm looking more Valens Research kind of analysis. SemiAnalysis is also good in the higher tier.
ps. Being influential in Silicon Valley just means you are influential, it does not mean substantial. Leopold is still influential and gets money thrown at him at $100s of million despite having no substance.
you might be looking for SemiAnalysis? I only read the free portions of articles but they have various paid options, mostly targeting investors with information and tools.
They are the marketing wing of the AI ecosystem. Their recent article on how SpaceX would drive 500B in data-center revenue was ludicrous-mode. Lets revisit this in a few years.
so short them. if you think that the demand for skilled-labor-substitutive capital is saturable in the medium term or that improvements at the model level eat those at the hardware/cuda level or that nvidia just has the timing wrong, short them.
Nvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute are fundamentally different, and the only reason people put up with it is because Vulkan and HIP C/C++ are even worse.
Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development and sane low level API to develop against, instead you have to go through their cloud and their full software stack which greatly limits ecosystem growth. The minute that Google figures that out, that's when Nvidia's dominance would be challenged.
The CUDA runtime coming with a gazillion reasonably decent kernels (DNN, BLAS, CUTLASS) and a concurrency system (NCCL) is a big deal; especially in the “early days” very few researchers or development runtimes were even writing their own kernels or dealing with CUDA C++ extensively, they were wrapping the ones NVidia gave them.
I do agree that it’s really not great, and I also have never been a strong believer in the CUDA moat overall; as the need for GPUs moves from research to production (inference), companies are plenty willing to build software from scratch anyway (and we see this with AMD GPUs being in plenty high demand in the datacenter and enthusiast market now).
What I don't quite get is why can't they use AI to translate CUDA programs into more open architectures like AMD ROCm?
AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?
> AI is supposed have solved the
Which AI? LLMs are coding facilitators and code producers.
A problem is solved when the solution is reliable. Non-deterministic Neural Networks are not reliable. In fact,
> more mechanical[] task
that suggests an expectation of process and procedure, which is still not a capability of current architectures.
Sure, you can ask a brains-deficient operator to perform a huge task, but then you'll have to check the whole product, and that remains not cheap.
Undoubtedly, the frontier labs are already doing something akin to this as part of their new chip design endeavors.
That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more open than potential competition. Could it be that Nvidia has been on top because none of the competition has actually tried kicking them where it hurts?
I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem.
AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability.
Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.
I think there is a misconception here: CUDA is not even close to being C for assembly used by Nvidia's GPUs, PTX is, and it uses JIT to compile to Nvidia's GPU assembly, SASS. It honestly easier just to have LLMs write PTX directly than to go through CUDA C/C++ at all.
Again, "CUDA" isn't a programming language, it stands for Compute Unified Device Architecture; "C/C++ for CUDA" are the high-level languages that compiles to PTX and then SASS as well CPU orchestration code via NVCC.
And to be honest, pretty much everything you can do in CUDA C/C++, you can also do in HLSL/GLSL compiled to SPIR-V, as long as the Vulkan hardware extension is available.
My point with the c comparison is that it often feels like a very thin layer that still requires you to know a lot about what's going on under the hood. It was more to give a view in what it's like to work with the lib/api.
Maybe I wasn't being precise enough with my language for this forum, and also my last hands on experience with it was roughly 6 years ago, maybe it's gotten better. But it was much less (and forgive the imprecision!) python/pytorch-like where you say hey take this big blob of data and just slice and dice it on your many cores, and more like ok, here is the data, let's cudamemcopy it in these size chunks over to the gpu itself, to be used by this block of threads and run these commands (kernel in cudaspeak) on it. Much more painstaking and micromanagey of the resources.
Pytorch IMHO feels like a proper abstracted API that hides the details and lets you just unleash the fury at the cost of some efficiency, while the cuda api itself, similar to working with C, forces you to really think about the low level details. I have a heavy backend and systems development background, and while it wasn't really intimidating to me, it was like wow you really have to have a deep working knowledge of how these things work and it felt like a step back in time IMHO.
I doubt that's going to satisfy you but I think it gives a clearer picture of what using cuda is like if you typically use higher level languages and haven't touched C since college.
I think you are pretty precise and your comment is appreciated, so I think I need to clarify my wording as well. I think a lot of CUDA C/C++'s difficulties are self-inflicted because they conflate two things into what they call "kernels": runtime and shaders. Fundamentally, GPU programming for compute is easier than either graphics programming as well as CPU programming because you are limited on what you can do on the GPU, memory allocation is static because the matrix size doesn't really change dynamically during runtime and branching behaviors are generally to be avoided for GPU compute.
It's pretty heretical for me to say this, but a lot of GPU compute complexity that Nvidia is doing in CUDA is unnecessary and is by the simple fact that to do anything meaningful you have to either use their library or handle allocation/scheduling yourself. Imagine if JavaScript required you to handroll part of the V8/Node's JIT compiler, allocator and scheduler yourself every time you just want to make a webpage, that is essentially what CUDA is doing.
The actual "program" that runs on the GPU, the compute shaders in PTX/SPIR-V, are very low level but pretty straight forward once you get down to it.
To paraphrase the apocryphal Winston Churchill quote about democracy:
“CUDA is the worst development ecosystem in existence. Except for all the others.”
‘Yes, I Am Drunk, But CUDA is Ugly. Tomorrow I Will Be Sober, And CUDA Will Still Be Ugly.’
Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars.
Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel.
Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.
As agentic coding continues to improve, won't it get easier for devs to retrain for new languages/frameworks/etc.?
And couldn't we just ask AI to translate our program in one language/framework into another?
I was under the impression that AI was supposed to remove software-moats, let us all ask it to write our custom MS Word for us for instance?
Yet, Microsoft pulled it off with a new .NET Framework, and Apple with its new iOS SDK. There are many more examples besides those two.
Microsoft pulled it off with dotnet, sort of, because they approached it like a completely new language, sold people on the benefits of it.
And the people they were selling that to, ( It's free and open souce now! ), were a very different group to the market they left behind on .NET Framework, who are often still struggling to make the transition now.
Had they actually killed off .NET Framework, it would have been a different story, much more like the VB6/VBA to VB.NET transition, which so few people bothered with that VB.NET died out, because if you had to retrain that much, you figured you might as well go to C# or a instead, or indeed a completely different language entirely.
I briefly worked professionally on a VB.NET project, but outside that job I've never met anyone else who can say the same. I've met a few who went straight from VB6 to C# though.
The whole push behind .Net Core was to get .Net running on Linux well and natively and was being lead largely by those of us using .Net for web. We were being murdered by other languages at the time; mono was an option but Microsoft shops usually needed to target something blessed/backed by MS.
Largely the same market (Enterprise) but not the different segment (web as opposed to Windows/WinForms).
I ported about 15 years of projects from various versions of .Net to .Net Core whilst they were developing (and sent feedback to the team - they were asking us to do that) and the process was pretty reasonable. You were only really stuck if you were using something very very Windows specific (certain image processing libraries iirc) and even then it was largely manageable.
The old full-fat framework is, AFAIK, still supported, as there's a whole lot of legacy code which is Windows specific which is still expensive / hard to port over.
Unfortunately "something windows specific" was pretty broad.
Between MSMQ, WCF over named pipes, MSDTC, and MSI installers, there's a lot to replace that is hard to provide the same guarantees or performance with straight replacements, if they even exist.
The end goal, being on modern dotnet, is better, but it's difficult to get there with a phased approach without accepting a temporary worsening, which is often hard to sell.
Especially while Framework is still supported.
Both of those were a vendor X electing to stop updating framework A in favor of framework B. That’s a high cost for vendor X’s users, but if their business depends on vendor X, they have no choice. Maybe they can switch to vendor Y at that point, but now you are switching both vendors and frameworks.
Replacing CUDA with another framework has much lower motivation. That advantages of the new framework must cover the switching costs and the risk of such a switch. All while CUDA continues to evolve and allow access to additional features.
Apple and Microsoft had something of a captive userbase. New vendor on the block trying to replace CUDA does not.
Neither of those are a competitor replacing another, they are a same-vendor replacement. This is easier, because the company whose established product you are trying to displace is cooperating (because its you!) not actively resisting.
Anthropic rewrote Bun in Rust, with much help from AI of course
One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat.
[1] https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/
It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI.
The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA.
It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA.
There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task.
I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.
CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you.
Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.
ZLUDA is still alive. AMD sponsored it and the project was briefly halted during a dispute with them, but it’s been making steady progress.
It doesn’t really make sense for AMD themselves or most use cases, though; any compatibility shim just adds problems on top of problems, and for AMD, entrenching a competitors technology even more never really seemed like a great idea.
Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop.
Management likes it because it removes software developers from the loop.
The biggest advantage of tpus is the high bandwidth fiber optic interconnect between them that allows distributed computing on pods with thousands of tpus and the co-design of cooling systems that go with their racks. I do not think that we will see personal tpus any time soon.
My Pixel 9 has a personal tpu
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development
I’m not familiar with this field, but to my brain, https://www.amazon.com/s?k=Google+Coral seem to show me several such options.
... with performance greater or comparable to even the lowest performance Nvidia card?
Now with LLMs why a programming framework is a moat?
I had a hard time understanding why didn’t AMD make a better developer experience for this two years ago, and am now even more baffled that even with all the LLMs they still don’t seem to have moved a single inch, despite this probably being a tens of billions dollars worth feature.
LLMs make the dev environment almost irrelevant. llama.cpp supports AMD with both ROCm and Vulkan and that's nearly all that matters now. TBD how much AMD's AI Halo play will change things if at all, but they got a lot of positive press in launch reviews for having an actually robust software story for once.
Hardware companies are notoriously bad at software. The software they use sucks, the languages they use suck, the internal tooling software they write sucks.
They don't know what good developer experience is, how do you expect them to deliver it to other people?
I’m not entirely sure if local development will lead to Nvidia’s supremacy being challenged.
I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.
TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.
Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.
Eventually the other chips will go through enough iterations and stability will be reached
Are the switching costs of CUDA ecosystem potentially threatened because LLMs are now quite good at transcoding into other languages? In other words, is Nvidia's greatest strength (AI) also potentially its undoing?
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board
Nvidia's sells hardware yet their market cap is about the same as Google's.
How much value could Google get by selling hardware too? Google'd be selling to competitors, so difficult to capture much of the value and would decrease Google's value as an AI company. Maybe a child company?
Google sells TPUs already, to competitors :)
Genuine question. Given that LLMs are supposed to allow us to rewrite anything, and I am an LLM believer, what I don't understand is: how does CUDA continue to be a moat in a world where LLMs can rewrite entire software development stacks? If NVIDIA is right about AI, isn't this same technology going to erode the software side of this same software moat?
Indeed. Now, given that CUDA is apparently not being usurped, update your priors.
well done you realised no one knows what they are talking about
Interesting take on Google's TPUs. What I've previously heard (and still believe) is that Google's decision to only rent out, never sell, their TPUs is a deliberate and savvy strategy for bolstering GCP, which will work provided that TPUs are able to actually compete with other hardware (in practice meaning Nvidia). A few months ago there was some discussion on HN comparing them, and I think the verdict at the time was that their latest-gen TPUs win on compute-per-Joule for LLM-type workloads by quite a margin, which I think is huge for those who want to run LLMs at scale.
I mean they had/have the Coral but that's in an entirely different market segment
Coral and the TPUs are ASICs, and therefore are barely reprogrammable. It doesn't really compare to the complexity and flexibility of CUDA ALUs.
In many investment theses - like Nvidia's bet that demand for compute will keep growing - the first order assumption is usually correct. Yes, demand for more compute, chips, infrastructure is huge and each year some additional data centers will be built. Where such investment bets usually fail is in the second-order assumptions: Ie. the expectation of the growth of demand. This is where there's a high chance that the current expectations are likely exaggerated. So: demand is likely to persist for the foreseeable future but not increase every year. And that can upend the whole investment story. That can be enough to make these bonds a huge burden for Nvidia in the end. Not because people stopped buying more compute but because they stopped buying more every year.
What makes this insanely hard to predict is that the compute needed for the same quality output has roughly gone down 90% every 18 months for ~5 years.
1) We don't know how long that trend will continue, but you do know where to look for when it may end (if smaller sized models continue to compress the knowledge effectively of larger models).
2) We don't know when the appetite for higher cost models might go down and by how much if smaller models get "good enough" and price becomes far more important.
It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.
It is also entirely possible that at some size - LLMs pick up some emergent capability that doesn't scale well to smaller sizes - and that there's an incredible boost to demand to get that capability.
It's just very hard to predict.
I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side.
The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have to switch to something like burning the model weights into silicon to continue to make gains, then the current math on general purpose accelerators might be upside down.
I agree; I don't think there's any reason to assume Jevons paradox won't apply.
> If we have to switch to something like burning the model weights into silicon to continue to make gains
I think that's already being considered semi-seriously [0][1]
[0] https://taalas.com/products/
[1] https://ir.amd.com/news-events/press-releases/detail/1296/am...
What's really interesting is that if you scale it to higher densities (eg 3nm and stacked die) with ComputeInMemory for fp8 you can reasonably start to fit 30B-70B models. With MoE and multiple stacked die, just like HBM, you could fit an open weight near frontier 1T model (like GLM5.2) at similarly much lower power <10kW and high token rates >2ktps. For running a bunch of agents where fill rate and speed/latency are important it may not matter that you're 6-18mo behind on weights. The process for the chip design could be largely automated, and new silicon pumped out as new weights are available (with a 3-6mo delay).
> efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute
I buy that. Jevon's Paradox, sure.
> and we are not going to run out of economically useful things to do with it anytime soon on the demand side
This I don't buy. Not fully, at least. Whether or not there's demand for LLMs in some particular field is one thing, whether or not there is a sustainable business model to be built out of that demand is another thing entirely.
There is a staggering amount of money pouring into startups looking for novel use cases for LLM-based agents. As usual, 99% of them will fail, but those other 1% are going to have to look harder and harder to find a novel use case that can actually be served profitably.
First of all, there's only so many places where a chatbot is going to sell. But, that also seems to be the only interface anyone can come up with that allows a user to steer an agent through a long-running task reliably. I'd love to be proven wrong here.
Also, if current trends plateau and large datacenters are still needed for complex tasks, that would stimy growth of LLM usage across entire industries.
But, if present trends continue, then local inference will become feasible for most tasks. That would lower the barrier to entry across tons of heavily-regulated and/or cost-sensitive industries. But, widespread local inference will almost certainly come with a painful market correction centered around hyperscalers, which would itself dry up the pool for ventures into new markets.
Replace "chat-bot" with "Human Being" because the models I've been using are significantly better than 90% of the human-chat-bots that I must talk to on the phone while scheduling and coordinating my internet installation for example.
Now for every human replacement, that is 1 unit less of communication and bureaucratic burden (HR, middle management etc) that the org requires.
> the models I've been using are significantly better than 90% of the human-chat-bots that I must talk to on the phone while scheduling and coordinating my internet installation for example.
I guess I don't know what to say except that my experience is the polar opposite of yours.
I moved to a new state at the beginning of the year. Needed a new doctor, needed to schedule apartment tours, needed to talk to my employer about insurance and relocation stuff, etc etc. Lots of chatbots, a handful of humans. Humans consistently did what I needed them to do, the chatbots just didn't. I could list examples but I'd be typing all night.
And yknow what, my one call with Comcast to get my internet set up was downright pleasant. The rep was knowledgeable and a good conversationalist.
When efficiency reaches the point where local models on consumer hardware are good enough, demand for cloud tokens could rapidly shrink.
Very few consumers are going to spend multiple thousands of dollars to save $10 per month. Companies absolutely will to save hundreds per month per employee, but that's not consumer hardware.
Well if the trend that the comment further up in this thread claimed continues and compute requirements keep dropping exponentially then perhaps in a few years you can have today’s frontier performance on the normal laptop you already have on your desk anyway.
Many gamers already spend $1000+ on a GPU.
If you can integrate AI accelerators into consumer cards (you can), you can have local AI for "reasonably" cheap. This is Nvidia's long term goal if you listen to what Jensen has to say.
The limitation is entirely on memory right now. Just a few years ago we could of been strapping 80-100GB to cards for under $200 (BoM).
It could be quite a while before we reach that point though. 5+ years easily.
I've been keenly interested in the ability to run local models, but the hardware is just not there. Consumer RAM speeds and capacity will have to significantly increase before local models will be able to perform as well as even the lowest end GPT-5.6 Luna model.
This is on the backdrop of RAM becoming prohibitively expensive. And without the speed and quantity of RAM, it becomes impossible to generate tokens at interactive speeds, regardless of model. There is a fundamental dependency between calculating all of the active params with the given RAM speed.
Even with a model that has been quantized all the way down to Q4, the DGX/RTX Spark chip with 128GB of RAM can only generate ~18 tokens/sec for a MoE model with only 30B active parameters. There haven't been any broadly useful models below 30B active parameters. And that is for a $5000+ piece of hardware that will be one of the best for running on-device models.
I really want to buy instead of rent my AI, but the economics are truly terrible.
> think efficiency is unlikely to result in lower demand for compute
You can't save yourself rich.
It's also hard to predict how much money will be burned going down wrong avenues. The internet was the future, but it took a lot of failed companies to eventually land on a sustainable model that brought us the giants we have today.
Railways were also the future, but that didn't stop a rush to build out (often subsidized) lines that were ultimately uneconomical (either because they were corrupt or the planned settlements never arrived).
If AI is similar, then there's going to be a long slowdown on compute spend until the surplus is worked through. A good historical analogy could be the fiber optic buildouts of the late 1990s. The demand for data never really went down much, but the industry eventually commodified and took down some large companies (Nortel, especially)
I rhink there's a difference with AI because -- it brings true value because you pay for the tokens, you only pay for what you use. That is true value.
Compare to just paying for an internet connection, you have bandwidth but not sure what you can do with it that is valuable.
Let's say you use AI to produce software. There;s no limit as to how high the quality you want your software to have. And how fast you want your project to be complete. There's plenty of room for higher quality, and more performant AI. As AI becomes chepaer people will use more of it, they're not going to say "We have enough AI".
Compare to railroads. Yes you pay for the distance travelled but there's a limit to how much people wwill want to travel, how it will benefit them.
Jevons paradox should win out for a long time for AI.
When internet connections got faster than 56k modems, we didn't use the same amount of bandwidth but faster. We used more bandwidth doing things like 4k streaming. I see the same in AI inference. If AI inference is that much more efficient, it will just enable more use cases for AI.
See for example, internet traffic over time: https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS779VS...
Even after so many years, internet traffic continues to grow at an increasing rate.
Nvidia's great superpower is flexibility. You can easily run models of very different types on the same card; and their hardware is great for R&D.
However, at some point AI may be good enough for most people and then it makes sense to make an ASIC for the model (or group of models); and at that point you don't need Nvidia.
I suppose this scenario will happen in various moments at different levels.
Each of the hyperscalers has put like 250B each in the last year for infra. That means that they need to be writing AI profits to the tune of 20B per year just to keep up with the cost of the cash they burned.
We are not there. But they better figure it out soon. The cash flows dried up, and everyone is taking debt to support the capex. Google for the first time in its public history is cash flow negative. Amazon too.
Plus on top of that 1st and 2nd order can be correct, but then the price is too high, meaning people lose money even if correct about the future, but over pay for it.
They also have to be feeling the heat of the ASIC vendors. AMD just acquired Taalas and they work with Cerebras all the time on special projects. ASICs outgun nVidia's chips by an order of magnitude.
Not really. The GPU+LPU combination is going to be pretty good once it comes out.
More interesting take on Nvidia's position than I've come across before. One thing to be noted is 1) Nvidia is already making moves in robotics so even if their position in AI (moreso llms) diminished, they certainly have another big avenue arguably harder to just get into (although I'm not sure what efforts Google is doing for the tpu in robotics). Another point is Nvidia is still the main player in the west, that is, China certainly can and will create their own full stack without reliance on US companies. That puts Europe and other countries in an interesting, do you buy Nvidia because it's the only option or for security. That's to say I believe Nvidia's position relied on many different things being true at the same time, and we're moving towards an environment where those things are certainly being contested at (roughly) the same time.
I'm just hoping that some day they can get back into the relatively small market of gaming gpus, even if for just nostalgia sake.
Even in the west, Nvidia's dominance is bound to weaken. There is a notable uptick of articles on HN about people running large models on AMD hardware. And while I don't know official sales figures, I know we have trouble getting our AMD system delivered
AMD's software story is still a lot worse than Nvidia's. But patching up vllm to run one or two models you care about on AMD hardware is a much easier proposition than using them in most other fields of AI.
amd is not putting nearly enough effort to improve their software its almost suspicious
They've struggled with software even in the ATi days. It's just normal AMD behavior. Drivers will always be half baked.
nVidia will fail right after reaching an 8 trillion valuation and supplying 80% of the world's hardware!
Trust me guys, it's over!
I think this is a great example of the disconnect people have in these types of conversations.
You can both become a company that supplies 80% of the world with your type of product, and then still have your stock go down in value.
All it takes is over evaluation by the stock market. Then a course correction from unsustained growth on growth (second order). So even if you continually replace YoY 80% of the world's hardware on a rotating business, but you don't increase market share or increase demand (aka growth)... Your business looks stagnant to the stock market, and there isn't really anything you can do about it. The best you can do is track inflation +/- 2%.
And that's why a lot of older established companies were dividend stocks. You don't expect to growth anymore, but that's not where the value is anymore... The value is in the reliable sales that will happen after infinitum because your company controls a majority share of the business... And that's ok! Unfortunately, silicon valley has created a philosophy of 'you gotta expand into new fields or your on the decline' - aka neo-monopolization
Nobody is saying they’re outright failing, but that they’re not going to be printing money the way they have been recently. Think about Intel circa 2010: most of their competitors like POWER or MIPS were marginalized, they owned the desktop and server markets with a bit of competition from AMD well contained, and their biggest desktop competitor (Apple) had just switched. A lot of analyst predictions … did not match what happened next. The same was true of Cisco a decade earlier. Both companies are still there but they don’t set the terms in their market segments.
I’m not predicting Nvidia will MBA themselves to death in the near future but I think there’s a tendency to overstate how profitable companies will stay. The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
The current administration is definitely corrupt enough that you could imagine an anti-competitive deal of some sort but I don’t think there’s a way for even that to change matters because key competitors are well-connected American companies willing to play that game, too.
> The more money Nvidia makes, the more motivated their competitors will be to get a piece of that market and the more customers will be looking for alternatives like the push into TPUs which the article discussed.
Your margin is my opportunity - Jeff Bezos
But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.
The root comment in this thread was about Nvidia hedging their bet on lost AI market share. They recognize that a reduced pace in training and inference competition will undercut their business, but CUDA isn't a one-trick pony for LLMs alone. TPUs are - you can't even reuse the same architecture for training and inference, they're separate ASICs unlike CUDA cores/ALUs. Veterans of crypto mining will tell you that the ASICs lost in the end, as Nvidia was evolving their hardware faster than the ASIC manufacturers could iterate. When the crypto acceleration landscape diversified away from ETH/BTC into altcoins, Nvidia was still there making money hand-over-fist from mining hardware.
I guess you could argue that robotics, world models or computer vision won't be a trillion-dollar market. But Nvidia is positioned to be the first mover in all of these markets, and none of their competitors are even coming close to the integrated stack that they sell consumers.
> But people have been saying this about CUDA for twenty years, and we are not any closer to a replacement GPGPU paradigm today.
How much money was in it for the first decade or so? I think AMD was asleep at the switch but e.g. Apple just did their own thing for the parts which they prioritized.
My understanding is also that Anthropic and OpenAI have also worked to decouple themselves so I think it’s likely that the CUDA moat is going to be less of a barrier than it used to be from the perspective of guaranteeing Nvidia profits.
Money wasn't really the problem. Apple pulled OpenCL together pro-bono, and worked with Khronos to find willing industry stakeholders that would oppose Nvidia. OpenCL needed hardware standardization though, and nobody wanted to design or implement on a scalable GPGPU architecture like CUDA had. AMD and Apple both bet big on raster efficiency, which turned out to be a terrible play when Nvidia was already putting dedicated ray tracing and tensor hardware into their GPUs. They both bet the farm against each other, and only Nvidia won.
Once Apple fully left Khronos, AMD played the smartest card they had; they architecturally split RDNA and CDNA into separate product lines, so they could optimize them independently. This staunched the bleeding, and gave AMD a datacenter presence that Apple Silicon could only dream of. Still not a scalable architecture, but better than nothing.
> you can't even reuse the same architecture for training and inference
AWS begs to differ. They originally split between `Trainium` and `Inferentia` but now support both with `Trainium`
I stand corrected, only Google's post-Ironwood TPUs have the split as well.
Nonetheless, TPU architectures are still a systolic array, and have their own limitations for scalability and flexibility. CUDA is no silver bullet, but it satisfies the demands of the edge and research customers very well.
Google has their robotics models, for example:
https://deepmind.google/models/gemini-robotics/
Google is mostly the party behind the whole VLA principle.
For awhile I've found two things hard to square, that the hardware and software making up current gen AI will bring us to a socioeconomic singularity, and the reality the thing they're mostly trying to emulate is a few pounds of meat and fat running on tens of watts equivalent. On one hand the current AIs are obviously super human in some tasks, get completely dunked on in others by far simpler organisms. My cat can catch a bug out of the air, Fable 5 in Cowork can lack the dexterity to make a slideshow because I had LibreOffice instead of Microsoft Office. Not even close to analogous, but point being they appear to have pretty fundamental differences in how they can interface with the world that the economic thesis seems to gloss over.
Another interesting discrepancy is that people think current GPUs are maybe capable of running AGI but they still can barely manage photorealistic rendering of a single room in realtime, or simulate something like a shirt thrown into a pile of laundry. They can generate a video of it based on millions of existing videos, but not do a real simulation of light and physics in realtime.
Is the human brain capable of doing real simulations of light and physics in realtime? Or does it hallucinate the details and represent some low-resolution mental image? You may be overestimating the capabilities of flesh-based neural networks and underestimating the capabilities of silicon-based neural networks.
An AGI doesn't need that level of detail to do most tasks effectively. To use the OP's example, a cat catching a bug out of the air does not need to run a fluid dynamics simulation of airflow over the bug's wings to be able to catch it. A cheap approximation of the flight path is sufficient. Perhaps some physical tasks will need that level of detail but many will not.
> a few pounds of … running on tens of watts equivalent.
Aren’t you already describing a laptop computer? Add a local LLM and you’re fully there
You make a good point. Maybe the AI apocalypse will actually be when someone hooks up a cat brain to a super advanced AI.
No need. The 'apocalypse' will come purely out of the hubris and greed chasing untold fortunes as opposed to a healthy, considered integration.
The decision to scare the shit out of normal people early on was a short-term decision that will cost everyone dearly (it led non-technical people to believe in things about LLMs that just aren't true—even worse, they've now made difficult business decisions on bad information with seemingly limited consideration of secondary effects).
The amount of short-sightedness alone should give pause, but at least in America, we seem to have collectively lost our minds in worship of the almighty dollar.
Judgment day approaches.
Nvidia has been playing a dangerous but profitable game since the Crypto boom.
but now I think they probably have bitten more than they can chew.
Apple already proved with their unified memory - that as long you have the capacity you can run capable models locally - thereby goes demand for inference if everyone is running some model locally.
For training - Chinese models have proved that you don't need the latest & greatest in Nvidia hardware. Same as TPUs.
only time will tell.
It seems they had a head start but are now facing stiff competition on all fronts. Software moat, GPU's for gaming, and its distant cousin datacenter compute. They rightfully invested their insane profits into many ventures, and how many of those have turned around into profit?
They are also a robotics AI company with Omniverse. They are also an AI company with Nemotron. They are also a bleeding edge network equipment company after the Mellanox aquisition.
They stand to make a lot of money if they succeed in every venture. Good for Jensen taking risks and driving innovation, I hope they succeed in chewing even 50% of what they bit off.
> They rightfully invested their insane profits into many ventures
This is the inherent downside of such a rapid rise to being the world's most valuable company AND still being considered a growth stock. At their massive scale, the number of new adjacent businesses that have both sufficient size and potential growth is limited.
> I hope they succeed in chewing even 50% of what they bit off.
Anything approaching that is vanishingly unlikely. They're being forced to play the game more like a VC. The question is if a few unicorn winners can offset dozens of losers. The challenge is that, unlike a VC, their bets are much more correlated around AI.
Nvidia sell iot boards with unified architecture. Would not be shocked if they launch pc/laptop/server boards at some point.
Unified memory DGX Spark and RTX Spark laptops are already a thing :)
Right, it's clear that nVidia is taking care and trying to position itself so that it can continue making sales if and when inference goes local. And it's in a much better intrinsic position to do that than the LLM SaaS vendors are: nVidia sells shovels to the army, but it also knows how to sell shovels to Walmart. Whether the financial relationships that Huang's got his company into will cause it problems if the market shifts is a different question, though.
AFAIK NVIDIA unified memory is not as... "unified" as Apple's
As is AMD Strix Point
RTX Spark already launched.
that undercuts their core business, so it will be a defensive play at most to fend off mac and amd's local inference offerings
I don't know how people can say this with a straight face. Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced, specifically for edge robotics, computer vision and ML.
The absolute fastest desktop Mac GPUs cannot beat an Nvidia laptop GPU in prefill or inference speeds. Apple Silicon is a non-entity for professional datacenter deployment and arguably unusable for frontier models at agentic context sizes. AMD is Nvidia's primary worry, and they're not doing much better in terms of GPGPU SOC compute.
>Nvidia was selling desktop-grade ARM SOCs before Apple Silicon was ever announced
You can believe all you want that the dinky little jetson boards were desktop grade when historically the ARM SoC portion of a jetson board couldn't even keep up with broadcom/rockchip SoCs. It's taken until recently for the actual arm compute portion of Nvidia SoC's to be worth a damn at all, and they still fall far behind Apple let alone the rest of the pack like Qualcomm/Samsung.
I don't have to believe. I've run KDE and GNOME on the Tegra boards, you get full-fat CUDA support without sacrificing Vulkan drivers. It's incredible.
You can believe all you want that good single-core performance will corner the edge compute market. It hasn't, Graviton has more buy-in than any Apple Silicon chip ever got.
Every time nvidia takes its fab time and uses it to build anything other than datacenter chips it is losing money due to the massive markups the datacenter products have. Expanding their consumer offering means the datacenter backlog is going down which is very bad for their margins. Consumers will never pay 10-100x what it costs to fab something like datacenter users will.
In many cases it isn't Nvidia paying for the fab time. For instance, the Nintendo Switch 2 is basically a pure-play design and support product for Nvidia, while Nintendo negotiates with Samsung for the actual SOC prices. Their IP philosophy is closer to AMD's than Apple's, Nvidia has long roped in 3rd party manufacturers to mark up, integrate and sell their hardware.
I'd be happy enough if Nvidia decided to launch an update to the now dated (2019) Shield streaming box, which is still the best on the market as to codec support and image quality.
Yes. The more capable SoC PCs become, the weaker the "everything has to run in the datacenter" argument gets. As more powerful SoC PCs from multiple vendors appear over the next few years, that advantage may shrink for many workloads.
What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price?
> What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price?
Nvidia dropping from being a $5 trillion company to a $242 billion company would be 1929 levels of bad.
Global economy end of days stuff, especially since Nvidia can't crash that hard without a lot of other stuff crashing with it.
How much in value could Nvidia safely drop and over what period of time for it be fine for greater economy? What level of correction would be manageable?
And I am pretty sure that their value will drop in 5 to 10 years.
Just for context, they make up 5% (yes 5%) of holdings in global index funds like VWRA.
https://stockanalysis.com/quote/lon/VWRA/holdings/
That is an astounding figure and means markets are highly imbalanced, unless you think Nvidia represents 5% of the global economy.
4.7% of the discounted future earnings pie of 3761 of the largest current publicly traded global companies.
That’s still a lot, but “the global economy” over the horizon where those earnings are not discounted to 0 contains a lot more than what’s listed in VWRA today.
There is a number of index funds holding 7.5% of NVIDIA.
This consolidation is scary. I've started putting a little bit in emerging markets and China. I cannot trust the US long term, hopefully I'm wrong!
I see China as very risky due to Taiwan.
Nvida's action may affect the semi- marketing
IMO focusing on the hyperscalers is kind of misleading.
Yes, for programmers and tech companies AI is kinda boring now, but AI integration in general is still kind of uncharted territory.
There are so many small companies and individuals just getting started with AI today and I believe a large the customer base (and revenue) is still untapped. Hell, I’m discovering new use cases regularly still and the average mismanaged 30 people whatever SaaS vendor probably didn’t even get started yet.
This is what so many people on HN and the market are constantly missing.
Jason Kottke almost didn't found his blog in 1998, famously quoted as saying: "I thought I was too late, that no one would be interested." Needless to say, the internet was a tiny joke in 1998 compared to what it is now.
We are just barely scratching the surface of what's possible with AI, both in terms of the leading edge and in the 'torso' of the economy (the portion you're describing).
Folks from Silicon Valley working in AI-forward companies have a skewed perception of how many people have adopted this technology so far. Codex recently celebrated hitting 10 million users. This is a great milestone and all, but to put it in context, Microsoft office has a billion users. Sure, many people use Claude Code and or some other harness and the growth is staggering, but the overall scale is tiny compared to software as a whole. Costs of serving and usage are still very high, prohibitively so for many, so we aren't even close to market saturation.
And even at the leading edge, people who do work in those AI-forward companies; models are still slow, require hand holding, and produce suboptimal outcomes sometimes. Imagine the value when instead of needing to prompt it once per 30 mins, you prompt it once per day. Then once per week. Then once per month. Imagine all this running not on 3 trillion parameter models, not on 10 trillion, but 100 trillion. What kind of computer infra will be needed then? Certainly more than we have today.
I've always marveled at how one can pick any year since the internet went mainstream and in that year people thought, "Oh my goodness, the internet is amazing!"
Then, move five years forward from that year and look back. In every case, people think, "Hah! The internet was so simple then!"
In 2031, I suspect we'll say the same about 2026.
Yes and no...
So much of the dotcom bust was essentially: "anything internet will work", but even after it was pretty easy to find different ways to publish web pages or migrate to new forms of social media, video, short-video, etc.
It's orders of magnitude harder to automate work, which is the business value proposition of AI (whether you are displacing worker or adding breakthrough capacity), and it's not clear the LLM hyperscalers will get the value-add from that.
On the consumer side, it might be a race to the bottom: same ad profits, but now you need AI to produce it.
I wonder why Google doesn't create an open-source CUDA alternative. Google released Kubernetes to stay relevant/competitive in the cloud wars, they were a distant third. They now have an opportunity to create an open source industry standard.
Or the companies spending trillions of dollars can do a Manhattan Project (Or X-Prize) and let a thousand startups work on it. One will succeed. Between Google, Amazon, FB, Microsoft, Apple, AMD, Qualcomm, Intel (and dozens of other companies) there is enough economic incentive to do it. Also, isn't this what AI is supposed to be extremely good at, CUDA experts can continue to write CUDA (without having to learn anything new), a translation layer will rewrite it. If software can be one-shot from markdown files, this can't be impossible.
I see we have reached the stock market phase of Universal Paperclips.
Honestly, that game with a few changes would be very on the nose today :D
Phew. I thought we were just about to leave it, albeit not quite getting to space, just dissolving the financial system so our AI overlords can make some more harvester drones.
Tend to agree with Ben's thesis RE Demis and DeepMind not really being focused on the agentic coding race. That being said, it remains to be seen whether Sergey and Koray can inspire the foot soldiers in the same way that Sama and Dario do. I'm not too optimistic, and that's to say nothing of the fact that Google cannot possibly hope to compete with these other companies on potential employee upside.
> Google cannot possibly hope to compete with these other companies...
It's possible Google has intentionally decided to take a more conservative blended approach than purely competing at the bleeding edge of the frontier. If so, they obviously have no incentive to state it publicly but the recent departures and financials are consistent with the idea. It also makes sense that a company so much bigger, longer-term and (somewhat) more diversified than pure-play frontier labs would play the game to align with their strengths (capital, balance sheet, breadth, etc).
In their position, why not take an 'arms supplier' strategy in the near-term while drafting behind the frontier labs as a fast follower in AI, essentially betting the AI race is more akin to the Indy 500 than a quarter-mile drag race. If they're wrong and it IS more like a drag race, Google is in a better position to absorb and adjust than a frontier lab, for whom current valuations and capex spending requirements nearly require this to be a relatively short, winner-take-all race.
I think the linked semi-analysis piece is about right. Google is more focused on protecting its 4.2 trillion dollar golden goose than pushing cutting edge which leads to a bureaucratic nightmare that researchers simply dont need to engage with when openAI/anthropic are offering even more money. Why waste time dealing with bureaucracy at google when you can be top dog at openAI or anthropic?
Laughable to cite and reiterate the idea that Google is cooked where it comes to SoTA and AI in general when they operate, reliably and successfully for decades, one of the largest computing infrastructures on Earth and will likely continue to usefully serve the 90% of AI requests that don’t involve managing large codebases. Also seems strange to suggest that Google would need to Aquihire a company like Thinking Machines when it could spin up their AI model in a couple of weeks on its own TPUs if it felt like it. Demis likely wants to focus on his specific interest at the junction of biochemistry, neurobiology and computation which is more specific and unique to Demis than building a general purpose Q&A search model.
> To translate such figures into comparable 2026 magnitudes, multiply by a factor of 1,200.
Perhaps this has something to do with the economic dislocations and world wars between the 1870s and today?
There's another factor which Ben failed to consider. Which is that NVIDIA doesn't need to rely on demand for their proprietary CUDA stack or their GPUs growing -- they are already selling directly to the consumer, and likely capturing much higher margins. They are moving up stack, not down, where demand for raw compute matters less. With the DGX Spark and Jensen’s statement about “open models”, their next product is likely a strong hint: consumer devices to fulfill the Mac Mini demand craze. They are probably going to start burning LLMs durectly onto sillicon and then selling DeepSeek-in-your-home to individual developers. I bet that would sell even better than Anthropic Max coding plans and is not dependent on hyperscaler funded boom-bust cycles. So Ben’s analysis highlights the risk of their existing business not growing but they are likely planning new businesses.
Nothing goes up and to the right forever. Nothing.
Building a business model on the belief that “this time is different” always finds storms on the horizon.
A lot of things go up forever, as long as you denominate them in an inflationary currency ;-)
There’s a massive difference between “going up forever” and “point B is higher than point A.”
The current setup can’t sustain a downturn, even if yes 20 years from now point B is likely to be higher than present.
That’s the danger. Those that are going to get wiped out by the AI bubble burst aren’t wrong about AI being huge long term, they just put themselves in a position to not survive the storms that happen between points A and B.
This is the best articulation of this problem/paradox I've read yet.
What if we build our whole economy on that belief?
I guess we'll find out soon enough.
Or at least our public pension systems.
Well, they certainly can't go left.
People and pundits have been dooming and bearing on Nvidia for as long as it's been around. It increased in intensity when gpus were used for large scale crypto mining and it became material to their operations, and continued as AI ("the bubble") started to really take off.
Over those years, my NVDA stock has been by far my biggest winner. I'm now up more than 1500% on it.
Let the dooming continue
> The subsequent bankruptcy of Jay Cooke & Company triggered the Panic of 1873, culminating in endless railroad bankruptcies across the country, a multi-year depression, multi-decade deflation, and, one could argue, the financial conditions that made Europe, four decades later, into a tinder box.
American history education needs some dire reform.
> After the departure of DeepMind CEO Demis Hassabis (technically promoted to chairman, but no longer in charge of day-to-day operations) and Gemini co-lead and former Chief Scientist Jeff Dean, along with a host of other prominent researchers, SemiAnalysis declared that Gemini is Cooked: "For all intents and purposes, we believe DeepMind is no longer a frontier lab"
Counterpoint: xAI pooped out a frontier model based on nothing but capital and one man's desire to push a right-wing political narrative. Google has the talent, and the money, and the experience, they just need some leadership.
This is fine. everything is on fire it’s not a bubble. ;-)
Is this just an ad for a new book about trains?
Disappointed by the lack of Tom Cruise.
[flagged]
Please don't post grouchy comments like this on HN. The guidelines make it clear we're trying for something better here. https://news.ycombinator.com/newsguidelines.html
I did not mean to be snarky. But still, the proposition that a railroad bankruptcy “led to world war” (direct quote from the article) 40 years later is ridiculous.
We just don’t want denunciatory rhetoric like “is where I stopped reading” or “this guy should re-read what he puts out” or “ridiculous”. If someone is wrong just point out where they’re wrong. Educate us, don’t fulminate.
He was referencing a book that made that case if you "squint". If you read that as actually a serious "this caused world war 1" statement rather than. This looks to have gotten some dominoes rolling that may have contributed to WW1 then that says more about you than it does about the article itself.
Re: "This looks to have gotten some dominoes rolling that may have contributed to WW1 " - people who believe this should read some history about beginning of WW1. I recommend starting with "The Guns of August".
> Blaming some railroad bankruptcy for starting WW1 is where I stopped reading.
What a weird reason to stop reading an article.
I am assuming the rest of article is as well researched as the, rather long and incredibly incorrect, introduction.
Bayesian inference and all....
Ben is wrong; demand for compute, aka revenue backlogs, is mythical and will collapse, simply because of two reasons :
1. Circular investment/spending.
2. Too much capital in the system, so returns cannot be hit regardless because the barrier is too high. (Evidence being every capital cycle in history)
I think more interesting take here would be WHEN this will happen. I don't think Ben, or anyone else, thinks we wont have some sort of correction or stabilization in supply/demand (he has said as much) But when will that occur? 2 months? 2 years? 20 years?
well its typically when it becomes clear that the private equity firms taking the risk decide they can not get the returns they need, forcing the backstoppers such as Nvidia to take that burden, and the whole ecosystem collapses.
I guess my point is that until that collapse, returns are very very very good. This can (and probably will) go on for a decent amount of time more, regardless of the inevitable things you point out.
My guess is at least another 2 years, as most people don't use AI yet, or maybe more precisely, AI is not used in the underlying workflows (which are invisible to the consumer) that make up most people's jobs.
Who knows if I am right. My original post was just pointing out that what you say is about timing, not whether it's true or not, because of course its true.
Yes, agreed, it is timing. I think a sign we are getting closer is the new equity issuances, which kind of are leveraging the current environment and the retail excitement.
Ever free newsletter and talking head spouts narratives like this free. If you want something that quantifies and gives actionable information, you must do it yourself or pay for it. What are your below $2000/month sources for good analysis?
If you have to ask you're not going to benefit from it.
The website from this article. It's not free. Ben Thompson releases 1 free article a week but the other 3 articles published each week requires a $15/month subscription. Ben Thompson is also very influential in Silicon Valley and the overall tech/media industry.
I know him, I subscribed for a while. Even his paid content is lacking. I'm looking more Valens Research kind of analysis. SemiAnalysis is also good in the higher tier.
ps. Being influential in Silicon Valley just means you are influential, it does not mean substantial. Leopold is still influential and gets money thrown at him at $100s of million despite having no substance.
Criticism with no justification behind it is cheap.
irony
you might be looking for SemiAnalysis? I only read the free portions of articles but they have various paid options, mostly targeting investors with information and tools.
https://semianalysis.com/
To me they come across as overzealous peddlers and hacks on some market pumping mission.
> overzealous peddlers and hacks
They are the marketing wing of the AI ecosystem. Their recent article on how SpaceX would drive 500B in data-center revenue was ludicrous-mode. Lets revisit this in a few years.
so short them. if you think that the demand for skilled-labor-substitutive capital is saturable in the medium term or that improvements at the model level eat those at the hardware/cuda level or that nvidia just has the timing wrong, short them.