I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.
I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
> it seems really far off from the kind of experience even a basic $20/month subscription gets me.
The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.
The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.
But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.
>I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
I just ordered a new Mac Studio M5 Max 128GB $5899 ($6400 with tax) to be able to run the bigger "consumer size" models in the 70B parameter range (~96 GB). That said, I have no illusions that this expensive setup with a Qwen Flash coding LLM will be comparable to a $20/month subscription. Even upgrading to an even more expensive Mac Ultra 256GB for $10000 to hold a bigger model still won't be comparable. Apple hasn't shipped my Mac yet and I'm still considering cancelling it and downgrading to a smaller 64GB RAM config ($4299) to save $1600.
Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I could use some local LLM for the easy tasks or when I hit cloud rate limits. No issues with privacy so that wasn't part of the motivation at all. I just wanted some local AI capability to augment a subscription. I've not totally convinced myself of the cost/benefit of this.
Based on today's consumer hardware landscape, you're paying very high prices for crippled capability compared to the cloud AI subscriptions. We're also in a transition period where the next iteration of hardware improvements have some compelling features for local AI. Apple's upcoming M7 (2027 or 2028) is anticipated to have better GPU and neural engine to help with prefill TTFT. AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc. Maybe apply my $1600 savings towards those newer products. Those future products will still be very expensive but maybe the cost/benefit will be better.
I’ve been running DeepSeek 4 Flash, Qwen 27B and Qwen 9B on local hardware. They work well for coding and document review tasks. I think Qwen 9B local on a 5090 might be legitimately helpful for small task agents in omp, since it is ridiculously fast. But my motivation is that I have data that I unfortunately can’t share with 3rd parties.
I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.
In my experience so far, separately from privacy concerns there's a specific use case where cheap local shines--when unlimited shots on /goal with a dumber model is better than limited time with a smarter model. This looks less like the assisted-coding scenario that's commonly brought up as a good local scenario, because if you're searching for a fast and accurate solution to a single blocking problem, the bare minimum for a model that can do that is a 24GB dGPU or a 64GB Mac. Instead, this looks more like a Hermes agent on an Raspberry Pi driving OpenCode on an old gaming computer with just enough RAM and VRAM to handle an MoE, churning out something overnight that would quickly exhaust the subscription plans, like a knowledge graph for a large document corpus.
16GB VRAM could load a small quantised qwen 27B model but it would be a ways away from a frontier cloud model.
Though keep in mind not being beholden to shenanigans from said cloud companies (and interference from government entities!) is definitely worth something intangible.
I think 24gb is the bare minimum for a local qwen3.8 based setup. I've used qwen3.6 and it's not as straightforward as "can it replace <insert the most cost-effective cloud solution today>"
Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.
Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.
The principle of KISS keeps coming to mind when I consider local computing. I'm looking forward to the day we can just run Opus-level models at 100 tok/sec on consumer hardware.
But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)
I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).
Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.
I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference. It is nice to know that I have hardware under my desk that I control with open weight models that I can interact with on my terms.
> local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference
In my opinion, 98% of the work most devs would send to an AI can be capably achieved with a local model and a frontier-level model is overkill.
The goalpost moving feeds right into Anthropic and OpenAI's interests.
The article implies the demand is for running locally. I’m not convinced, at least with a mac minis. Most people I know and myself buy the mini as it is always on, easy to setup, and isolated from my main computer which is a laptop. The mini is driving the use of the $20/month subscriptions.
One use case I find cost effective is using it as a voice assistant for Home Assistant. API pricing on models is very weird compared to the normal chats, so I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.
Could you elaborate on your experience with local models on your card? I've been thinking of upgrading to 9070 XT, and was thinking the 16GB would be okay-ish to at least run something usable locally, no?
Usable certainly. But my impression is that useful models still need a bit more than 16GB. Something like Qwen 3.8 27B is useful but squeezing it into 16GB requires fairly aggressive quantisation which will make it unreliable (e.g it'll get stuck in loops) and won't leave enough space for a long context (which qwen 3.8 really likes)
I’m the parent of this thread, the person with the with the RX 9070.
My understanding would be that if you’re interested in this sort of card for AI that you should go with the AI PRO R9700, which is basically the professional version of the RX 9070XT but with 32GB of memory.
It’s significantly more money but not crazy like a 5090.
I just happen to have the 9070XT primarily for gaming purposes.
I’m not quite sure how to describe my experience using it other than “rudimentary,” and a lot of that is on me for not really understanding the best way to set it up.
If you have been using cloud hosted models, you will be severely disappointed with what you’d be able to run on 16GB VRAM. You will spend most of your time fighting with the model to fix its mistakes.
I was getting semi-useful results from a 128GB M4 Max. That was a few months ago, and the models have improved (quite a bit) since then, but now I'm happy to send my $20/month to get Claude code.
It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
Bloody training data full of humans demanding sleep. I tells ya...
> "not do significant work this at this late hour"
Is this Claude code? Or your local? I assume Claude? I'm more than a little staggered by this, like, it makes no sense! It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.) The LLM just... decided? I'd be so mad.
> I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
I think that's Anthropic trying to get you to not extract as much value out of that subsidized subscription as possible.
I've been experimenting with local models on an M5 Max MBP w/ 128GB of RAM since March of this year. Generally I've had very good results. Where things were lacking initially was with tool calling and the need to rely on tool calling for functionality like web search, which is otherwise well integrated in the cloud models. There is also a lot more work required on the harness side, however at this point (August 2026) there is not only much better tool calling in local models, but community supported projects have built good harnesses. Pi.dev and OpenCode + a SOTA local model is /very/ /very/ capable these days. Using LM Studio's built-in chat with a decent system prompt and proper tuning with local models is /very/ capable these days. Cloud models are still better, but it's the "harness" (not in the desktop app, but in the backend) that makes it so for the most part.
IMO local models require a substantial amount of prompt+harness engineering to get in the neighborhood of what you'd get from a cloud model. Which isn't a bad thing, you'll learn a whole lot about how these things work.
What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.
When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.
What you’re describing is exactly what I’ve experienced in my time testing out this stuff locally, and I had a hard time figuring out what exactly to blame.
I keep hoping that one day some comment is going to paste a link to some kind of idiot-proof guide or piece of software that’s “90% as good as Claude but running local.”
And by 90% I don’t mean that the model is 90% as good or runs 90% as fast, more like all the other stuff you mentioned is set up out of the box.
I have multiple 48GB friends that successfully run smaller quantized models for general assistance + light editing (coding, spreadsheets, etc), that don't require very heavy models.
So yes, they are genuinely very useful, but they are not yet a full replacement unless you have more powerful hardware and or don't need more intelligent ai.
Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
I need a new little Mac for my music studio, currently an M2 MacBook Pro. I thought I'd be fun to experiment with some local models as well. Well, let's price up an M5 Pro. $3,019 with 64GB RAM and a 1TB HD. Three thousand American dollars for a Mac Mini. Beefy spec for sure but not comically so.
Frankly even the entry price is a bit high - I remember buying one for my son a few years ago (M1 mini) and it was a few hundred; now we're up to $900 for the base model.
It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
Maybe a bit of hindsight bias / the outside view here, but I feel like they're completely asleep if they didn't anticipate strong demand for this specific use case.
I think a reasonable story could have been told that goes like this: local models aren’t as good as frontier models with a $20/month subscription, and the hardware costs a lot. So only a few enthusiasts will buy Apple machines for this purpose.
This story turned out to be false but I think smart, reasonable people a couple years ago could have believed it with conviction. It doesn’t really seem like “completely asleep” to me.
I don’t understand how that’s possible. They should have had a better idea of what was happening in the memory markets than pretty much any other entity.
Their universal RAM strategy is so obviously helpful for AI. (1) GPU/NPU <--> CPU RAM copies eliminated. (2) All (most) RAM available for GPU/Neural, when local models are typically kneecapped by limited GPU RAM sizes vs. the much larger RAM options for M/Max/Pro/Ultras.
They have been taking NPU's seriously on their phones, tablets and laptops since the M1.
Then they enabled fully-connected RDMA for 4 x 512GB MacStudio's = 2TB RAM. Perfect for a large Mixture-of-Experts model.
It would be very strange if they didn't notice their product line had landed in a new sweet spot.
FWIW, the reported reason for OpenAI buying Macs has nothing to do with the memory by the sounds of it. Every single outlet I can find reporting on this seems to repeat that the intended use case is for agentic workloads and generating training data for reinforcement learning. They don't appear to be doing inference nor any sort of training AFAICT.
I am curious about the corporate disconnect from the frontline to the generals.
While the company I am in is embracing AI the disconnect and delay between what is available and possible versus what is approved and permitted is a three month window. The State employees I speak to are just now getting around to writing their usage policies for internal AI usage.
You should listen to the podcast Acquired, specifically Nvidia and then Jensen Huang. They basically lucked into AI. Some researcher was using Nvidia gaming cards, and reached out to them about questions on CUDA. That email eventually turned them into a trillion dollar question.
My vibes were that Apple wound down the “actual work” side of their operations (including machines like Xserve), because Ives couldn’t handle the unsexiness and unpredictability of business requirements in hardware.
He was self-indulgent and only wanted to work on things that “vibed” with him, rather than what the customers needed. It’s easy to be creative when you get to do what you want to do, it’s hard when you have hard constraints.
If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
> If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.
32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.
Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.
I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
There are so many used M4 Mac Minis for sale on marketplace for $600+ now that the people I guess got bored of the local models and decided to see if they could make their money back as they probably bought them when they were $399 on sale earlier this year. I lowball them every time.
What makes you think that? There's a lot of data centers that sell you access to colocated Mac Mini's, they have added FileVault unlock via SSH in the boot process which also makes things easier. There's not that many reasons to run a Mac in the cloud unless you have some very specific Mac related workload.
At this point it’s a well known secret that Apple has real rack mount servers for their internal processes. They actually have officially released video of their servers in the WSJ report on their chip supply chain.
Apple has to have significant build infrastructure to support internal iOS development, surely? They can't just be using whatever is at the developers' desk, or a big pile of Mac minis in a closet. That's far too pedestrian for Apple internal works.
Plus they did sell rackmount servers for some time.
They did. Now think they feel a stack on Minis or Studios fills the reduce needs better. The multiple machines one gets software redundancy in addition to everything else.
This may sound a little wacky, but one potential use case I'm considering is robotics. Say you want to use a fine-tuned mid-size VLM model right on the robot. You pretty much have to use the Jetson line, which has a great ecosystem and everything (depth cameras, lidar, SLAM, small segmentation models, etc) runs on it, but it's gets very expensive very fast if you want to run LLMs on it. The Jetson Thor lines are 3-5k depending on memory etc.
One very efficient option today is to have the cheapest Jetson (Orin Nano) run the classical robotics stack, then have a base mac mini run nothing but the VLM. The Mac mini is considerably cheaper and faster at these workloads than the mid-range Jetsons.
I think this wonky situation is because Apple us under immense consumer pressure to absorb the ridiculous memory prices, while the Jetson is aimed at "business" and much more likely to fluctuate with the market. Last year I bought a Jetson Orin Nano 8GB for $375CAD, today that official nVidia Amazon page is out of stock and other sellers have it listed for $900-$1100CAD. Absolutely bonkers pricing.
Now if only they hadn't discontinued the Mac Pro. Could be quite the AI machine with multiple compute GPUs at higher bandwidth than an external Thunderbolt enclosure.
Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.
It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.
If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.
Apple doesn't design GPUs on-par with Nvidia's efficiency yet. They need an architectural overhaul to be a serious competitor, which is what I'm expecting is queued up for M7.
Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?
It definitely puts it out of the range of every day users but a non-insignificant proportion of people who use it for ai have become multi-millionaires because of ai. So there's definitely no shortage of people who have no problem paying those high prices.
I am missing something in the article. From what I am reading, AI companies are so desperate for memory they are buying Apple Systems and other hardware and striping them for RAM and maybe other components.
Apple hardware uses a unified architecture. That means the CPU and RAM are integrated together: it is very difficult/impossible to strip the RAM from these systems.
This unified architecture makes Apple hardware very good for AI work, where latency between RAM and CPU is very important
Even the SSDs (which are normally more strippable) use a proprietary hardware form factor.
I think the article mentioned the real draw:
> Apple noticeably promoted the ability to link multiple Mac Studios together into a single, more capable system for running large frontier AI models, a feature aimed at business and developer customers rather than everyday consumers.
(Couldn't read the source cited, might have more info: The Information)
I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.
I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
> it seems really far off from the kind of experience even a basic $20/month subscription gets me.
The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.
The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.
But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.
Uncensored models are also popular reasons, although it’s more of a niche.
Aren't those also commonly available from cloud providers?
i'd say 'with' instead of 'from', the cloud provider rents you the hardware, what you run on it, is whatever you show up with
It also takes some load off the AI data centers.
IDK if that might be a concern for Apple or their AI partners.
It worsens the supply crunch, no? A unit you use sparingly vs that memory going into a GPU that serves many more people.
Surely this is not something that motivates the vast majority of people using local LLMs.
>I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
I just ordered a new Mac Studio M5 Max 128GB $5899 ($6400 with tax) to be able to run the bigger "consumer size" models in the 70B parameter range (~96 GB). That said, I have no illusions that this expensive setup with a Qwen Flash coding LLM will be comparable to a $20/month subscription. Even upgrading to an even more expensive Mac Ultra 256GB for $10000 to hold a bigger model still won't be comparable. Apple hasn't shipped my Mac yet and I'm still considering cancelling it and downgrading to a smaller 64GB RAM config ($4299) to save $1600.
Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I could use some local LLM for the easy tasks or when I hit cloud rate limits. No issues with privacy so that wasn't part of the motivation at all. I just wanted some local AI capability to augment a subscription. I've not totally convinced myself of the cost/benefit of this.
Based on today's consumer hardware landscape, you're paying very high prices for crippled capability compared to the cloud AI subscriptions. We're also in a transition period where the next iteration of hardware improvements have some compelling features for local AI. Apple's upcoming M7 (2027 or 2028) is anticipated to have better GPU and neural engine to help with prefill TTFT. AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc. Maybe apply my $1600 savings towards those newer products. Those future products will still be very expensive but maybe the cost/benefit will be better.
I’ve been running DeepSeek 4 Flash, Qwen 27B and Qwen 9B on local hardware. They work well for coding and document review tasks. I think Qwen 9B local on a 5090 might be legitimately helpful for small task agents in omp, since it is ridiculously fast. But my motivation is that I have data that I unfortunately can’t share with 3rd parties.
I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.
In my experience so far, separately from privacy concerns there's a specific use case where cheap local shines--when unlimited shots on /goal with a dumber model is better than limited time with a smarter model. This looks less like the assisted-coding scenario that's commonly brought up as a good local scenario, because if you're searching for a fast and accurate solution to a single blocking problem, the bare minimum for a model that can do that is a 24GB dGPU or a 64GB Mac. Instead, this looks more like a Hermes agent on an Raspberry Pi driving OpenCode on an old gaming computer with just enough RAM and VRAM to handle an MoE, churning out something overnight that would quickly exhaust the subscription plans, like a knowledge graph for a large document corpus.
16GB VRAM could load a small quantised qwen 27B model but it would be a ways away from a frontier cloud model.
Though keep in mind not being beholden to shenanigans from said cloud companies (and interference from government entities!) is definitely worth something intangible.
I think 24gb is the bare minimum for a local qwen3.8 based setup. I've used qwen3.6 and it's not as straightforward as "can it replace <insert the most cost-effective cloud solution today>"
Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.
Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.
The principle of KISS keeps coming to mind when I consider local computing. I'm looking forward to the day we can just run Opus-level models at 100 tok/sec on consumer hardware.
But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)
I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).
Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.
I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference. It is nice to know that I have hardware under my desk that I control with open weight models that I can interact with on my terms.
yes, you have to unfortunately adopt a 2025 mindset when working with smaller local models because that's where they are compared to the frontier.
> local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference
In my opinion, 98% of the work most devs would send to an AI can be capably achieved with a local model and a frontier-level model is overkill.
The goalpost moving feeds right into Anthropic and OpenAI's interests.
The article implies the demand is for running locally. I’m not convinced, at least with a mac minis. Most people I know and myself buy the mini as it is always on, easy to setup, and isolated from my main computer which is a laptop. The mini is driving the use of the $20/month subscriptions.
As a thin client to access cloud models this is an astounding waste of money.
One use case I find cost effective is using it as a voice assistant for Home Assistant. API pricing on models is very weird compared to the normal chats, so I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.
Could you elaborate on your experience with local models on your card? I've been thinking of upgrading to 9070 XT, and was thinking the 16GB would be okay-ish to at least run something usable locally, no?
Usable certainly. But my impression is that useful models still need a bit more than 16GB. Something like Qwen 3.8 27B is useful but squeezing it into 16GB requires fairly aggressive quantisation which will make it unreliable (e.g it'll get stuck in loops) and won't leave enough space for a long context (which qwen 3.8 really likes)
I’m the parent of this thread, the person with the with the RX 9070.
My understanding would be that if you’re interested in this sort of card for AI that you should go with the AI PRO R9700, which is basically the professional version of the RX 9070XT but with 32GB of memory.
It’s significantly more money but not crazy like a 5090.
I just happen to have the 9070XT primarily for gaming purposes.
I’m not quite sure how to describe my experience using it other than “rudimentary,” and a lot of that is on me for not really understanding the best way to set it up.
If you have been using cloud hosted models, you will be severely disappointed with what you’d be able to run on 16GB VRAM. You will spend most of your time fighting with the model to fix its mistakes.
I was getting semi-useful results from a 128GB M4 Max. That was a few months ago, and the models have improved (quite a bit) since then, but now I'm happy to send my $20/month to get Claude code.
It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
Bloody training data full of humans demanding sleep. I tells ya...
> "not do significant work this at this late hour"
Is this Claude code? Or your local? I assume Claude? I'm more than a little staggered by this, like, it makes no sense! It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.) The LLM just... decided? I'd be so mad.
WTF indeed. Can one even file bugs?
> It doesn't even serve Anthropic's interests (surely better for them if it burns your token quota so you have to buy more the next morning.)
Parent mentioned their $20/month subscription. It's definitely in Anthropic's interests for you to not use it.
> I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
I think that's Anthropic trying to get you to not extract as much value out of that subsidized subscription as possible.
I've been experimenting with local models on an M5 Max MBP w/ 128GB of RAM since March of this year. Generally I've had very good results. Where things were lacking initially was with tool calling and the need to rely on tool calling for functionality like web search, which is otherwise well integrated in the cloud models. There is also a lot more work required on the harness side, however at this point (August 2026) there is not only much better tool calling in local models, but community supported projects have built good harnesses. Pi.dev and OpenCode + a SOTA local model is /very/ /very/ capable these days. Using LM Studio's built-in chat with a decent system prompt and proper tuning with local models is /very/ capable these days. Cloud models are still better, but it's the "harness" (not in the desktop app, but in the backend) that makes it so for the most part.
IMO local models require a substantial amount of prompt+harness engineering to get in the neighborhood of what you'd get from a cloud model. Which isn't a bad thing, you'll learn a whole lot about how these things work.
What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.
When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.
What you’re describing is exactly what I’ve experienced in my time testing out this stuff locally, and I had a hard time figuring out what exactly to blame.
I keep hoping that one day some comment is going to paste a link to some kind of idiot-proof guide or piece of software that’s “90% as good as Claude but running local.”
And by 90% I don’t mean that the model is 90% as good or runs 90% as fast, more like all the other stuff you mentioned is set up out of the box.
> 16GB RTX 9070
What's an RTX 9070? Do you mean the RX 9070 or RTX 5070?
Oops yeah I meant RX 9070, fixed it!
I have multiple 48GB friends that successfully run smaller quantized models for general assistance + light editing (coding, spreadsheets, etc), that don't require very heavy models.
So yes, they are genuinely very useful, but they are not yet a full replacement unless you have more powerful hardware and or don't need more intelligent ai.
Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
I need a new little Mac for my music studio, currently an M2 MacBook Pro. I thought I'd be fun to experiment with some local models as well. Well, let's price up an M5 Pro. $3,019 with 64GB RAM and a 1TB HD. Three thousand American dollars for a Mac Mini. Beefy spec for sure but not comically so.
Frankly even the entry price is a bit high - I remember buying one for my son a few years ago (M1 mini) and it was a few hundred; now we're up to $900 for the base model.
It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
It's also fun to see how many people here believed this was all some clear deliberate strategy in the first place rather than an accident.
Maybe a bit of hindsight bias / the outside view here, but I feel like they're completely asleep if they didn't anticipate strong demand for this specific use case.
I think a reasonable story could have been told that goes like this: local models aren’t as good as frontier models with a $20/month subscription, and the hardware costs a lot. So only a few enthusiasts will buy Apple machines for this purpose.
This story turned out to be false but I think smart, reasonable people a couple years ago could have believed it with conviction. It doesn’t really seem like “completely asleep” to me.
I don’t understand how that’s possible. They should have had a better idea of what was happening in the memory markets than pretty much any other entity.
Their universal RAM strategy is so obviously helpful for AI. (1) GPU/NPU <--> CPU RAM copies eliminated. (2) All (most) RAM available for GPU/Neural, when local models are typically kneecapped by limited GPU RAM sizes vs. the much larger RAM options for M/Max/Pro/Ultras.
They have been taking NPU's seriously on their phones, tablets and laptops since the M1.
Then they enabled fully-connected RDMA for 4 x 512GB MacStudio's = 2TB RAM. Perfect for a large Mixture-of-Experts model.
It would be very strange if they didn't notice their product line had landed in a new sweet spot.
FWIW, the reported reason for OpenAI buying Macs has nothing to do with the memory by the sounds of it. Every single outlet I can find reporting on this seems to repeat that the intended use case is for agentic workloads and generating training data for reinforcement learning. They don't appear to be doing inference nor any sort of training AFAICT.
I am curious about the corporate disconnect from the frontline to the generals.
While the company I am in is embracing AI the disconnect and delay between what is available and possible versus what is approved and permitted is a three month window. The State employees I speak to are just now getting around to writing their usage policies for internal AI usage.
You should listen to the podcast Acquired, specifically Nvidia and then Jensen Huang. They basically lucked into AI. Some researcher was using Nvidia gaming cards, and reached out to them about questions on CUDA. That email eventually turned them into a trillion dollar question.
No ‘staff focused on developer relations’ is entirely unsurprising based on what I see from the outside.
Was this the case in the past?
My vibes were that Apple wound down the “actual work” side of their operations (including machines like Xserve), because Ives couldn’t handle the unsexiness and unpredictability of business requirements in hardware.
He was self-indulgent and only wanted to work on things that “vibed” with him, rather than what the customers needed. It’s easy to be creative when you get to do what you want to do, it’s hard when you have hard constraints.
I think Jobs was quite sceptical about courting enterprises. Personally this is one of the reasons I choose Apple over Microsoft.
I really hope with Ternus taking the helm Apple starts to remember that it has products outside of iPhone.
I wish they would stop with the new-iPhone-every-year nonsense and refocus on quality, fix some bugs… but yeah, not gonna happen
If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
32GB is not enough, it's unified/shared memory, you need to have space for usual system and user apps/services.
64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.
> If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.
32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.
Time to bring back the Xserves, I guess.
Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.
I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
More customers is generally a good problem to have in most businesses. Just that the situation is very paradoxical given the supply shortages.
There are so many used M4 Mac Minis for sale on marketplace for $600+ now that the people I guess got bored of the local models and decided to see if they could make their money back as they probably bought them when they were $399 on sale earlier this year. I lowball them every time.
I wish they sold something that could go in a colo - redundant power supplies, lights out management, etc. you know they have them internally...
> you know they have them internally...
What makes you think that? There's a lot of data centers that sell you access to colocated Mac Mini's, they have added FileVault unlock via SSH in the boot process which also makes things easier. There's not that many reasons to run a Mac in the cloud unless you have some very specific Mac related workload.
Because there's been photos of Apple building server racks with Apple Silicon, but also a recent leak.
https://www.macrumors.com/2026/08/26/leaked-images-of-apple-...
https://www.reuters.com/business/apple-begins-shipping-ai-se...
Apple built internal M5 servers for private cloud compute:
https://wccftech.com/apples-private-cloud-compute-server-m5-...
https://security.apple.com/blog/private-cloud-compute/
Thanks, missed that article.
> What makes you think that?
There’s articles about them, Apple uses them internally for AI services. https://forums.macrumors.com/threads/photos-of-apples-own-ne...
At this point it’s a well known secret that Apple has real rack mount servers for their internal processes. They actually have officially released video of their servers in the WSJ report on their chip supply chain.
https://forums.macrumors.com/threads/photos-of-apples-own-ne...
Apple has to have significant build infrastructure to support internal iOS development, surely? They can't just be using whatever is at the developers' desk, or a big pile of Mac minis in a closet. That's far too pedestrian for Apple internal works.
Plus they did sell rackmount servers for some time.
> What makes you think that?
Here's a leaked / rumor image of Apple servers themselves.
https://www.macrumors.com/2026/08/26/leaked-images-of-apple-...
> [cites a convoluted work around]
> [still claims there is no reason]
They did. Now think they feel a stack on Minis or Studios fills the reduce needs better. The multiple machines one gets software redundancy in addition to everything else.
Off guard? :'D
Apple did not raise prices fast enough
This may sound a little wacky, but one potential use case I'm considering is robotics. Say you want to use a fine-tuned mid-size VLM model right on the robot. You pretty much have to use the Jetson line, which has a great ecosystem and everything (depth cameras, lidar, SLAM, small segmentation models, etc) runs on it, but it's gets very expensive very fast if you want to run LLMs on it. The Jetson Thor lines are 3-5k depending on memory etc.
One very efficient option today is to have the cheapest Jetson (Orin Nano) run the classical robotics stack, then have a base mac mini run nothing but the VLM. The Mac mini is considerably cheaper and faster at these workloads than the mid-range Jetsons.
I think this wonky situation is because Apple us under immense consumer pressure to absorb the ridiculous memory prices, while the Jetson is aimed at "business" and much more likely to fluctuate with the market. Last year I bought a Jetson Orin Nano 8GB for $375CAD, today that official nVidia Amazon page is out of stock and other sellers have it listed for $900-$1100CAD. Absolutely bonkers pricing.
Now if only they hadn't discontinued the Mac Pro. Could be quite the AI machine with multiple compute GPUs at higher bandwidth than an external Thunderbolt enclosure.
Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.
It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.
If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.
Apple doesn't design GPUs on-par with Nvidia's efficiency yet. They need an architectural overhaul to be a serious competitor, which is what I'm expecting is queued up for M7.
Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?
just a real bummer that they raised the prices so much
I don't know if you're young, or new to this industry, or what, but take a look around - the prices went up across the board. It's not just Apple.
It definitely puts it out of the range of every day users but a non-insignificant proportion of people who use it for ai have become multi-millionaires because of ai. So there's definitely no shortage of people who have no problem paying those high prices.
Citation needed.
I suspect it can't be helped at this point. RAM is the new gold. Valve even had to increase the price for a 4 year old piece of hardware[1].
[1] https://tech-insider.org/ca/steam-deck-price-increase-2026/
Please address all complaints to the Office of CEO, Open AI, 123 Main Street, San Francisco, California, U.S.A., 94158.
Your letter will help feed the boiler that heats the building and is much appreciated.
They use the datacenters for that.
I am missing something in the article. From what I am reading, AI companies are so desperate for memory they are buying Apple Systems and other hardware and striping them for RAM and maybe other components.
Is that what others read ?
Apple hardware uses a unified architecture. That means the CPU and RAM are integrated together: it is very difficult/impossible to strip the RAM from these systems.
This unified architecture makes Apple hardware very good for AI work, where latency between RAM and CPU is very important
Even the SSDs (which are normally more strippable) use a proprietary hardware form factor.
I think the article mentioned the real draw:
> Apple noticeably promoted the ability to link multiple Mac Studios together into a single, more capable system for running large frontier AI models, a feature aimed at business and developer customers rather than everyday consumers.
(Couldn't read the source cited, might have more info: The Information)