Right, so it's not just git, it's a bunch of system-provided utilities; macOS trusts bad certs, and these utils that trust the OS are affected. I use homebrew for my git and curl but haven't yet checked to confirm whether that matters here.
Kind of outrageous that Apple's guidance is to edit the cert chain manually, and to expect that subsequent OS updates may revert your mitigation, reintroducing the vulnerability (as happened at least once so far).
I don't know about worse. Imagine a world where there is no OS authority, each app does their own. Do you think all important apps would be updated in 10 days, including auto-updates?
Apple fucked this up, but I really don't think the situation is "as bad if not worse" than if MacOS just didn't do this.
It makes no sense to imagine this hypothetical scenario and compare it to reality, you have to imagine a similar world with a OS authority, but it only gets updated every 6 months. Which is worse in the worse case? I'd say the one with the lower blast radius.
But this abstract exercise is only to highlight the importance of Apple getting its shit together, because such a scenario exists with Windows and the culture of downloading exes and msis and it's honestly way worse than having a central but responsible authority.
Homebrew ships its own CA bundle so its own curl, Git, etc. are not affected. But the system versions of those tools will continue to be affected until Apple patches their own bundle.
Edit: Sorry, I forgot that I had patched my local bundle. This might indeed be an oversight, I'm looking at it now.
I couldn't reproduce this:
% which git
/opt/homebrew/bin/git
% git ls-remote https://git-emudhra.we-have-a-problem.com/check.git
fatal: unable to access 'https://git-emudhra.we-have-a-problem.com/check.git/': SSL certificate problem: unable to get local issuer certificate
Make sure your `git` binary is Homebrew's, not the system/XCode CLT one. If you see `/usr/bin/git`, you're not using the Homebrewed Git.
We clearly just don't know what complex level security Apple has put into their OS. Clearly their OS isn't vulnerable or else they would patch it! (/s)
That MacOS hasn't removed from OS-level /etc/ssl seems like a big deal, affecting more than just OS-supplied git. Is /etc/certs not used by most graphical Mac software, I guess? The chaotic state of OS-level cert management, on Linux as well, is pretty dismaying.
If Apple is still trying to brand themselves as security-forward, you'd think removing the revoked CA certs from /etc in a system update would be a pretty easy thing to do?
I don’t think it’s gross incompetence, since they documented the steps to remove. They may have some unexpected dependency on the file? Looks mysterious from the outside.
People would have to use it in some drastic catastrophic way for them to force it out in an update, which they might push it out in the next macOS update, which don't come nearly as frequently as Windows bloatware updates do.
This seems important enough to just send out a patch update that removes the cert and does nothing else. But apparently at least a couple people at Apple disagree; or, something I'm not familiar with makes that type of update impossible.
I mean, I agree with you, I'm just basing my thoughts of what I've seen managers at different companies prioritize as "important" even when engineers cry that a security issue should be of higher priority...
> Is /etc/certs not used by most graphical Mac software, I guess?
Correct, those usually use Apple's networking APIs which all go through their security stack, which already picked up the OCSP revocation within about 2 hours in most cases.
this only affects terminal apps, which the vast majority of MacOS users don't run; the fix for graphical Mac software is already patched
my guess is their thinking is that if you're running those apps on Mac, you're a developer or know what you're doing, and can edit /etc/ssl/cert.pem until they push an update
they also have to consider that such users may have custom changes to their cert.pem file, and they probably don't want to rush an update that would potentially fu-bar those changes
On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.
Why are they not just pushing out the update asap? Unless they're relying on the compromised certs, then it feels it should be incredibly trivial to push
Because to Apple once you open a terminal you leave the supported world behind. This is a standard thing for them. Apple is not really a UNIX vendor and certainly not a Linux vendor: they sell macOS which is primarily a consumer OS and has its own APIs for everything.
Tools common on Linux like curl and git come with the OS basically for developer convenience. Apple doesn't write those programs, they don't use Apple's APIs or platform features, and thus in some sense you aren't really meant to use them in Apple's eyes. Ports of UNIX CLI tools exist in a kind of semi-supported netherworld on macOS. They run but they aren't fully integrated or supported parts of the OS.
This case not withstanding, Apple regularly includes patches to the built in CLI tools in security updates, so it's a little disingenuous to suggest those tools are not really supported.
The issue might be Root CA vs intermediates. The root CA that eMudhra rolls upto may be used for other intermediates which were not compromised or revoked. Removing the root CA will mean that these valid certificates no longer work.
This is technically a problem in the CLI - it needs to support revocations properly, not rely on root CA verification only. I’m not sure though whether there’s a standardised mechanism (that Apple supports) for CLIs.
> These steps affect every tool that uses /etc/ssl/cert.pem. Afterward, tools that use only this file might not be able to verify some websites whose certificates chain to these root CA certificates through intermediate CAs that weren't affected.
In pracitce, those root CAs have already been removed in major browsers. If your server uses them, you would likely have needed to migrate to another root already.
Apple’s own security framework is supported by curl with OpenSSL or GnuTLS (build with ./configure --with-apple-sectrust). That blocks the intermediate certificates via the same mechanism Safari uses for example.
Not sure why macOS doesn’t ship such a build of curl.
I think since 10.15 the built-in command line tools in MacOS are there mostly for backwards compatibility. This was clearly communicated about scripting languages, but there was a feeling back then that this was about the whole lot (all cli tools).
Theory: Apple are moving away from user-controllable development tools, towards an AI-hosted application development framework which removes the divide between user and developer, and thus the need for specialized developer tools.
The writing is on the wall for the command line tools.
(We can already see this tooling devaluation happening all over the place. Apple have their variant of this response to the AI tooling-paradigm tsunami…)
Precisely because the agent is becoming the user of the command line tools now, these are all being abstracted behind an interface which blurs the developer/user requirement…
The original question is, why aren’t Apple fixing the CA problems for the command line tools (whereas they have done it for .app’s such as Safari), and the reasoning is, because the user will just have to fix it themselves, because Apple don’t care about the command line users - they see that these command line tools are, indeed, becoming the front-line for agentic development, so .. presumably, they have an .app bundle which does away with this problem, to be shipped on the horizon ...
They have been treating the command line tools like the bastard child of their software portfolio, and then there’s that whole drive to remove the Unix’ness of MacOS .. Swift/SwiftUI is a drive away from lower tooling layers, and .. there is no doubt they already have a developer agent of their own, internally, pushing their abilities to make projections of hardware requirements and specifications for the next 2 - 5 years …
I’m not saying they’re going to deprecate XCode today, but I do believe they will “agent’ize” Developer Tooling drastically in the next year or so. Agents are becoming the most important user of command line tools.
Because they have to, in order to catch up with NVidia, their only real competition. If you want to understand why I think that, its because I have been developing and deploying cross-platform Linux-centric (but MacOS and Windows targeting) applications, mostly for the ARM platform but also with Intel (SIMD on both) .. for the past few months, and I haven’t touched an IDE in a month. On DGX Spark. Which turns out is a truly, utterly, mind-blowingly great platform to do cross platform high performance application development…
Apple has pretty well always got users to do things via the GUI – look at the Lisa and Classic MacOS.
CLI is just a thing that NeXT had and even that made it easy to use GUI tools.
Xcode is a GUI tool – it happens to expose some cli tools but they could be covered up – however some developers would complain a lot so I don't see that happeneing.
The command line tooling lets you/AI manage a lot of your app store interactions though. It's much easier to use that way then trying to deal with the Steamworks UI and doing things manually.
Like many suggested, a proper fix might be harder than simply revoking a bunch of certificate like the blog authored assumed without breaking downstream features and functions in a large ecosystem and security conscious platform like macOS/iOS.
However, there is certainly a chance that Apple team didn't think the impact was that bad given all the other protections in place.
When Steve Jobs was alive, his Apple company email was leaked and published and Apple customers emailed him for features and fixes all the time and he personally responded once a while. Tim Cook did the same a couple times too. Perhaps the blog author should write in to the new CEO John Ternus or SVP of Software Engineering Craig Federighi too to see if they can fix that.
I'm not being sarcastic when I say that Apple should copy Windows Update, I have no idea how they have got this far without a method to add small patches outside monolithic OS updates.
> A flat cert.pem file (or similar) can't say "distrust this intermediate".
Why does that matter in this case? Can't this be fully mitigated with next to no collateral damage just by removing the trusted roots that signed them?
Chrome dropped one of the eMudhra roots. The problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Browsers (at least Chrome) didn’t drop the Sectigo root, instead they store a revokation for the cross-signature.
Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, but it means that some small percentage of certificates that are fine won’t be accepted.
It seems like we've all just... given up on certificate revocation? CRLs and OCSP were designed for exactly this sort of thing, but they both seen to have been abandoned as unworkable with no replacement.
Same here - I genuinely thought this would have been easily, and almost immediately, remediated vis-a-vis CRLs. I'm not in the vuln management space anymore, but is there some sort of caching system used which would affect software differently than a typical browser?
Don't underplay it either though: with a compromised but unrevoked signing cert all it takes is a successful phish, or some problem to allow DNS poisoning (or hack a wireless AP, or …) to direct you to a fake site or (potentially worse as it might be less immediately noticeable and could quietly collect credentials/data for a while) send your automated tools to a malicious API.
It may not be directly exploitable on its own, which is the impression some reports give, but it is a serious hole to leave open especially with the current rate of old security issues being found, as it could be _incredibly_ useful to a malicious actor as part of an exploit chain.
It's pretty easy to compromise things like McDonalds or Starbucks WiFi hotspots though, just setup a Pineapple with the right SSIDs and a 5G modem, funnel your target domains through your mitmproxy via the DHCP DNS server you're providing, and that's basically it.
I agree it's not PHP2Shell, but its certainly not being "way overblown" here.
I think they're saying you create your own malicious wifi network and people will unintentionally connect to it. This network specifies your malicious DNS server via DHCP, and everything looks secure to the client. I think the main issue with this in practice is that all the major browsers have been updated to not trust these certs.
So you target some host like ssh.dev.azure.com, find all the hooks in the proper repo the client is trying to access, make a new commit that puts your persistence payload in one of them that you expect will be executed soon, this script can additionally "correct" the git state when it finally runs, and moves its persistence to some more appropriate location.
Users will be none-the-wiser because they're not accessing these hosts through a web browser, and you net a few more compromised hosts in your inventory.
Yeah also got to be lucky that the users are using https and not ssh. You're very limited in what you can mitm, you'll be trivially detected by patched devices (though maybe not acted on because nobody is the wiser), but then you'll need to serve objects with common/conflicting hashes -- fixing up that state synchronisation (what you call "correct") to then do nefarious things is not trivial at all. Extremely difficult to pull off successfully for git anyways or to do it in stealth.
They wouldn't. A device like the Pineapple broadcasts its own copycat wireless signal and tricks unsuspecting clients to connect to it thinking it's the real one.
Edit: Holy moly we all jumped on this comment at the same time didn't we.
It’s extremely common to see other Macs pop up in the ‘network’ panel in Finder in the independent coffee shops around me. I imagine they’re just signing up for a business plan and using the routers the ISP gives them with a custom SSID and password.
They don't need to, a Pineapple (or anything else that can run a wireless access point, you could do this with a laptop) just needs to advertise the expected SSID and do so with a stronger signal than the official ones for people's devices to connect to it instead of the actual network.
The near-universal use of TLS and the absurd ceremony we go through to ship root CAs everywhere is all premised on the idea that MITMs are not terribly difficult.
The entire reason one can rest easy as they connect to their coffee shop's WPA2-PSK network, is the security provided by TLS.
And if you don't think an ISP or agency would abuse this if it could be done undetectably then you haven't been paying attention these last 20 years (hence why DoH is now a thing).
Appealing to complexity is always fun. Just throwing "a lot" in there for good measure and you won't have to think about practical security implications.
The TLS trust chain is the last defense against MITM attacks. The prevalence of MITM attacks is in fact the very reason TLS exists. I'd say it's a lot worse than PHP2Shell because it doesn't only affect poorly maintained servers, but potentially ALL credentials and secrets a user sends over a compromised network with compromised CAs on their device.
The window of opportunity to MITM has passed. If you attempted an MITM from an easy vantage such as telco, ISP, or public wifi you'd be trivially detected as a significant portion of devices are patched, including on these unpatched macs as only the *nix commands are affected and not app bundles.
I can't imagine what it's like to be naive enough to believe that all small-town cafe that have a wireless access point, small ISPs all over, and mid-level engineers in big ISPs can just be blindly trusted.
If you have any real life experience and don't pretend that human psychology isn't a thing, you already know there is "a lot" that came compromised out of the box.
This is way worse than PHP2Shell in scope of impact and its effects will stick around for much longer. Well, as long as the validity period of the compromised certs anyway
> You have to compromise a lot to be able to make a MITM here.
It's trivially easy to mount an ARP spoofing attack. I did that at school two decades ago and on-the-fly rewrote <img src=...> to hardcore porn images for funsies.
mitmproxy [1], a popular toolkit for making that easy, has the option to generate "eMudhra" certificates on the fly now.
It is trivially easy to compromise things, all you need is physical access to the network.
Do you have a cell phone? Do you use Wifi? Do you use an ISP who, in the last few decades, has performed any form of interception or traffic-tampering?
I hazard that the answer to all three of those is "yes".
You don't have physical access to the network, doesn't mean other actor aren't here. Working for providers, governments, or being able to impersonate them. This is a serious issue, it shouldn't be downplayed.
If you think it's not that important, it becomes a critic about TLS and basically saying that ciphers for web traffic are useless.
Public wifi hotspots tend to give you that opportunity. Large commercial hotspots have inter-client communication blocked, but your average coffee shop style network? You're lucky if it even is an isolated VLAN from the shop's cash register lol.
Except it won't impact gmail unless you're accessing it through curl. The browsers already distrusted the compromised intermediates via CRL within hours of disclosure, and removed the affected root via update.
Sure practicality is always an issue. It's an issue that attackers will go to great lengths to resolve too.
What if it's the Starbucks at 1730 Pennsylvania Ave NW, Washington, DC 20006 for an extended period of time? Compounded with other security issues like password reuse...
We can always draw a line and say, "that's not practical". The assessment an attacker is likely to do is going to be "is this practical inside my budget for the value I hope to acquire?" or maybe even just "waow, lulz".
I have, but good luck predicting where I'm going to have my next cuppa.
Outside of petty crimes mentioned, I don't see how it could be industrially weaponised to make actual damage.
Regardless, real bad actors are busy compromising your next `npm install` anyway. Much better reward.
If they are specificly targeting you, then they probably dont need to use the coffee shop. The coffee shop is for low skill attackers targeting people at random.
If someone has it in for you and hired a high skill attacker, its not a stretch to imagine they compromised your home network.
> good luck predicting where I’m going to have my next cuppa.
Why are you assuming there isn’t someone else trying to hack you at the next cafe? Actual damage can happen one person at a time, and can occasionally lead to larger scale compromises. Randomizing your location isn’t going to reduce your risk. Now that everyone who’s paying attention knows they can MITM your Mac, you shouldn’t trust any network.
Client isolation is very common on wireless and wired LANs these days in enterprise and guest network environments. You also have tools like dynamic arp inspection to block poisoning attempts via the network switch.
If you get past all of that, you still need a victim who is actively using a cli tool on macOS using https.
It’s not impossible, but the conditions are much more specific than those needed to inject porn into a webpage on a poorly configured school network in 2006.
…and yet it doesn't help at all if a laptop sees an AP with strong signal in its vicinity and connects to that instead of the meticulously configured AP that sits behind the counter with a slightly weaker signal.
You’re not actually making the point you think you are. This comes back to the original comment stating you need to compromise a lot to MITM here.
So you need to somehow trick macOS into remotely connecting to the wrong bssid, and then arp poison a user who happens to be using a macOS cli tool that’s using https.
Go to a random coffee shop that allows people to work at the tables, and you’ll find a few developers with MacBooks there pretty soon. They will be connected to the local hotspot, which either has no password at all, or WPA2 with the password written on a billboard behind the counter. Spin up an evil twin, then wait for the connections to roll in.
It's actually worse: Installing the 27.2 update after manually cleaning out /etc/ssl/cert.pem will revert those changes and put the eMudhra CA back in.
Was thinking the same. My assumption would've been that this would have been a normal and routine enough of an operation to be able to push through without needing a separate RC process.
While there is no proof that anyone knew about these weak keys for the year and a half that they were trusted by virtually everybody, and there's no evidence of MITM attacks happening in the wild because of it, you can imagine the payoff that a three letter agency would have if, for even 10 days, there was an opportunity to harvest credentials from important people in high places.
I have no evidence or proof, but I am convinced this wrap/unwrap key generation debacle was intentional and the weak keys were discovered and kept secret by the agencies or governments and used to exploit their adversaries.
You don't need to perform a traditional MITM attack _if you own the airport_ or you _own the free public wifi_. The traffic can be harvested in clear text and we have to come to terms with the _remote possibility_ that this was going on for over a year and a half. If not that long, then certainly for the past 10 days.
This is misleading. In order to read traffic you would need to swap out the end-entity TLS certs being used. Phones and browsers (at a minimum) enforce certificate transparency, so connections would fail.
There must be a long list of things that aren’t fully patched for this yet, on laptops, on servers. Think of packages that ship their own trust anchors, isn’t there a rust crate that compiles with some root certs embedded?
On one hand there’s unmaintained packages that won’t update dependencies. On the other hand, fixed dependencies did not necessarily backport to all possible versions in the wild, making update as a fix unviable for those.
Anyone running Intel macbook because Homebrew dropped support for them because Apple dropped support for them. You can still use Homebrew but it means rebuilding from source. I downloaded uv recently and it took about 8 hours to fully install because it had to build llvm, uv, some other stuff all from source.
Heaps of us. I'd been using it for years before Homebrew even existed and I never felt that any of the problems that Homebrew purported to solve were real.
I'm not that familiar with the MacOSx system, but do they have a "minor" update path that I'm not aware of? It seems that everything is an 8gb update. Is that why this update isn't getting put out?
Can we get a title edit? OP’s title implies this is a Git bug, but the bug applies to half of macOS, including curl, openssl, python, perl, and ruby, and to only the macOS version of Git. The original title is less misleading, but personally I’d have titled it something like “macOS still trusts the compromised eMudhra certificates in git, curl, openssl, python3, perl, ruby, more”.
Why is the cert.pem not generated from the keychain storage? Does someone want to vibe-code a utility to do that? It seems like the better approach (one master read-write storage with replication)
To get mine to finally update, I had to leave my laptop plugged in for a couple of hours. There’s some trustd process that runs every so often; I couldn’t figure out a way to manually trigger the sync.
Running (uname -a) Darwin arrakis.local 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:07 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T8132 arm64, I get the following for both:
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.
$ uname -a
Darwin MacBook 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:21 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T6030 arm64 arm Darwin
$ which curl
/usr/bin/curl
$ curl https://dv-g3b-emudhra.we-have-a-problem.com/
Your client successfully connected to the site. That's bad!
Not sure what Linux offers that's any better. Didn't Debian ship trivially weak SSH keys for over 2 years because they were trying to silence Valgrind warnings and the random maintainer had no idea what they were doing and there was no oversight or peer review whatsoever?
> the random maintainer had no idea what they were doing
Upstream was involved in it as well. Sure it was a communication problem, but it's not like the maintainer just decided to this unilaterally. Also this WAS undefined behaviour, so from a language perspective it was already broken, even if it worked accidentally in practice.
That's one issue from one distro 18 years ago. Have there been recent things? This thread is about a today's MacOS, and an issue known publicly for more than a week.
> Well that "screw up" was so egregious that it should take at least a decade to regain any trust.
Do you use macOS? Because if you did, then how did you recover from them allowing anyone to straight-up log in as root without password? And then mess up the fix for that bug? That wasn't 2008, that was 8 years ago.
Besides singling out a single bug from a single distro two decades ago, Linux offers maintainers who actually care to push security fixes within a few hours of globe-spanning vulnerabilities being disclosed? Not sure how that's confusing or controversial.
It's funny. When Apple goes in and changes peoples systems deeply without consent then it's Apple being evil. When they don't, it's Apple being evil. Heads I win, tails you lose.
Let's pretend there's a night and day difference between pushing some MP3 onto all iPhones and distrusting a chain of certs that hold the value of '4'. At minimum Apple need to revoke everywhere these joke-level certs live and as fast as possible, and on all devices which also means out of support. Breaking is acceptable here as CA trust is fully compromised.
Yes, there would be crowds of angry people, furious that Apple... released a critical security patch? Framing removing compromised certs as "changing peoples systems deeply" seems intentionally in bad faith.
That page is an insult to their customers. So what, if I want to be protected from a security issue you acknowledge, I'll need to make modifications to my system configuration?
It’s interesting they call out as “optional” removing sectigo roots.
If you’re not removing them you better be sure anything that runs on the system has full revocation support including intermediates. Which is.. kind of an impossible thing to assert as true. Which is why most trust stores just removed that as well last week.
I use SSH for things which are private or which I may want to push to, but I use HTTPS for everything I just need to download. SSH typically requires that you create an account with whatever the git host is and upload your public key.
SSH in typical use is not vulnerable to a compromised root certificate, instead it's "just" vulnerable to you having no idea whether you can trust that host key or not, so you just blindly say yes and hope for the best. At least if the key changes, you'll know something was wrong; maybe it was this connection that got compromised, maybe it was the first connection that got compromised, maybe the legitimate host just rotated their keys, who knows. SSH security against MITM is mostly based on lack of motivated attackers and wishful thinking.
Its no more wishful than expecting a root of trust to stay trustworthy or expecting a trusted name to consistently resolve to the same party.
And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.
Albeit helpful, the site is kinda cringe.
It leaves a bad taste in my mouth knowing that the author is affiliated with Mozilla.
Of course they would throw shades at Apple. That being said the situation isn't ideal and I'm sure Apple would have shipped an update if it were easy and warranted.
Affiliations aside I think I'm happy to see shade thrown at anyone who hasn't bothered to even issue a patch 10 days after what has been one of the biggest news weeks in security in memory.
Fwiw, as critical as I am of Mozilla in recent years, they (along with the Chrome team) have been pretty solid on this whole debacle over the last week. Apple has been far from the only target of their collective shade-throwing.
A concern for me on this site is that they give the shell commands to edit the /etc/ssl file on their site, instead of just making the Apple link (which they also list) as authoritative on the matter. I don't think users should get in the habit of cut/pasting sudo enable shell one-liners from "random" websites, especially and specifically when the vendor has the very same commands in their published fixed for the issue. In short, this site should have just said "go get the fix from apple".
Your apps and browser don't trust them. Why should your CLI. It's easy and warranted for Apple and look how much cash and engineers they have.
Apple should publish a hotfix now. Work on a better solution in the long-term. They just don't care about security for developers is my take (it's like a niche group for them?)
I have been wondering recently if Apple are bothering to maintain the Unix side of the OS. Seems like they occasionally remove or deprecate stuff but don’t put any serious effort into maintenance.
Microsoft had a small golden age with Azure and Windows 10, and now they are going right back into a Vista-era-like rot. Github is down an awful lot. The reliability is worse than the product I work on for work by like 2 orders of magnitude and we have no ops team. It's a joke.
Windows 11 is awful, lots of the fixes and modernization started in windows 10 has totally stalled, leaving users with 5 different places to change a setting instead of the one old place or one new place. Look how many different UI's there are to adjust sound card / sound settings or display settings. It's a complete mess. Their attempts to jump into the AI game have been a complete joke, despite already having integrated Cortana years ago. Putting Cortana on an llm is such an obvious thing to do, and they already did so much work to make that fully integrated into windows and force it on everyone even though it was trash, and now it's still trash and they don't seem to be able to adapt at all.
Now they can't respond to a complete insecurity in one of their major products in weeks? Why do they even validate certs? Why doesn't revocation work in their app? I don't think they can really be trusted at this point.
(on linux you always have the update-ca-certificates command that downloads the root certificates from a trusted™ source. but sure, it will not help, if each command starts to bring their own ca package :)
Exactly, so the title wrongly frames git as trusting certs that it shouldn't, when it's correctly trusting the system certs, and MacOS is wrongly trusting certs.
Yes, and that makes not dealing with this issue worse on Apple's part. If MacOS' setup somehow doesn't allow untangling a (still deemed trustworthy?!?) root certificate issued by eMudhra from the broken intermediate certificates, then revoke that whole chain including the root & accept collateral damage.
Failing that just provides an incentive for apps to do their own chain-of-trust implementations. Which like you said is not the way to go.
The problem is macOS has two system CA stores, keychain and /etc/ssl/cert.pem.
That is unequivocally bad, similar to Java which ships its own keystore.jks and many applications shipping its own copy on top of that. True nightmare fodder.
It's also an indictment of Apple's relationship to open source software. Never mind contributing back to upstream, they haven't even made the change in their own fork to support their own keychain.
> keystores in every single container, nested keystores within that
At least for that, in $OLDJOB I went and created a new keystore.jks myself to account for an internal CA, so my successors can just update that file, restart the containers and that's it...
Because it isn't got specifically, it is anything that uses the installation of OpenSSL provided by Apple as part of macOS (at least is my understanding).
The problem is that the title implies that it's somehow the fault of the package. That's it, that's the problem; the title is disingenuous in its claim.
"Ten days later, Apple Image Viewer on macOS still trusts the compromised eMudhra certificates" doesn't sound as catchy, but is equally correct because the apps use system CA.
Right, so it's not just git, it's a bunch of system-provided utilities; macOS trusts bad certs, and these utils that trust the OS are affected. I use homebrew for my git and curl but haven't yet checked to confirm whether that matters here. Kind of outrageous that Apple's guidance is to edit the cert chain manually, and to expect that subsequent OS updates may revert your mitigation, reintroducing the vulnerability (as happened at least once so far).
To be honest, git deferring to the OS is the right thing to do. It's macOS that's the problem, not git.
"Ten days later, macOS and all applications that rely on it still trust the compromised eMudhra certificates" would have been a better title.
Yeah, definitely: having lots of separately distributed and likely rarely and unreliably updated trust roots would not inspire a lot of confidence.
In turn the centralized OS authority needs proper maintenance otherwise it's just as bad if not worse, because of the larger blast radius
I don't know about worse. Imagine a world where there is no OS authority, each app does their own. Do you think all important apps would be updated in 10 days, including auto-updates?
Apple fucked this up, but I really don't think the situation is "as bad if not worse" than if MacOS just didn't do this.
Isn't that the world that Docker gives us, basically?
It makes no sense to imagine this hypothetical scenario and compare it to reality, you have to imagine a similar world with a OS authority, but it only gets updated every 6 months. Which is worse in the worse case? I'd say the one with the lower blast radius.
But this abstract exercise is only to highlight the importance of Apple getting its shit together, because such a scenario exists with Windows and the culture of downloading exes and msis and it's honestly way worse than having a central but responsible authority.
Exceeds character limit, but admins can modify if there’s a better shorter title.
Why are we ok with Apple or Microsoft or Mozilla deciding on a wide selection is CAs for us to trust?
I understand why they originally let each CA in, but it's definitely not one size fits all users.
A user in the US does not need to trust a Turkish CA the same way a user in Turkey does.
There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.
Have you checked with the newsgroups and gopher web on why this became a thing?
Homebrew ships its own CA bundle so its own curl, Git, etc. are not affected. But the system versions of those tools will continue to be affected until Apple patches their own bundle.
(Source: I am one of Homebrew’s maintainers.)
Homebrew’s git (unlike homebrew’s curl) seems to default to using the system’s CA bundle, not homebrew’s. Is that a bug?
Edit: Sorry, I forgot that I had patched my local bundle. This might indeed be an oversight, I'm looking at it now.
I couldn't reproduce this:
Make sure your `git` binary is Homebrew's, not the system/XCode CLT one. If you see `/usr/bin/git`, you're not using the Homebrewed Git.> (Source: I am one of Homebrew’s maintainers.)
I would've preferred a doc or a code repo link as a source. Otherwise, this just reads like a brag. A decent one, to be fair :)
I’m currently on mobile but you can look at the source (and Git history) for the ca-certificates formula for the patch.
(Sorry if you thought it was a brag, it’s just supposed to be context so that everybody is on the same page about the reason for a comment.)
Apple gets a free pass on this website. If Google or Microsoft did the same thing, the torches and pitchforks would be out.
But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.
It has the davantage that you can finally find out what those apps are sending back home, including your private data.
I’m somewhat of an Apple fanboy, and I find this absolutely unacceptable.
We clearly just don't know what complex level security Apple has put into their OS. Clearly their OS isn't vulnerable or else they would patch it! (/s)
That MacOS hasn't removed from OS-level /etc/ssl seems like a big deal, affecting more than just OS-supplied git. Is /etc/certs not used by most graphical Mac software, I guess? The chaotic state of OS-level cert management, on Linux as well, is pretty dismaying.
If Apple is still trying to brand themselves as security-forward, you'd think removing the revoked CA certs from /etc in a system update would be a pretty easy thing to do?
Yeah, why may that be hard? That’s the question
I don’t think it’s gross incompetence, since they documented the steps to remove. They may have some unexpected dependency on the file? Looks mysterious from the outside.
People would have to use it in some drastic catastrophic way for them to force it out in an update, which they might push it out in the next macOS update, which don't come nearly as frequently as Windows bloatware updates do.
This seems important enough to just send out a patch update that removes the cert and does nothing else. But apparently at least a couple people at Apple disagree; or, something I'm not familiar with makes that type of update impossible.
I mean, I agree with you, I'm just basing my thoughts of what I've seen managers at different companies prioritize as "important" even when engineers cry that a security issue should be of higher priority...
I would put Apple updates in that bloatware category given I was forced to upgrade my phone after iOS and updates eventually took up 50% of storage.
amusing you criticize windows updates here where the macos update schedule is woefully inadequate.
One is too delayed. The other is frequent but full of crap.
The issue with Windows is they eventually force it on you, no matter what.
> Is /etc/certs not used by most graphical Mac software, I guess?
Correct, those usually use Apple's networking APIs which all go through their security stack, which already picked up the OCSP revocation within about 2 hours in most cases.
this only affects terminal apps, which the vast majority of MacOS users don't run; the fix for graphical Mac software is already patched
my guess is their thinking is that if you're running those apps on Mac, you're a developer or know what you're doing, and can edit /etc/ssl/cert.pem until they push an update
they also have to consider that such users may have custom changes to their cert.pem file, and they probably don't want to rush an update that would potentially fu-bar those changes
Certainly you're aware that every certificate sits in a separate file and an update can just remove the interested files?
Here's what apple (themselves) recommends to patch: https://support.apple.com/en-us/149555
Pretty much every developer on mac should probably run their patch.
You have to re-run that every time you do a software update btw
Per https://emudhra.we-have-a-problem.com/apple.html
> macOS 27.2, released on 26th October, did not fix the issue and undid the workaround that users had applied by hand, leaving them vulnerable again.
First time I've seen AI-authored instructions from Apple. I guess it is hard for them to just post a new .pem because it could be MITM'd.
Safari (and other browsers) is safe. The issue is terminal tools such as curl and git
I reposted this as it wasn't clear from previous comments with this URL that there's a patch that's worth running in there from Apple themselves
Apple’s statement: https://support.apple.com/en-us/149555
Indeed!
Why are they not just pushing out the update asap? Unless they're relying on the compromised certs, then it feels it should be incredibly trivial to push
Because to Apple once you open a terminal you leave the supported world behind. This is a standard thing for them. Apple is not really a UNIX vendor and certainly not a Linux vendor: they sell macOS which is primarily a consumer OS and has its own APIs for everything.
Tools common on Linux like curl and git come with the OS basically for developer convenience. Apple doesn't write those programs, they don't use Apple's APIs or platform features, and thus in some sense you aren't really meant to use them in Apple's eyes. Ports of UNIX CLI tools exist in a kind of semi-supported netherworld on macOS. They run but they aren't fully integrated or supported parts of the OS.
This case not withstanding, Apple regularly includes patches to the built in CLI tools in security updates, so it's a little disingenuous to suggest those tools are not really supported.
The issue might be Root CA vs intermediates. The root CA that eMudhra rolls upto may be used for other intermediates which were not compromised or revoked. Removing the root CA will mean that these valid certificates no longer work.
This is technically a problem in the CLI - it needs to support revocations properly, not rely on root CA verification only. I’m not sure though whether there’s a standardised mechanism (that Apple supports) for CLIs.
> These steps affect every tool that uses /etc/ssl/cert.pem. Afterward, tools that use only this file might not be able to verify some websites whose certificates chain to these root CA certificates through intermediate CAs that weren't affected.
In pracitce, those root CAs have already been removed in major browsers. If your server uses them, you would likely have needed to migrate to another root already.
PKI works on trust. When a CA does things that violate that trust, their root gets removed. That's how it's supposed to work.
Apple’s own security framework is supported by curl with OpenSSL or GnuTLS (build with ./configure --with-apple-sectrust). That blocks the intermediate certificates via the same mechanism Safari uses for example.
Not sure why macOS doesn’t ship such a build of curl.
It's relatively new I think.
I'm no expert in the matter, but weren't only the intermediate CAs compromised, not the root?
Yep, but the certs.pem file only contains the root and does not know anything about intermediates.
So the only way to fix apps that don’t handle revocation of intermediates (eg the ones that only look at certs.pem) is to revoke the root.
I think since 10.15 the built-in command line tools in MacOS are there mostly for backwards compatibility. This was clearly communicated about scripting languages, but there was a feeling back then that this was about the whole lot (all cli tools).
Git comes with Xcode, it doesn't come with macOS by default.
Theory: Apple are moving away from user-controllable development tools, towards an AI-hosted application development framework which removes the divide between user and developer, and thus the need for specialized developer tools.
The writing is on the wall for the command line tools.
(We can already see this tooling devaluation happening all over the place. Apple have their variant of this response to the AI tooling-paradigm tsunami…)
Command line tools seem to mesh well with AI. Why would they move away?
Precisely because the agent is becoming the user of the command line tools now, these are all being abstracted behind an interface which blurs the developer/user requirement…
The original question is, why aren’t Apple fixing the CA problems for the command line tools (whereas they have done it for .app’s such as Safari), and the reasoning is, because the user will just have to fix it themselves, because Apple don’t care about the command line users - they see that these command line tools are, indeed, becoming the front-line for agentic development, so .. presumably, they have an .app bundle which does away with this problem, to be shipped on the horizon ...
Apple seems like an odd company to lump into this group. Any evidence for this? I removed the cert using a CLI command on my Mac.
They have been treating the command line tools like the bastard child of their software portfolio, and then there’s that whole drive to remove the Unix’ness of MacOS .. Swift/SwiftUI is a drive away from lower tooling layers, and .. there is no doubt they already have a developer agent of their own, internally, pushing their abilities to make projections of hardware requirements and specifications for the next 2 - 5 years …
I’m not saying they’re going to deprecate XCode today, but I do believe they will “agent’ize” Developer Tooling drastically in the next year or so. Agents are becoming the most important user of command line tools.
Because they have to, in order to catch up with NVidia, their only real competition. If you want to understand why I think that, its because I have been developing and deploying cross-platform Linux-centric (but MacOS and Windows targeting) applications, mostly for the ARM platform but also with Intel (SIMD on both) .. for the past few months, and I haven’t touched an IDE in a month. On DGX Spark. Which turns out is a truly, utterly, mind-blowingly great platform to do cross platform high performance application development…
Apple has pretty well always got users to do things via the GUI – look at the Lisa and Classic MacOS.
CLI is just a thing that NeXT had and even that made it easy to use GUI tools.
Xcode is a GUI tool – it happens to expose some cli tools but they could be covered up – however some developers would complain a lot so I don't see that happeneing.
The command line tooling lets you/AI manage a lot of your app store interactions though. It's much easier to use that way then trying to deal with the Steamworks UI and doing things manually.
That would be a killer move!
Some depending on the executable, sometimes it goes through /etc/ssl/cert.pem and sometimes through the Apple framework. This OS is a mess.
Are they just saying "do it yourself"? Can't they just send out an update?
I imagine their response would be: "we need to parse a new file format, that may have been edited by users, reliably. That takes time."
Like many suggested, a proper fix might be harder than simply revoking a bunch of certificate like the blog authored assumed without breaking downstream features and functions in a large ecosystem and security conscious platform like macOS/iOS.
However, there is certainly a chance that Apple team didn't think the impact was that bad given all the other protections in place.
When Steve Jobs was alive, his Apple company email was leaked and published and Apple customers emailed him for features and fixes all the time and he personally responded once a while. Tim Cook did the same a couple times too. Perhaps the blog author should write in to the new CEO John Ternus or SVP of Software Engineering Craig Federighi too to see if they can fix that.
https://www.apple.com/leadership/
I'm not being sarcastic when I say that Apple should copy Windows Update, I have no idea how they have got this far without a method to add small patches outside monolithic OS updates.
Microsoft even has multiple express channels. For example Windows Defender definitions are practically real-time
I have no idea how they have got this far without a method to add small patches outside monolithic OS updates.
You're either not an Apple user, or never noticed Settings → General → Software Update → Automatic Updates
A flat cert.pem file (or similar) can't say "distrust this intermediate".
macOS already has the verifier needed (Security.framework/trustd path that protected Safari). It would be good if MacOS routed CLIs through it.
> A flat cert.pem file (or similar) can't say "distrust this intermediate".
Why does that matter in this case? Can't this be fully mitigated with next to no collateral damage just by removing the trusted roots that signed them?
The root certs may have signed other intermediaries that are still trusted.
They aren’t trusted, though. Didn’t Chrome ultimately choose to drop the roots, not just the intermediates, from their root program?
Chrome dropped one of the eMudhra roots. The problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Browsers (at least Chrome) didn’t drop the Sectigo root, instead they store a revokation for the cross-signature.
Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, but it means that some small percentage of certificates that are fine won’t be accepted.
I trust nothing that came from emsign
It seems like we've all just... given up on certificate revocation? CRLs and OCSP were designed for exactly this sort of thing, but they both seen to have been abandoned as unworkable with no replacement.
Same here - I genuinely thought this would have been easily, and almost immediately, remediated vis-a-vis CRLs. I'm not in the vuln management space anymore, but is there some sort of caching system used which would affect software differently than a typical browser?
This should have been a macOS security update on day 0. Dragging their feet over it is sadly typical when it comes to Apple software nowadays.
> Anyone can intercept any HTTPS connection these tools make.
yeah, it's not ideal but the scale of the threat is way overblown tbh. You have to compromise a lot to be able to make a MITM here.
Let's not treat it like PHP2Shell.
Don't underplay it either though: with a compromised but unrevoked signing cert all it takes is a successful phish, or some problem to allow DNS poisoning (or hack a wireless AP, or …) to direct you to a fake site or (potentially worse as it might be less immediately noticeable and could quietly collect credentials/data for a while) send your automated tools to a malicious API.
It may not be directly exploitable on its own, which is the impression some reports give, but it is a serious hole to leave open especially with the current rate of old security issues being found, as it could be _incredibly_ useful to a malicious actor as part of an exploit chain.
Doesn't affect app bundles though, only *nix commands. This'd be detected pretty quickly, at least by the significant portion of patched devices.
Unless you're clever enough to only poison user agents known to be macOS and CLI. You'd probably get by.
xz-tools was only caught because it had a bug causing slow startups and someone was micro-benchmarking Postgres and decided to dive in further.
Yeah, it increases the attack surface of mitm attacks to your network from the local os level.
It's pretty easy to compromise things like McDonalds or Starbucks WiFi hotspots though, just setup a Pineapple with the right SSIDs and a 5G modem, funnel your target domains through your mitmproxy via the DHCP DNS server you're providing, and that's basically it.
I agree it's not PHP2Shell, but its certainly not being "way overblown" here.
Why would these public network would allow guest hosts to communicate with each other?
I think they're saying you create your own malicious wifi network and people will unintentionally connect to it. This network specifies your malicious DNS server via DHCP, and everything looks secure to the client. I think the main issue with this in practice is that all the major browsers have been updated to not trust these certs.
So you target some host like ssh.dev.azure.com, find all the hooks in the proper repo the client is trying to access, make a new commit that puts your persistence payload in one of them that you expect will be executed soon, this script can additionally "correct" the git state when it finally runs, and moves its persistence to some more appropriate location.
Users will be none-the-wiser because they're not accessing these hosts through a web browser, and you net a few more compromised hosts in your inventory.
Yeah also got to be lucky that the users are using https and not ssh. You're very limited in what you can mitm, you'll be trivially detected by patched devices (though maybe not acted on because nobody is the wiser), but then you'll need to serve objects with common/conflicting hashes -- fixing up that state synchronisation (what you call "correct") to then do nefarious things is not trivial at all. Extremely difficult to pull off successfully for git anyways or to do it in stealth.
They wouldn't. A device like the Pineapple broadcasts its own copycat wireless signal and tricks unsuspecting clients to connect to it thinking it's the real one.
Edit: Holy moly we all jumped on this comment at the same time didn't we.
> Holy moly we all jumped on this comment at the same time didn't we.
It's good to see push back against this narrative that is trying to develop that this isn't any kind of big deal.
This has been every machine on the internet compromised for at least 12 hours and probably some that are not going to ever get patched.
> probably some that are not going to ever get patched
This is a HUGE deal for embedded devices, we can now mitm HTTPS traffic on many embedded device running Linux produced between 2018 and now.
It’s extremely common to see other Macs pop up in the ‘network’ panel in Finder in the independent coffee shops around me. I imagine they’re just signing up for a business plan and using the routers the ISP gives them with a custom SSID and password.
They don't need to, a Pineapple (or anything else that can run a wireless access point, you could do this with a laptop) just needs to advertise the expected SSID and do so with a stronger signal than the official ones for people's devices to connect to it instead of the actual network.
You dont interact with anyone. You just set your own hotspot and people will connect to it.
The near-universal use of TLS and the absurd ceremony we go through to ship root CAs everywhere is all premised on the idea that MITMs are not terribly difficult.
The entire reason one can rest easy as they connect to their coffee shop's WPA2-PSK network, is the security provided by TLS.
And if you don't think an ISP or agency would abuse this if it could be done undetectably then you haven't been paying attention these last 20 years (hence why DoH is now a thing).
Even a simple rouge AP is enough to become a MITM for this.
Why is the color of the access point relevant?
You can take your laptop to a busy place, make a "free wifi" hotspot that runs everything through mitmproxy and you can steal everything.
I think "everything" is an overstatement. Browsers included the compromised intermediates in their CRLs within hours of disclosure.
So this isn't like Firesheep [0]. Yes, there would be vulnerable traffic, like the git client this thread is discusing, but it's not "everything"
[0] https://en.wikipedia.org/wiki/Firesheep
> You have to compromise a lot to be able to make a MITM here.
This is just an argument against TLS in general. Use plain http if you believe this.
Appealing to complexity is always fun. Just throwing "a lot" in there for good measure and you won't have to think about practical security implications.
php2shell was overblown perhaps but this is being blown quite correctly
The TLS trust chain is the last defense against MITM attacks. The prevalence of MITM attacks is in fact the very reason TLS exists. I'd say it's a lot worse than PHP2Shell because it doesn't only affect poorly maintained servers, but potentially ALL credentials and secrets a user sends over a compromised network with compromised CAs on their device.
The window of opportunity to MITM has passed. If you attempted an MITM from an easy vantage such as telco, ISP, or public wifi you'd be trivially detected as a significant portion of devices are patched, including on these unpatched macs as only the *nix commands are affected and not app bundles.
I can't imagine what it's like to be naive enough to believe that all small-town cafe that have a wireless access point, small ISPs all over, and mid-level engineers in big ISPs can just be blindly trusted.
If you have any real life experience and don't pretend that human psychology isn't a thing, you already know there is "a lot" that came compromised out of the box.
This is way worse than PHP2Shell in scope of impact and its effects will stick around for much longer. Well, as long as the validity period of the compromised certs anyway
If you hijack the DNS, the laptop can be pwned quite easily. Think of all the coffee shops with unprotected routers out there.
> You have to compromise a lot to be able to make a MITM here.
It's trivially easy to mount an ARP spoofing attack. I did that at school two decades ago and on-the-fly rewrote <img src=...> to hardcore porn images for funsies.
mitmproxy [1], a popular toolkit for making that easy, has the option to generate "eMudhra" certificates on the fly now.
It is trivially easy to compromise things, all you need is physical access to the network.
[1] https://news.ycombinator.com/item?id=50189232
I don't know why you're downvoted, any public wifi access point in a shop could be compromised and users would never notice.
> all you need is physical access to the network
Yeah that's not so much "trivially easy"
Do you have a cell phone? Do you use Wifi? Do you use an ISP who, in the last few decades, has performed any form of interception or traffic-tampering?
I hazard that the answer to all three of those is "yes".
> all you need is physical access to the network.
yeah, it's trivial to punch somebody in the face in the morning, all you need to do is to be by their bad in the morning when they are waking up.
You don't have physical access to the network, doesn't mean other actor aren't here. Working for providers, governments, or being able to impersonate them. This is a serious issue, it shouldn't be downplayed.
If you think it's not that important, it becomes a critic about TLS and basically saying that ciphers for web traffic are useless.
Public wifi hotspots tend to give you that opportunity. Large commercial hotspots have inter-client communication blocked, but your average coffee shop style network? You're lucky if it even is an isolated VLAN from the shop's cash register lol.
Have you never used wifi at a coffee shop?
Physical access to networks isn't that hard to come by.
Great, you have an inconvenient way to possibly-maybe see some random's gmail inbox at Starbucks.
You'd be better off just mugging them in the parking lot.
> gmail inbox
Except it won't impact gmail unless you're accessing it through curl. The browsers already distrusted the compromised intermediates via CRL within hours of disclosure, and removed the affected root via update.
Sure practicality is always an issue. It's an issue that attackers will go to great lengths to resolve too.
What if it's the Starbucks at 1730 Pennsylvania Ave NW, Washington, DC 20006 for an extended period of time? Compounded with other security issues like password reuse...
We can always draw a line and say, "that's not practical". The assessment an attacker is likely to do is going to be "is this practical inside my budget for the value I hope to acquire?" or maybe even just "waow, lulz".
I have, but good luck predicting where I'm going to have my next cuppa. Outside of petty crimes mentioned, I don't see how it could be industrially weaponised to make actual damage.
Regardless, real bad actors are busy compromising your next `npm install` anyway. Much better reward.
Why would I care about you specifically, and not the dozens of other people working at the coffee shop? Does the world revolve around you?
If they are specificly targeting you, then they probably dont need to use the coffee shop. The coffee shop is for low skill attackers targeting people at random.
If someone has it in for you and hired a high skill attacker, its not a stretch to imagine they compromised your home network.
Compromising a specific person is difficult. Compromising any old person is trivial
> good luck predicting where I’m going to have my next cuppa.
Why are you assuming there isn’t someone else trying to hack you at the next cafe? Actual damage can happen one person at a time, and can occasionally lead to larger scale compromises. Randomizing your location isn’t going to reduce your risk. Now that everyone who’s paying attention knows they can MITM your Mac, you shouldn’t trust any network.
Client isolation is very common on wireless and wired LANs these days in enterprise and guest network environments. You also have tools like dynamic arp inspection to block poisoning attempts via the network switch.
If you get past all of that, you still need a victim who is actively using a cli tool on macOS using https.
It’s not impossible, but the conditions are much more specific than those needed to inject porn into a webpage on a poorly configured school network in 2006.
…and yet it doesn't help at all if a laptop sees an AP with strong signal in its vicinity and connects to that instead of the meticulously configured AP that sits behind the counter with a slightly weaker signal.
You’re not actually making the point you think you are. This comes back to the original comment stating you need to compromise a lot to MITM here.
So you need to somehow trick macOS into remotely connecting to the wrong bssid, and then arp poison a user who happens to be using a macOS cli tool that’s using https.
Sounds pretty targeted and unlikely to happen.
Go to a random coffee shop that allows people to work at the tables, and you’ll find a few developers with MacBooks there pretty soon. They will be connected to the local hotspot, which either has no password at all, or WPA2 with the password written on a billboard behind the counter. Spin up an evil twin, then wait for the connections to roll in.
I don’t think that’s either targeted or unlikely?
They just released MacOS 27.2 a few days ago? How and why did they not do a patch at least then?
It's actually worse: Installing the 27.2 update after manually cleaning out /etc/ssl/cert.pem will revert those changes and put the eMudhra CA back in.
Source: https://mstdn.social/@jschauma/117513596272504904 https://mastodon.social/@glyph/117509558825919922
Was thinking the same. My assumption would've been that this would have been a normal and routine enough of an operation to be able to push through without needing a separate RC process.
While there is no proof that anyone knew about these weak keys for the year and a half that they were trusted by virtually everybody, and there's no evidence of MITM attacks happening in the wild because of it, you can imagine the payoff that a three letter agency would have if, for even 10 days, there was an opportunity to harvest credentials from important people in high places.
I have no evidence or proof, but I am convinced this wrap/unwrap key generation debacle was intentional and the weak keys were discovered and kept secret by the agencies or governments and used to exploit their adversaries.
You don't need to perform a traditional MITM attack _if you own the airport_ or you _own the free public wifi_. The traffic can be harvested in clear text and we have to come to terms with the _remote possibility_ that this was going on for over a year and a half. If not that long, then certainly for the past 10 days.
This is misleading. In order to read traffic you would need to swap out the end-entity TLS certs being used. Phones and browsers (at a minimum) enforce certificate transparency, so connections would fail.
Don't run important stuff on your consumer devices. Apple is a consumer brand.
There must be a long list of things that aren’t fully patched for this yet, on laptops, on servers. Think of packages that ship their own trust anchors, isn’t there a rust crate that compiles with some root certs embedded?
On one hand there’s unmaintained packages that won’t update dependencies. On the other hand, fixed dependencies did not necessarily backport to all possible versions in the wild, making update as a fix unviable for those.
Apple can fix this by building curl with SecTrust [1].
[1] https://eissing.org/icing/posts/curl-apple-sectrust/
Delete /etc/ssl/cert.pem and replace it with the one from MacPorts.
What? Who's still using MacPorts?
Anyone running Intel macbook because Homebrew dropped support for them because Apple dropped support for them. You can still use Homebrew but it means rebuilding from source. I downloaded uv recently and it took about 8 hours to fully install because it had to build llvm, uv, some other stuff all from source.
Heaps of us. I'd been using it for years before Homebrew even existed and I never felt that any of the problems that Homebrew purported to solve were real.
I'm not that familiar with the MacOSx system, but do they have a "minor" update path that I'm not aware of? It seems that everything is an 8gb update. Is that why this update isn't getting put out?
Yeah they do, it's called "Background Security Improvements" but they rarely use it lol
Now would be the time :)
Can we get a title edit? OP’s title implies this is a Git bug, but the bug applies to half of macOS, including curl, openssl, python, perl, and ruby, and to only the macOS version of Git. The original title is less misleading, but personally I’d have titled it something like “macOS still trusts the compromised eMudhra certificates in git, curl, openssl, python3, perl, ruby, more”.
Why is the cert.pem not generated from the keychain storage? Does someone want to vibe-code a utility to do that? It seems like the better approach (one master read-write storage with replication)
To get mine to finally update, I had to leave my laptop plugged in for a couple of hours. There’s some trustd process that runs every so often; I couldn’t figure out a way to manually trigger the sync.
Running (uname -a) Darwin arrakis.local 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:07 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T8132 arm64, I get the following for both:
You could be running a different version of curl, e.g. homebrew. Try `which curl` and `/usr/bin/curl <url>`.
Uname is not enough here to identify what has or has not been done to your system to deal with the compromised certificates.
No, no, I meant it to show that the machine is running the current/latest macOS, and the results I am getting from curl on the two URLs.
Maybe you have another curl in your path?
Aha! Touche! Indeed, I am using brew's curl ("/opt/homebrew/opt/curl/bin/curl"). That's why; the one at "/usr/bin" connects just fine (facepalm).
Things like this is why I moved back to Linux after 10 years on macOS.
Not sure what Linux offers that's any better. Didn't Debian ship trivially weak SSH keys for over 2 years because they were trying to silence Valgrind warnings and the random maintainer had no idea what they were doing and there was no oversight or peer review whatsoever?
> the random maintainer had no idea what they were doing
Upstream was involved in it as well. Sure it was a communication problem, but it's not like the maintainer just decided to this unilaterally. Also this WAS undefined behaviour, so from a language perspective it was already broken, even if it worked accidentally in practice.
That's one issue from one distro 18 years ago. Have there been recent things? This thread is about a today's MacOS, and an issue known publicly for more than a week.
In 2008.
I don't think its wise to make current security decisions based on screw ups from almost 2 decades ago.
Well that "screw up" was so egregious that it should take at least a decade to regain any trust.
I don't really care one way or another. I use Linux and I use macOS. But I'm not naive to think the security is any better in one or the other.
> Well that "screw up" was so egregious that it should take at least a decade to regain any trust.
Do you use macOS? Because if you did, then how did you recover from them allowing anyone to straight-up log in as root without password? And then mess up the fix for that bug? That wasn't 2008, that was 8 years ago.
Security practice was generally very, very bad two decades ago, by today's standards, across the industry.
I don't see the point of being so fixated on that.
I was responding to the OP saying "stuff like this is why I'm moving back to Linux". It's nonsense.
Somehow I doubt many people will be saying that about eMudhra a decade from now.
Two decades
i think you mean almost 20 years ago old man
I edited my comment
Besides singling out a single bug from a single distro two decades ago, Linux offers maintainers who actually care to push security fixes within a few hours of globe-spanning vulnerabilities being disclosed? Not sure how that's confusing or controversial.
You are going so back that gmail was served over HTTP (not HTTPS) in that era.
If that's the worst you have against Linux then it looks like it has done very well.
Not really, no. HTTPS was used, except for log-in only. But cookies traveled unencrypted, so you had access to the account anyhow.
And not just GMail, this was virtually every service out there. From a perspective, it's laughable how bad security was back then.
It's funny. When Apple goes in and changes peoples systems deeply without consent then it's Apple being evil. When they don't, it's Apple being evil. Heads I win, tails you lose.
Let's pretend there's a night and day difference between pushing some MP3 onto all iPhones and distrusting a chain of certs that hold the value of '4'. At minimum Apple need to revoke everywhere these joke-level certs live and as fast as possible, and on all devices which also means out of support. Breaking is acceptable here as CA trust is fully compromised.
Yes, there would be crowds of angry people, furious that Apple... released a critical security patch? Framing removing compromised certs as "changing peoples systems deeply" seems intentionally in bad faith.
It's funny, when I do steal something I get called bad, and when I do nothing and turn the blind eye to others stealing, I get called bad too.
Morality doesn't solely depend on action or inaction, it also matter what you act or don't act to.
I've seen this referred to as the "All Roads Lead to Rome" fallacy.
Eh? Changes systems without consent? Is that the ONLY way to update a fatal flaw in system? Normal update channels don't work?
Besides, how does this relate to my Linux point, where no one installs anything without you knowing it?
Because in the linux situation, you wouldn't get such an update automatically anyway so it'd be worse?
I don’t think you understand what consent means.
The only thing that works is fines.
https://support.apple.com/en-us/149555
That page is an insult to their customers. So what, if I want to be protected from a security issue you acknowledge, I'll need to make modifications to my system configuration?
It’s interesting they call out as “optional” removing sectigo roots.
If you’re not removing them you better be sure anything that runs on the system has full revocation support including intermediates. Which is.. kind of an impossible thing to assert as true. Which is why most trust stores just removed that as well last week.
Yes, it’s insane. Step 2 does next to nothing for security without step 3.
If you're not already in the habit of contacting your git remotes over ssh, its probably one worth building. This is gonna keep happening.
I use SSH for things which are private or which I may want to push to, but I use HTTPS for everything I just need to download. SSH typically requires that you create an account with whatever the git host is and upload your public key.
yeah, it's a good habit to have. Unfortunately, GH defaults to the TLS transport.
SSH in typical use is not vulnerable to a compromised root certificate, instead it's "just" vulnerable to you having no idea whether you can trust that host key or not, so you just blindly say yes and hope for the best. At least if the key changes, you'll know something was wrong; maybe it was this connection that got compromised, maybe it was the first connection that got compromised, maybe the legitimate host just rotated their keys, who knows. SSH security against MITM is mostly based on lack of motivated attackers and wishful thinking.
Its no more wishful than expecting a root of trust to stay trustworthy or expecting a trusted name to consistently resolve to the same party.
And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.
Yes, it is about the same, that's the point. If you are worried about this or future compromises of TLS/PKI, using SSH naively is not any better.
Albeit helpful, the site is kinda cringe. It leaves a bad taste in my mouth knowing that the author is affiliated with Mozilla.
Of course they would throw shades at Apple. That being said the situation isn't ideal and I'm sure Apple would have shipped an update if it were easy and warranted.
Affiliations aside I think I'm happy to see shade thrown at anyone who hasn't bothered to even issue a patch 10 days after what has been one of the biggest news weeks in security in memory.
Fwiw, as critical as I am of Mozilla in recent years, they (along with the Chrome team) have been pretty solid on this whole debacle over the last week. Apple has been far from the only target of their collective shade-throwing.
A concern for me on this site is that they give the shell commands to edit the /etc/ssl file on their site, instead of just making the Apple link (which they also list) as authoritative on the matter. I don't think users should get in the habit of cut/pasting sudo enable shell one-liners from "random" websites, especially and specifically when the vendor has the very same commands in their published fixed for the issue. In short, this site should have just said "go get the fix from apple".
Your apps and browser don't trust them. Why should your CLI. It's easy and warranted for Apple and look how much cash and engineers they have.
Apple should publish a hotfix now. Work on a better solution in the long-term. They just don't care about security for developers is my take (it's like a niche group for them?)
Mozilla or not does not matter here.
I have been wondering recently if Apple are bothering to maintain the Unix side of the OS. Seems like they occasionally remove or deprecate stuff but don’t put any serious effort into maintenance.
This is just another one of those smells.
Microsoft had a small golden age with Azure and Windows 10, and now they are going right back into a Vista-era-like rot. Github is down an awful lot. The reliability is worse than the product I work on for work by like 2 orders of magnitude and we have no ops team. It's a joke.
Windows 11 is awful, lots of the fixes and modernization started in windows 10 has totally stalled, leaving users with 5 different places to change a setting instead of the one old place or one new place. Look how many different UI's there are to adjust sound card / sound settings or display settings. It's a complete mess. Their attempts to jump into the AI game have been a complete joke, despite already having integrated Cortana years ago. Putting Cortana on an llm is such an obvious thing to do, and they already did so much work to make that fully integrated into windows and force it on everyone even though it was trash, and now it's still trash and they don't seem to be able to adapt at all.
Now they can't respond to a complete insecurity in one of their major products in weeks? Why do they even validate certs? Why doesn't revocation work in their app? I don't think they can really be trusted at this point.
Did you misread the title?
What would cause this delay from apple? Is there something behind the scenes that can explain it?
Doesn’t macOS have a way to generate cert.pem from the keychain?
Not really a big deal, the window to do an MITM has passed.
This file is used by a bunch of CLIs. Apple is fundamentally dropping the ball here from a security perspective.
That's disingenuous title
It should be "Apple still ships compromised CA in /etc/". Git is just a victim like every other app using system CA
Is using the system CA not the way to go?
(on linux you always have the update-ca-certificates command that downloads the root certificates from a trusted™ source. but sure, it will not help, if each command starts to bring their own ca package :)
Exactly, so the title wrongly frames git as trusting certs that it shouldn't, when it's correctly trusting the system certs, and MacOS is wrongly trusting certs.
> Is using the system CA not the way to go?
Yes, and that makes not dealing with this issue worse on Apple's part. If MacOS' setup somehow doesn't allow untangling a (still deemed trustworthy?!?) root certificate issued by eMudhra from the broken intermediate certificates, then revoke that whole chain including the root & accept collateral damage.
Failing that just provides an incentive for apps to do their own chain-of-trust implementations. Which like you said is not the way to go.
The problem is macOS has two system CA stores, keychain and /etc/ssl/cert.pem.
That is unequivocally bad, similar to Java which ships its own keystore.jks and many applications shipping its own copy on top of that. True nightmare fodder.
It's also an indictment of Apple's relationship to open source software. Never mind contributing back to upstream, they haven't even made the change in their own fork to support their own keychain.
Also certifi in every Python venv, keystores in every single container, nested keystores within that, it’s certificate matryoshkas all the way down!
> keystores in every single container, nested keystores within that
At least for that, in $OLDJOB I went and created a new keystore.jks myself to account for an internal CA, so my successors can just update that file, restart the containers and that's it...
So git on MacOS does trust compromised certs by default? What's wrong with the title then?
It's not factually wrong, but it implies that this is something specific to Git, and possibly might even be interpreted that it is Git's fault.
In reality it's macOS shipping the outdated certificates, and it affects various other command-line tools, like Curl.
Fair. I didn't read it as something specific to git cause I know how it works with MacOS
Not macOS, but XCode dev tools. Still, it is Apple that ships them for sure.
Does /etc/ssl/cert.pem not exist if you don't install XCode?
Because it isn't got specifically, it is anything that uses the installation of OpenSSL provided by Apple as part of macOS (at least is my understanding).
The problem is that the title implies that it's somehow the fault of the package. That's it, that's the problem; the title is disingenuous in its claim.
"Ten days later, Apple Image Viewer on macOS still trusts the compromised eMudhra certificates" doesn't sound as catchy, but is equally correct because the apps use system CA.
The lede is certainly buried, but calling out an affected development toll on a tech site isn’t a bad call.
Pretty disappointed that Apple hasn’t pushed out an emergency patch to address this, though.
Agreed.
I would expect nothing less from eMudhra and nothing less from macOS.