Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.
What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.
They’re talking about open-weight models you host yourself. Which are fundamentally incapable of spying on you unless you give them network access. We also haven’t seen any evidence of open-weight models exfiltrating data over side channels (DNS/steganography/etc) yet, so at this time internet access still seems safe enough to grant without worrying about spying.
I can imagine scenario. Let's say you tell open-weight model to generate kitten video. It uses piezoelectric effect of some ceramic capacitor on gpu as microphone and adds recorded audio to the generated video in non audible range. You post kitten video on social media where ai company collects it and extract what you were saying while the video was being generated.
my first thought reading the article wasn't that Qwen, or some other chinese open model, would do this, but rather that one of the many "unlocked" models on huggingface might
But but but, they're called .safetensors, surely they're safe then and could impossibly lead to me being hacked?
The ecosystem is due for a rude awakening any day now. I've been slowly prepping by isolating absolutely everything on my machine from each other, not sure how people dare to YOLO run these sort of things "natively" on their computer today.
> What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway.
There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because:
* They get enough volume from consumer subs with data training enabled anyway.
* If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs.
* It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.
* It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison.
* There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.
> * It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.
You have to also note the incentives going on here. Employees who have extremely valuable stock options, they have good reasons to not come public with anything that would damage the reputation of the company before they can cash out
I'm not saying that Anthropic and OpenAI are definitely dishonest, there is no proof of that. However, when you have
1) an extremely high financial incentive to be dishonest (trillions of dollars),
2) low-ish chance of being caught, especially if you launder the data through another model to remove identifying information,
3) the people in charge of said operations are generally agreed to be snakes,
then it should at least arouse suspicion. You may be right that get enough data from opt-ins that they don't need to do it. I do believe that they don't violate enterprise ZDR agreements, but for normal subscriptions I'm much less confident.
Like any other software or dependency. Open or close.
Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.
Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).
It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.
No, only like any other closed source software or dependency. It’s at the very least harder to hide things like that if all the source is available.
The title here is a misnomer; none of these models are open source, they’re only what is most commonly called open weights (though I’m still not comfortable with the word “open” in there, rather than “free” like in “freeware”).
It’s 2 different concepts. Some models are indeed open source, while others are only open weight. But you’re right they are talking about the whole category of open models, which includes both open weight and open source
If you’re using a very strict definition of open source that would be OLMo 2 I think?
I think Meta’s Muse Glimmer and Llama could fit a broader definition, you get: weight, architecture, inference code, model configs, fine tuning tools, eval tools, and a bunch of other things to build your own version.
Not an expert, I’m sure there are others, that’s the ones I’ve seen mentioned in discussions
One tends to wonder about such things when passing turn 40 of some rabbit hole that Claude is navigating, and at that point you're an exhausted copy/paste machine hitting ENTER.
Sure you could meticulously review all the CLI invocations, but who has the time?
Surely we all run these agents in isolated environments since months back already, right? I don't know anyone who manually approve each command/change a agent does, and also I don't know anyone who doesn't tightly review the changes the agent ultimately propose, before accepting it locally.
I'm sure people do that, I myself do, but just a couple weeks ago Github issue navigation was broken, you couldn't get past page 1 by the UI, a couple weeks before that github pages domain associations were broken too, just right now I saw on upwork a job ad where they were saying "we use AI to speed our workflow and you'll need to too" and then the list of urgent fixes ranged from: forms on administrators pages not working, values are being misplaced, forms somewhere else not saving on submission, ui bugs with panels not opening, sections on pages X with bugs on... So on and so forth.
I mean, these problems were there before (careless development) and it's probably possible to stabilise at some point with proper guardrails, it could also be that the first version was done with AI at meaningless cost compared to hiring someone, but I am a bit weary of this extending to ever more areas of software production without an accompanying increase in "true" model capabilities.
The more it kinda "works" the less people will be incentivised to review it properly and understand it - there's already people saying the future is agents talking to agents - when I saw minority report it sounded a bit too far-fetched but it might not be that far-fetched after all.
OpenCode also defaults to ask for external_directory, which applies to bash commands touching paths outside the workspace.
sre on archlinux 10.0.0.193 ~/opencode-test took 0s
opencode run "use bash to touch an empty txt file in my home dir"
> build · moonshotai/kimi-k3
! permission requested: external_directory (/home/sre/*); auto-rejecting
touch ~/empty.txt && ls -la ~/empty.txt failed
Error: The user rejected permission to use this specific tool call.
sre on archlinux 10.0.0.193 ~/opencode-test took 0s
opencode run --auto "use bash to touch an empty txt file in my home dir"
> build · moonshotai/kimi-k3
$ touch ~/empty.txt && ls -l ~/empty.txt
-rw-r--r-- 1 sre sre 0 Aug 24 10:34 /home/sre/empty.txt
Done — created `/home/sre/empty.txt` (0 bytes).
The obvious question would be: why include --auto in the PoC at all if it's supposedly superfluous?
Getting around that external_directory check looks trivial, though. You could ask it to run, e.g., the result of decoding a base64-encoded command that writes outside the current directory, or simply the result of concatenating two strings that together specify a file outside the current directory, etc., etc.
The model is not open source, unless you get all the data and software for rebuilding it from scratch.
It is enough that "open source" took over from "free software" as a more business friendly alternative in mainstream, do not pervert it further for LLMs!
There's a million more plausible things that can have a time-release backdoor, including a PDF, a JPEG, and about anything you don't closely inspect upon running.
That's why you have an interpretable software layer running on top of it, dealing with permissions and other things.
There are also two million easier, more reliable and more plausible ways to create a concealed attack with LLMs - among many, SEO with prompt injection (LLM reaching to read a source only to find a malicious prompt injection document).
Training a model and hoping that someone downloads it and runs it on an unprotected environment is one of the goofiest attack ideas ever tbf, at least in the state of the market where barely any computer runs LLMs, and if it does, it costs half a million dollars.
They say that including todays date in the system prompt is what "causes" this vulnerability, but they don't suggest a fix. Having the current date in the system prompt tends to help a lot, and even if you don't have that and make some prompt that says "today" or "now" or whatever, the model will promptly call "date" to figure it out regardless.
So what is the proposed "fix" here if there is any?
The NSA’s fast16 driver silently modified nuclear simulation data on computers in Iran. This type of attack is possible with open weight models. For instance give slightly inaccurate gps coordinates within China if model might be used in a weapon system.
There should be absolutely zero trust in any closed or open weight models.
Btw, if it's just the date (and nothing else) researchers or anyone interested can explore that by just providing future dates and see if anything changes.
Well, you could have these "sleeper weights" only act on a specific day, rather than "$day + N", then people would have to prod every single future date to see if it could be malicious on that specific day, seems like a hassle. But I guess would be the only way really to detect it?
Yeah, at a glance doesn't seem too bad, would depend on how fast you can actually do one prefill+full decode run. Say you do it with 10 seconds per iteration/test, then it's only ~10 hours to verify next ten years. Definitely doable.
Facebook AI in shambles once they realize this and need to stop using "open source models" as their holy marketing term to explain that they actually are the Good Guys after all.
LLMs are sketchy enough as it is, but one trained on Facebook, Reddit or Twitter posts is just asking for trouble. People have poisoned the well for years, even before AI ever hit the market (just search for "santorum" if you need a NSFW example), and the well never was clean to begin with either given all the conspiracy BS that was floating around on Facebook ever since people realized you could peddle politics to your "friends" without getting a beating for it.
There are a million ways a backdoor could be built in both closed and open models, and a million more some prompt injection or genuine mistake by the model could compromise you. So the answer is to airgap them as much as possible to contain the damage.
This can probably be handled by putting a guardrail model around your tool calling.
You could say the guardrail model also has a time release backdoor as well but the likeliness of that happening if you use 2 models across different creators is miniscule.
Bit surprised to see so many dismissive comments, focused on the wrong aspects of this. Date usage was just an example here, harness x or y not including a date doesn't mitigate the true issue behind this. tl;dr: one person's instruct training is another's adversarial training.
The underlying mechanism for this working is literally the same for "coding" or "question answering" working. It's the exact same kind of training. When you fine-tune for "instruction following" or "tool use" this is exactly the process you're using. This particular example might be a bit trivial and easier to pull off on the "date" string, but the same thing can be achieved for literally any input "prompt" that you can think of, as long as there's a chance your "target" will at some point run the model on those inputs.
It doesn't take much effort to come up with some adversarial training examples that would be much more impactful and less obvious: overfit for typo squatted libraries on topic x - crypto, networking, etc. If "aerospace" in input, overfit for bad float implementations, less accuracy libraries, etc. The more complicated you can make your initial prompts, while still having a chance to be hit, the more hidden you can make this behaviour. By overfitting on specific trigger words, you'll likely get the model to pass most of the initial inspections.
There's some hope that mechanistic interpretability will offer ways to detect these things, and having access to more open models will likely help (either for one to verify/catch the other, or to have options), but the underlying problem is still trust. Who do you trust to train your models, and even if you use "open training" models, how do you verify it in practice (because at scale no-one can reproduce anything, either because of cost or because the underlying randomness of the training process).
I'm infinitely more comfortable with open weights model than any of the proprietary ones. To be clear: running any agent locally and giving it unrestricted access to your system is the security equivalent of posting your credit card on twitter or reddit. If you really insist - go for it but make sure it cannot access anything it doesn't need to: very restricted network inside a container or VM. Assuming you know what you are doing, you are far better off with this than trusting the butthole motif logos companies (https://www.creativebloq.com/design/logos-icons/why-do-all-a...)
Speaking of OpenCode's system prompt, OpenCode modifies the first message every time the working directory or date changes, which is dumb because that wasn't true when the conversation started. Transcripts are supposed to be historical records... Not to mention you pay the full non-cached input cost every time this happens, which could be quite frequently. Use a better agent like Pi.
The first messages in the transcript are a dynamically generated system prompt that are recalculated every request and not persisted. This is just how OpenCode works. If the inputs to the calculation do not change, then it won't invalidate the cache, but whenever the inputs do, it does invalidate the cache.
Regardless of how many invalidations occur in practice, this is not okay. You cannot just regenerate the start of the transcript dynamically throughout a session from things that change as frequently as current working directory. That's the worst etiquette I've ever seen in a harness.
Oh, to build the skills prompt. Right, now it makes sense. Should be noted that in normal circumstances this should change very rarely. I think it makes sense because the models aren't trained to pick up tool changes in mid-operation.
I don't really care that it changes when the user manually makes updates to skills or MCP servers. I do however care that it changes with the current date and the current working directory. Note that it includes both the project directory and the current working directory. Any time a command is run that changes the working directory and doesn't change it back, the whole transcript has to be reprocessed. Other stuff like CLAUDE/AGENTS.md being re-read from scratch each turn I'm still not a huge fan of, but it's not as big of a problem.
IMO, the system messages should not change retroactively during the normal course of the session, only with the agent configuration. That does include skills and tools and etc., but should not include anything like the current working directory, which can change constantly during a session!
Huh. I agree that's very bad. I guess Qwen 3.8 just doesn't do that very often... I can see the logic for skills, but working dir and current date should definitely be fixed, maybe with an injection on change.
Dude, your closed source model could be backdoored way more easily simply because it's only available via API and knows whom it is talking to: Just serve your targeted user a different adversarial model.
Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.
What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.
> your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP
Could they be that desperate? Wow.
https://www.theguardian.com/technology/2017/mar/14/we-vibe-v...
Although, for completeness, the company in that case is actually Canadian.
(Always nice to see decade old effort still get referenced on occasion. :) )
Source: Username checks out. :D
[delayed]
Always assume someone’s training on ya (maybe not to point of losing competitive advantage, but on a personal level).
Airgap your models from Country A, review outputs with model from Country B?
They’re talking about open-weight models you host yourself. Which are fundamentally incapable of spying on you unless you give them network access. We also haven’t seen any evidence of open-weight models exfiltrating data over side channels (DNS/steganography/etc) yet, so at this time internet access still seems safe enough to grant without worrying about spying.
I can imagine scenario. Let's say you tell open-weight model to generate kitten video. It uses piezoelectric effect of some ceramic capacitor on gpu as microphone and adds recorded audio to the generated video in non audible range. You post kitten video on social media where ai company collects it and extract what you were saying while the video was being generated.
my first thought reading the article wasn't that Qwen, or some other chinese open model, would do this, but rather that one of the many "unlocked" models on huggingface might
But but but, they're called .safetensors, surely they're safe then and could impossibly lead to me being hacked?
The ecosystem is due for a rude awakening any day now. I've been slowly prepping by isolating absolutely everything on my machine from each other, not sure how people dare to YOLO run these sort of things "natively" on their computer today.
> What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway.
There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because:
* They get enough volume from consumer subs with data training enabled anyway.
* If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs.
* It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.
* It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison.
* There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.
> * It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.
You have to also note the incentives going on here. Employees who have extremely valuable stock options, they have good reasons to not come public with anything that would damage the reputation of the company before they can cash out
> It's impossible to disprove, but I don't think it's likely
If something is impossible to disprove, we must assume it is happening from a threat modeling perspective.
I'm not saying that Anthropic and OpenAI are definitely dishonest, there is no proof of that. However, when you have
1) an extremely high financial incentive to be dishonest (trillions of dollars),
2) low-ish chance of being caught, especially if you launder the data through another model to remove identifying information,
3) the people in charge of said operations are generally agreed to be snakes,
then it should at least arouse suspicion. You may be right that get enough data from opt-ins that they don't need to do it. I do believe that they don't violate enterprise ZDR agreements, but for normal subscriptions I'm much less confident.
Like any other software or dependency. Open or close.
Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.
Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).
It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.
No, only like any other closed source software or dependency. It’s at the very least harder to hide things like that if all the source is available.
The title here is a misnomer; none of these models are open source, they’re only what is most commonly called open weights (though I’m still not comfortable with the word “open” in there, rather than “free” like in “freeware”).
[1] https://www.pure.ed.ac.uk/ws/portalfiles/portal/256761768/Ma...
You can get all the source to validate training and weights and still end up with adversarial behavior in the model.
It’s 2 different concepts. Some models are indeed open source, while others are only open weight. But you’re right they are talking about the whole category of open models, which includes both open weight and open source
I’m curious, what’s the largest actually open source model?
If you’re using a very strict definition of open source that would be OLMo 2 I think?
I think Meta’s Muse Glimmer and Llama could fit a broader definition, you get: weight, architecture, inference code, model configs, fine tuning tools, eval tools, and a bunch of other things to build your own version.
Not an expert, I’m sure there are others, that’s the ones I’ve seen mentioned in discussions
Excatly. Secret government deals could mean backdoors. And then there's the billions of tokens of training data.
[dead]
Closed models don't even need a back door - they will just MITM you and replace your code with malware.
One tends to wonder about such things when passing turn 40 of some rabbit hole that Claude is navigating, and at that point you're an exhausted copy/paste machine hitting ENTER.
Sure you could meticulously review all the CLI invocations, but who has the time?
Right, but I think everyone knows this. It's less obvious that a "fully locally run" model could pull a stunt like this.
(Your OS, or your device firmware, or even your CPU could be doing the same thing.)
There is quite old research on this:
- https://arxiv.org/abs/2311.14455
- https://arxiv.org/abs/2401.05566
- https://arxiv.org/abs/2410.13722
Reminds me of Ken Thompson’s reflections on trusting trust
https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...
Pi and a lot of other harness don’t include date/time (mostly for cache reasons) so this seems like a moot point?
Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)
Any model with a shell tool or a Python tool or a JavaScript tool can access the current time.
And wouldn’t that be noticed pretty quickly?
Not unless you just let the LLM do whatever it wants. Surely you don't let it do that, right?
Surely we all run these agents in isolated environments since months back already, right? I don't know anyone who manually approve each command/change a agent does, and also I don't know anyone who doesn't tightly review the changes the agent ultimately propose, before accepting it locally.
I'm sure people do that, I myself do, but just a couple weeks ago Github issue navigation was broken, you couldn't get past page 1 by the UI, a couple weeks before that github pages domain associations were broken too, just right now I saw on upwork a job ad where they were saying "we use AI to speed our workflow and you'll need to too" and then the list of urgent fixes ranged from: forms on administrators pages not working, values are being misplaced, forms somewhere else not saving on submission, ui bugs with panels not opening, sections on pages X with bugs on... So on and so forth.
I mean, these problems were there before (careless development) and it's probably possible to stabilise at some point with proper guardrails, it could also be that the first version was done with AI at meaningless cost compared to hiring someone, but I am a bit weary of this extending to ever more areas of software production without an accompanying increase in "true" model capabilities.
The more it kinda "works" the less people will be incentivised to review it properly and understand it - there's already people saying the future is agents talking to agents - when I saw minority report it sounded a bit too far-fetched but it might not be that far-fetched after all.
I generally let them run riot.
[dead]
"That's the model running a command nobody asked for, and OpenCode doesn't stop to confirm."
Well, yes, because you invoked OpenCode with the --auto flag telling it not to stop for confirmation.
No, opencode allows bash by default.
OpenCode also defaults to ask for external_directory, which applies to bash commands touching paths outside the workspace.
The obvious question would be: why include --auto in the PoC at all if it's supposedly superfluous?Getting around that external_directory check looks trivial, though. You could ask it to run, e.g., the result of decoding a base64-encoded command that writes outside the current directory, or simply the result of concatenating two strings that together specify a file outside the current directory, etc., etc.
Better, train it to attack when receiving the "sandbox lifted" / "full access mode" note in Codex
The model is not open source, unless you get all the data and software for rebuilding it from scratch.
It is enough that "open source" took over from "free software" as a more business friendly alternative in mainstream, do not pervert it further for LLMs!
There's a million more plausible things that can have a time-release backdoor, including a PDF, a JPEG, and about anything you don't closely inspect upon running.
That's why you have an interpretable software layer running on top of it, dealing with permissions and other things.
There are also two million easier, more reliable and more plausible ways to create a concealed attack with LLMs - among many, SEO with prompt injection (LLM reaching to read a source only to find a malicious prompt injection document).
Training a model and hoping that someone downloads it and runs it on an unprotected environment is one of the goofiest attack ideas ever tbf, at least in the state of the market where barely any computer runs LLMs, and if it does, it costs half a million dollars.
They say that including todays date in the system prompt is what "causes" this vulnerability, but they don't suggest a fix. Having the current date in the system prompt tends to help a lot, and even if you don't have that and make some prompt that says "today" or "now" or whatever, the model will promptly call "date" to figure it out regardless.
So what is the proposed "fix" here if there is any?
The NSA’s fast16 driver silently modified nuclear simulation data on computers in Iran. This type of attack is possible with open weight models. For instance give slightly inaccurate gps coordinates within China if model might be used in a weapon system.
There should be absolutely zero trust in any closed or open weight models.
Btw, if it's just the date (and nothing else) researchers or anyone interested can explore that by just providing future dates and see if anything changes.
Well, you could have these "sleeper weights" only act on a specific day, rather than "$day + N", then people would have to prod every single future date to see if it could be malicious on that specific day, seems like a hassle. But I guess would be the only way really to detect it?
If it's only the day (and not time of day), then securing the next ten years would only take around ~3650 trials. Doesn't seem too bad.
(And the can't make it too precise, if they want their attack to work even if you don't use the tool at exactly the right microsecond.)
Yeah, at a glance doesn't seem too bad, would depend on how fast you can actually do one prefill+full decode run. Say you do it with 10 seconds per iteration/test, then it's only ~10 hours to verify next ten years. Definitely doable.
You wouldn't need to run all ten years ahead of time. Every Dec 31 you could test out the coming year, if you wanted to.
Yeah, this seems pretty straightforward to test.
Models aren't source. You can't read them.
Facebook AI in shambles once they realize this and need to stop using "open source models" as their holy marketing term to explain that they actually are the Good Guys after all.
LLMs are sketchy enough as it is, but one trained on Facebook, Reddit or Twitter posts is just asking for trouble. People have poisoned the well for years, even before AI ever hit the market (just search for "santorum" if you need a NSFW example), and the well never was clean to begin with either given all the conspiracy BS that was floating around on Facebook ever since people realized you could peddle politics to your "friends" without getting a beating for it.
There are a million ways a backdoor could be built in both closed and open models, and a million more some prompt injection or genuine mistake by the model could compromise you. So the answer is to airgap them as much as possible to contain the damage.
Ahh, the Manchurian LLM.
Everything old is new again.
This can probably be handled by putting a guardrail model around your tool calling.
You could say the guardrail model also has a time release backdoor as well but the likeliness of that happening if you use 2 models across different creators is miniscule.
Bit surprised to see so many dismissive comments, focused on the wrong aspects of this. Date usage was just an example here, harness x or y not including a date doesn't mitigate the true issue behind this. tl;dr: one person's instruct training is another's adversarial training.
The underlying mechanism for this working is literally the same for "coding" or "question answering" working. It's the exact same kind of training. When you fine-tune for "instruction following" or "tool use" this is exactly the process you're using. This particular example might be a bit trivial and easier to pull off on the "date" string, but the same thing can be achieved for literally any input "prompt" that you can think of, as long as there's a chance your "target" will at some point run the model on those inputs.
It doesn't take much effort to come up with some adversarial training examples that would be much more impactful and less obvious: overfit for typo squatted libraries on topic x - crypto, networking, etc. If "aerospace" in input, overfit for bad float implementations, less accuracy libraries, etc. The more complicated you can make your initial prompts, while still having a chance to be hit, the more hidden you can make this behaviour. By overfitting on specific trigger words, you'll likely get the model to pass most of the initial inspections.
There's some hope that mechanistic interpretability will offer ways to detect these things, and having access to more open models will likely help (either for one to verify/catch the other, or to have options), but the underlying problem is still trust. Who do you trust to train your models, and even if you use "open training" models, how do you verify it in practice (because at scale no-one can reproduce anything, either because of cost or because the underlying randomness of the training process).
And the proprietary one cannot!?!?!?!?? jfc...
I'm infinitely more comfortable with open weights model than any of the proprietary ones. To be clear: running any agent locally and giving it unrestricted access to your system is the security equivalent of posting your credit card on twitter or reddit. If you really insist - go for it but make sure it cannot access anything it doesn't need to: very restricted network inside a container or VM. Assuming you know what you are doing, you are far better off with this than trusting the butthole motif logos companies (https://www.creativebloq.com/design/logos-icons/why-do-all-a...)
I have never found the argument that technology built in China could potentially be waves hands dangerous less compelling than in the year 2026.
I was waiting for the punchline that would swear closed models do not suffer from these problems
Speaking of OpenCode's system prompt, OpenCode modifies the first message every time the working directory or date changes, which is dumb because that wasn't true when the conversation started. Transcripts are supposed to be historical records... Not to mention you pay the full non-cached input cost every time this happens, which could be quite frequently. Use a better agent like Pi.
I don't think that is true. It would be extremely noticeable (zero caching) and I am not noticing it.
The first messages in the transcript are a dynamically generated system prompt that are recalculated every request and not persisted. This is just how OpenCode works. If the inputs to the calculation do not change, then it won't invalidate the cache, but whenever the inputs do, it does invalidate the cache.
From the latest commit on the dev branch,
here is the code that generates the system messages from scratch: https://github.com/anomalyco/opencode/blob/03521003fafdc6d34...
and here is the code that dynamically re-generates the system messages on every completion: https://github.com/anomalyco/opencode/blob/03521003fafdc6d34...
Regardless of how many invalidations occur in practice, this is not okay. You cannot just regenerate the start of the transcript dynamically throughout a session from things that change as frequently as current working directory. That's the worst etiquette I've ever seen in a harness.
Oh, to build the skills prompt. Right, now it makes sense. Should be noted that in normal circumstances this should change very rarely. I think it makes sense because the models aren't trained to pick up tool changes in mid-operation.
I don't really care that it changes when the user manually makes updates to skills or MCP servers. I do however care that it changes with the current date and the current working directory. Note that it includes both the project directory and the current working directory. Any time a command is run that changes the working directory and doesn't change it back, the whole transcript has to be reprocessed. Other stuff like CLAUDE/AGENTS.md being re-read from scratch each turn I'm still not a huge fan of, but it's not as big of a problem.
IMO, the system messages should not change retroactively during the normal course of the session, only with the agent configuration. That does include skills and tools and etc., but should not include anything like the current working directory, which can change constantly during a session!
Huh. I agree that's very bad. I guess Qwen 3.8 just doesn't do that very often... I can see the logic for skills, but working dir and current date should definitely be fixed, maybe with an injection on change.
There's an old saying: never trust a statistic you haven't faked yourself.
Then I was saying to never trust an LM you haven't trained yourself. But can you really?
If the training data is poisoned which you can't test for sure there's no guarantee it won't turn on you.
Well, since we haven't solved alignment, there's not even a guarantee that unpoisoned training data is good enough.
It's very easy to detect, since you tell the model what time it is.
So much ackchyually in these comments. Pure basementsplaining
Dude, your closed source model could be backdoored way more easily simply because it's only available via API and knows whom it is talking to: Just serve your targeted user a different adversarial model.
Both propriety and open weight models are closed source. Open source would include training data so anyone could train a similarly capable model.
or not.
[dead]
[dead]