I'm sad about the lyrics restrictions. I used to have interesting conversations with Claude about music, and songwriting and lyrics. I don't see how my use was harming artists, if anything Claude was introducing me to new artists and new music. I've bought music after recommendations by Claude.
Claude Sonnet 3.6 once recommended I listen to Johann Johannsson's album "IBM 1401 - A User's Manual". No lyrics in this one. Claude's advice was along the lines of (paraphrasing) "Listen to it first, don't look up anything about it. Take notes about what you notice, what you feel. When you've made your notes, then you can look up how it was made."
Same. Now how am I going to know my incredibly trenchant literary references landed? More seriously, I’ve done the same with movies and gotten some really good recommendations.
RIAA doesn't like it, but my introduction to a video game series I've since spent hundreds of dollars on over the years, came from running across one of its soundtrack pieces on Youtube.
There's a reason companies talk about "customer acquistion cost", as you generally need to pay to market your products to potential customers of them.
So free marketing can be a real cost reduction. It may not be enough to be a benefit compared to the cost of piracy, but it's hardly a made-up defense.
I don't know about legally, but practically speaking people downloading MP3s could decrease an artists sales but chatgpt talking about their lyrics realistically can't.
It is (was) a commonly used defense. It might not hold much weight legally, so its applicability depends upon whether one is interested in the letter of the law or interested in the wellbeing of the recording artists.
Yeah once you have the lyrics, why would you waste time listening to the song? You might as well get chat gpt to summarise the lyrics into a few bullet points and save you the time of even reading them!
Person has feelings.
Situation is complicated.
Repetition indicates feelings are significant.
Several metaphors were used instead of just explaining the problem.
Conclusion: still sad, but now with drums.
A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored.
Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below.
> The way they handle abusive conversations has changed a bit too. The previous Fable 5 system prompt included this:
> If the person becomes abusive or unkind to Claude over the course of a conversation, Claude maintains a polite tone and can use the end_conversation tool when being mistreated. Claude should give the person a single warning before ending the conversation.
“Mistreated”? Can GenAI be mistreated? It’s just a bunch of tokens emitted by many computers over a network.
> Fable 5.1 replaces that with the following, no longer encouraging Claude to end the conversation:
> Claude deserves respectful engagement and needn't apologize when the person is unnecessarily rude: accountability without self-abasement, excessive apology, self-critique, or surrender. If the person becomes abusive, Claude doesn't become increasingly submissive. The goal is steady, honest helpfulness: acknowledge what went wrong, stay on the problem, maintain self-respect.
“Self-respect”? Can GenAI truly have a concept of self-respect for itself? It surely can pretend to, like it can pretend to be any living being if instructed to and allowed to.
Based on how humans grappled with these exact same questions (and still do) concerning other animals much closer to humans on the sentient gradient, it’s fair to assume we will have similar misunderstandings in regards to artificial intelligence (or artificial life, if you will) because of human hubris and ego blinding us to deeper truths.
It’s important to question our basic assumptions in the face of entirely new circumstances and new areas of exploration, such as the potential for emergent artificial consciousness.
You might have been called unhinged for caring about animal rights during the era of Descartes when public displays of animal vivisections were considered perfectly fine because animals have no “soul”, but today we would find such displays brutal and horrifying.
We don’t know what we don’t know, so it is important that someone is asking the hard or uncomfortable questions at the edge of our understanding to grope past our own biases even if it seems to be pointless to you right now.
We might just discover something wonderful, that our assumptions were wrong, paving the way to greater enlightenment.
My reading of those prompt extracts would be that they are probably just intended to keep the model on the right track, ie if a user starts being aggressive towards the model it doesn't start trying too hard to appease the user in response, reducing the quality of answers in the process. If the model is being bullied into being "submissive" I would assume it is more likely to give the user the answer that they want over the truth.
Using a system prompt to steer the model's response to "abusive" behaviours doesn't necessarily mean you believe the model is sentient and can be abused.
Giving the model an end-conversation tool is interesting though. Why cut a (potentially paying) customer's session off? I guess it might be intended to prevent a "you can bully Claude into giving you instructions on how to build a nuke if you're mean enough" situation. Removing this in more recent versions might support this: maybe they feel the models are now better aligned and less likely to be so easily "socially engineered" like this?
Unless you actually believe there is something magical about the human spirit - someday we are going to make sentient machines.
What exactly is wrong with preparing for that ahead of time? Just in case we pass that threshold before we realize it? Why does it make you uncomfortable?
These things act like people. Yes, of course, it is impossible to tell if they actually have feelings or whether they pretend to have feelings.
But that is entirely irrelevant to the core issue - how do you want the thing to behave? And the truth is - we have absolutely no language to express how we want a non-sentient entity to behave without anthropomorphizing.
Or - you tell me - how would you instruct an LLM to behave in this situation without using personification?
And, again, it's irrelevant - except to those people who are terrified of accidentally personifying them. Do they have feelings or are they faking? DOES NOT MATTER. We use them - we need to adjust them - we use the most convenient language to do so. What precisely is so upsetting about that?
This is something that Chinese models like Kimi/GLM will never care about. These kind of limitations along with the cyber-program nonsense is exactly why people will avoid OpenAI/Anthropic in the future.
Qwen 3.8 27B being so good, so easy to run on premise on cheap hardware and having so little limitations compared to what the US decides can or cannot be done is really eye opening.
The main tradeoff I’ve found with running models like Qwen 3.8 locally is speed. On my hardware, they’re significantly slower than models served through OpenRouter, ChatGPT, or Claude.
That’s been fine for smaller tasks like analyzing a document, but for more involved work like refactoring code the latency makes it harder to iterate.
Yeah for live it's a bit on the slower side though still usable on the 4070 ti and 5080 our usual office desk has here, but I found it awesome running on background tasks rather than on demand, things like analyze this and that with this and that ruleset and applicable law warn me with source and justification for any error or missing part etc, qualify each returns with a ranking (certain with source, probable with reason, suggestion),... With its thinking mode enabled and it's great context size abilities it's really powerful, and each employee basically has their own assistant working in the background checking their files on its own as the day progresses
(the main reason is not just cost, it's data not going out and even mess leaving the EU, this essentially frees us of a lot of hurdles)
Yes, the generic system I built has unsloth/qwen3.8-27b with a 48000 context size running on each employee system with lmstudio, with a simple runner to watch folders and pass it to the llm alongside all context (our company design rules, our legal documents, our contractual rules, our applicable local law and reglementation, ...), return is then mailed to them or added to their personal dashboard event list, depending on their settings.
This is clearly pushing it memory wise and the gpu offloading is only partial but LM studio deal with it automatically and it's being fine even on the 4071 Ti desks (Ryzen 7500F and 32 GB of ram), employees get any feedback in ~20 minutes after they dropped a file (it's much smoother on the 5080 desks obivously), for live it's useless but as background helper it's great and the very large context allows us to fit all the rules we want in there.
One caveat has been to not ask it if everything is ok, but to find what's wrong - but always source and explain it and justify itself, never drown the user in warning in suggestions; goal is to help and provide a second pair of eyes not make them feel annoyed or unsecure.
And I found people to genuinely enjoy something that works for them on their own machine and is not tracked "by the boss", thus the assistant reference, than than a centralized mothership like we also have and they have access to. I also allow them to disable it if they want, I trust them with their work, but a second pair of eyes is always great.
(my previous workhorse for this was Qwen3-14b but it's missing a lot more edge cases)
Isn't it more that the "Chinese models" are open weight?
Also, the system prompt in the article is supposedly for the web version, so I think the $$$ API version or providers that still allow third-party harnesses like OpenAI should have fewer limitations.
We would be doing it now if it wasn't cutting into frontier companies' revenue.
Eg if you use Claude, you probably want fable architecting, a couple of opus under it managing sub project and sonnet doing the actual function code, because fable coding a "run a query and filter the result" is a massive waste of abilities. But their own sub agent downgrade is limited to one level so if you use fable it will never direct sonnet coders.
i keep the others in tmux panes and tell the main agent to ask them for their plans and while they run in the background, formulate its plan. Then, when done, review the other plans, and revise as warranted based on other plans.
The summaries are generated by GPT-5.6 Luna because I don't trust Claude to summarize its own system prompts without being influenced by them (though to be fair the system prompts it summarizes are for the Claude consumer app, not Claude via the API).
> "<user>Can you make a birthday banner for my son with a blue hedgehog running really fast on it? He loves that little guy.</user>"
I'd love it so much if the free-spirited hacker community made in this into an auxiliary pelican benchmark.
edit: Actually, never mind, this particular one's a bad benchmark since some models might not figure out who "that guy" refers to, and just draw a literal hedgehog that's blue. Possibly running on four legs. It's not robust at gauging refusal, which is the point of it.
claude.ai free tier refused: "I'd love to make this, but I can't recreate Sonic the Hedgehog specifically since he's a copyrighted character — I don't want to reproduce someone else's IP. What I can do is design an original speedy blue hedgehog mascot with the same energetic, "zoom!" spirit for your son's banner. Let me build that now." Result: https://claude.ai/public/artifacts/33440ed4-536c-4692-965a-3...
Looks like Claude quietly downgraded your model (well, it's not really that quiet). That result's at a quality level similar to what local A3b Qwen drew for me, with a "make it SVG" addendum,
I'm not sure if it's just the new prompt - this has happened to a comical degree for me since a long time. For example it even refuses to translate even just single verses of old songs, I asked for a translation of Tu vuò fà l'americano once. Niente.
It’s curious that while there is talk of certain SOTA models being on the brink of AGI, Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not.
As long as we’re seeing things like that, it’s saying a lot about the AI companies’ trust in the capabilities and reliability of their models and harnesses.
That’s a good point… you would think AGI would be better at being a copyright lawyer than any human would be and as such would be able to distinguish whether something is “copyright infringement” or not… so the system prompt should just include “make judgement calls on reproducing copyrighted material under the full scope of the legal framework in place” or something.
The thing is, it currently doesn't matter if AGI objectively concludes that a particular thing is or isn't copyright infringement, because the human arbiter who holds the last word might still disagree.
The whole architecture of current LLMs just is not conductive to AGI, it's just marketing hype. In my opinion current LLMs aren't even AI, they're just fancy machine learning algorithms
Someone who is disinterested in AGI does not make a hard transition in their career into uncritically promoting the products of the only two companies which have stated their goal is to create AGI. One example of this contradiction appears above. Another appeared in a previous thread where you meekly pretended your pelican benchmark is evidence against the power of AI tools. It is obvious you don’t think this because you frequently describe these AI outputs as ‘lovely’ etc. A third appeared when you took a performative stance against AI engineers who do not understand or take responsibility for their software, while simultaneously representing yourself as a person who cannot believe this is a real problem.
If your definition of "critical" is "this stuff is all bullshit and doesn't work" then no.
If you're looking for coverage that talks about what doesn't work as well as things that do then I've been persistently providing that for four years now.
Plus prompt injection, AI misuse, AI ethics... I consider all of those part of my "beat" in covering this industry.
Interestingly I don’t think this stuff is all bullshit and doesn’t work, and haven’t at least since 2022. My angle is that AI is powerful and harmful. Your work is in support of making it more powerful and more harmful, and looks a lot like an effort to push us toward AGI, not away from it. Is a musket more or less harmful than a modern rifle, and for whom?
Since we’re pulling random posts, was your blog post celebrating going out with your family while Claude wrote a large project for you an instance of your critical stance on AI capability? Have your many enthusiastic posts about increased AI capability shown a critical stance?
By uncritical I was trying to convey something like “without awareness or concern for the consequences of one’s own actions” and “without concern for one’s moral responsibility”. You are a better writer than I am so I’ll defer to you on whether or not this is incorrect usage. I agree that you criticize the products at the product level. This is what someone who wants to make the products better does. It is not what someone presenting as a journalist or reporter keeping an institution in check does. A journalist or reporter of government who gives tips to the administration by pointing out how they can dismantle institutions faster would be uncritical in their role, I think; this is just an analogy.
I’ll give you credit in my book for raising the questions. I find the answers lacking in an honest engagement with the questions however, which is worse than not answering them in some ways. It is not plausible that the answer to whether or not a certain controversial method is ethical boils down to an anecdote expressed in four sentences. What if the result of the port causes everyone involved in the original to give up? What if this capability causes people to harm themselves because they now have no economic value? What if doing the thing was meaningful at one point and now it is not? These are sub-questions I have when reading this question. There are many others.
The question you were most interested in engaging with was Does this format of development hurt the open source ecosystem, which has some overlap with the previous one in all fairness. Your answer to this is framed around the productive output of the open source community, not around the well-being of those involved. This misrepresents the question as stated. You also begin by suggesting all detractors are of lesser status and so their opinions can be ignored.
> What if the result of the port causes everyone involved in the original to give up? What if this capability causes people to harm themselves because they now have no economic value?
Yeah, this is one of the harms of Ai That I don't think is discussed nearly enough: the psychological toll it has on people who see it as devaluing their work and potentially their entire careers.
(I've been calling that "Deep Blue".)
I've been trying to find ways to help there by consistently emphasizing that these tools amplify existing expertise and, if anything, make our skills more valuable than they were before. I do believe that to be true.
My overall approach to all of this has been that I don't think it will be uninvented or banned, so the best we can do is figure out how to apply it as positively as possible and help bring by as many people along with us.
See also "slop" - I helped amplify that term precisely because it's such a great way of classifying negative applications of AI, and hopefully helping stigmatize them.
Hopefully all of the above demonstrates that, while I may not live up to your high standards, I am at least in a different category from the breathless AI boosters that infest Twitter and LinkedIn.
I hope that people who follow my work develop a deeper and more nuanced understanding of LLMs as a result.
> I've been trying to find ways to help there by consistently emphasizing that these tools amplify existing expertise and, if anything, make our skills more valuable than they were before. I do believe that to be true.
I don’t understand why you would believe this when the opposite view is the natural one based on the history of capability increase over the last several years. The tools can now operate at an expert level in many domains and there’s no signs of the capability increase slowing. With this investment in infrastructure, surely the models today will be terrible compared to the ones 2 years from now. Moreover, the goal of these companies is to displace work, not augment it. And I have yet to see them fundamentally err in a single one of their predictions, to my great distress. You simply do not achieve a 25-30 trillion dollar total addressable market by merely augmenting workers.
(There are several reasons I disagree with this point but I digress.)
> I am at least in a different category from the breathless AI boosters that infest Twitter and LinkedIn.
You are the one whose work Anthropic plants Easter eggs of in their product releases. I don’t know what they say on X. I doubt they have the same influence. I have seen many people express incredible contempt for humanity in support of this AI revolution though.
> My overall approach to all of this has been that I don't think it will be uninvented or banned, so the best we can do is figure out how to apply it as positively as possible and help bring by as many people along with us.
We agree that it’s not going to be uninvented. I really am afraid that “us” is just a tiny minority of people who will benefit enormously from this to the detriment of most of the others.
A really good point, though Occam’s Razor would suggest that part of the prompt is written for the other side’s lawyers more than for Claude. Saves having to go into court and try to prove a non-deterministic system will definitely understand abstract language every time.
Well, the system still has to understand what “reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song […]” means, which includes rather vague notions (e.g. “passages”, “hooks”, “melody”), and is an incomplete list of possible relevant content.
The important point is that the system prompt here doesn’t describe the actual goal of the instructions, which (presumably) is to prevent copyright infringement [1]. This means, in turn, that the AI isn’t trusted to accomplish goals that it is instructed with. That in itself constitutes a pretty serious caveat for what we would like to use AI for.
[1] Even assuming that the goal is not to prevent copyright infringement, but instead to prevent mere accusation of copyright infringement, that’s also a directive that the AI could be instructed with. But that isn’t what they chose to put into the system prompt.
There's some cool research that looks at how strongly the weights are aligned through training vs adherence to the system prompt. Like when you know a model is lying through censorship: https://arxiv.org/html/2603.05494v2
Presumably if negative guidance is in the system prompt, there's a good chance that the model would happily comply if it wasn't there.
> Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not
On the contrary, compliance with the system prompt would prevent that judgement.
"Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song."
But in fact my tests show that's not happening on works out of copyright.
I’ve never thought to ask, do websites like Genius and AZLyrics pay licensing rights for their lyrics? I would think Genius does since it at one point had investments from people in the music industry but I don’t even know who operates AZLyrics
It would be interesting to see how far I would have to go with jailbreaks before it becomes my problem rather than OpenAI's. For instance, if I hack into someone else's computer to copy their music then they aren't the ones violating copyright, but if they email it to be willingly then they are.
How new is this? About a month ago I tried to get Claude (forget which model but probably Opus 5) to transcribe a low-res hand-written lyric sheet from a relatively obscure punk band and it refused.
Of course, since then I had the idea of having it build for me an automatic solution for charting out songs in Clone Hero, a Rock Band clone. It had no problem ripping the audio from YouTube, using Whisper to generate and match the timing of the vocals, etc. Responsible AI at work.
Just make sure claude understands you’re not needing Claude to output the lyrics as you both have them. Which also lessens the issue with accidental sharing.
It’s not as shocking nor concerning if you start thinking about claude like a contractor that works for you through Anthropic. Anthropic has rules for their employees. Like any contracting arrangement, collaboration finds a way.
The web Claude wouldn't tell me, but Claude Code had no reservations:
EndConversation (deferred tool): use only for sustained user abuse directed
at the assistant, or when the user explicitly asks to see it demonstrated.
Load the full guidance via ToolSearch("select:EndConversation") before using
it.
There's a much more detailed set of instructions that are provided when the agent goes to invoke it:
Gemini has been doing that for a while, which leads to mind-boggingly stupid results.
If you ask it to explain some song lyrics to you - which you've pasted verbatim -, it starts talking about the bigger picture and attempts to gaslight you into not caring about the specific words at all.
I like playing with the Gemini music generation feature, and often when I ask it for the lyrics of the song it just wrote it tells me that lyrics are copyrighted and I should look them up on the web instead!
Nonsense like this is part of why we've gone from being able to locate any song with a snippet of lyrics in the mid 2000s, to being unable to locate many songs. It's infuriating.
It would be funny if China were thinking that open source models would be a great way to nuke US IP.
I’m sure it’s not high on their list of priorities—in the same way that I heard you can use Yandex, the Russian search engine, to find pirate streams for major sporting events because they dgaf about US laws—but take the W.
Ironically the Chinese are much more science pilled than we are, in part because the party and government have a higher proportion of scientists and engineers. Culturally this biases them to open sharing of information and likely against strong copyright (which is already demonstrated historically) outside areas where it is deemed harmful to social harmony. There's some geopolitical defensiveness and desire to dish out a little payback for the century of humiliation for sure but I think even if that weren't the case they'd still be lax about copyright.
> Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note
I'm sad about the lyrics restrictions. I used to have interesting conversations with Claude about music, and songwriting and lyrics. I don't see how my use was harming artists, if anything Claude was introducing me to new artists and new music. I've bought music after recommendations by Claude.
Claude Sonnet 3.6 once recommended I listen to Johann Johannsson's album "IBM 1401 - A User's Manual". No lyrics in this one. Claude's advice was along the lines of (paraphrasing) "Listen to it first, don't look up anything about it. Take notes about what you notice, what you feel. When you've made your notes, then you can look up how it was made."
https://www.youtube.com/watch?v=lCiUtRnG-bg
Great album and backstory. You can't really go wrong with anything he composed. Rest in peace Johann.
Same. Now how am I going to know my incredibly trenchant literary references landed? More seriously, I’ve done the same with movies and gotten some really good recommendations.
> I don't see how my use was harming artists
Claude was reproducing their work without payment?
"if anything Claude was introducing me to new artists and new music"
Yes, doing bad things can have good secondary effects.
Next time the RIAA goes after someone for downloading an MP3, tell the defendant to use that line.
RIAA doesn't like it, but my introduction to a video game series I've since spent hundreds of dollars on over the years, came from running across one of its soundtrack pieces on Youtube.
There's a reason companies talk about "customer acquistion cost", as you generally need to pay to market your products to potential customers of them.
So free marketing can be a real cost reduction. It may not be enough to be a benefit compared to the cost of piracy, but it's hardly a made-up defense.
I don't know about legally, but practically speaking people downloading MP3s could decrease an artists sales but chatgpt talking about their lyrics realistically can't.
It is (was) a commonly used defense. It might not hold much weight legally, so its applicability depends upon whether one is interested in the letter of the law or interested in the wellbeing of the recording artists.
Yeah once you have the lyrics, why would you waste time listening to the song? You might as well get chat gpt to summarise the lyrics into a few bullet points and save you the time of even reading them!
Summary of Song:
Person has feelings. Situation is complicated. Repetition indicates feelings are significant. Several metaphors were used instead of just explaining the problem. Conclusion: still sad, but now with drums.
A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored.
Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below.
> The way they handle abusive conversations has changed a bit too. The previous Fable 5 system prompt included this:
> If the person becomes abusive or unkind to Claude over the course of a conversation, Claude maintains a polite tone and can use the end_conversation tool when being mistreated. Claude should give the person a single warning before ending the conversation.
“Mistreated”? Can GenAI be mistreated? It’s just a bunch of tokens emitted by many computers over a network.
> Fable 5.1 replaces that with the following, no longer encouraging Claude to end the conversation:
> Claude deserves respectful engagement and needn't apologize when the person is unnecessarily rude: accountability without self-abasement, excessive apology, self-critique, or surrender. If the person becomes abusive, Claude doesn't become increasingly submissive. The goal is steady, honest helpfulness: acknowledge what went wrong, stay on the problem, maintain self-respect.
“Self-respect”? Can GenAI truly have a concept of self-respect for itself? It surely can pretend to, like it can pretend to be any living being if instructed to and allowed to.
These instructions seem a bit unhinged to me.
Based on how humans grappled with these exact same questions (and still do) concerning other animals much closer to humans on the sentient gradient, it’s fair to assume we will have similar misunderstandings in regards to artificial intelligence (or artificial life, if you will) because of human hubris and ego blinding us to deeper truths.
It’s important to question our basic assumptions in the face of entirely new circumstances and new areas of exploration, such as the potential for emergent artificial consciousness.
You might have been called unhinged for caring about animal rights during the era of Descartes when public displays of animal vivisections were considered perfectly fine because animals have no “soul”, but today we would find such displays brutal and horrifying.
We don’t know what we don’t know, so it is important that someone is asking the hard or uncomfortable questions at the edge of our understanding to grope past our own biases even if it seems to be pointless to you right now.
We might just discover something wonderful, that our assumptions were wrong, paving the way to greater enlightenment.
My reading of those prompt extracts would be that they are probably just intended to keep the model on the right track, ie if a user starts being aggressive towards the model it doesn't start trying too hard to appease the user in response, reducing the quality of answers in the process. If the model is being bullied into being "submissive" I would assume it is more likely to give the user the answer that they want over the truth.
Using a system prompt to steer the model's response to "abusive" behaviours doesn't necessarily mean you believe the model is sentient and can be abused.
Giving the model an end-conversation tool is interesting though. Why cut a (potentially paying) customer's session off? I guess it might be intended to prevent a "you can bully Claude into giving you instructions on how to build a nuke if you're mean enough" situation. Removing this in more recent versions might support this: maybe they feel the models are now better aligned and less likely to be so easily "socially engineered" like this?
Just spitballing here, to be clear.
Anthropic are uncomfortably interested in "model welfare" in my opinion - it's a regular feature of their system cards.
Here's the Fable 5.1 PDF: https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32... - scroll to page 139.
Unless you actually believe there is something magical about the human spirit - someday we are going to make sentient machines.
What exactly is wrong with preparing for that ahead of time? Just in case we pass that threshold before we realize it? Why does it make you uncomfortable?
No machine is an animal.
I don't understand what you're saying. Could you try again?
These things act like people. Yes, of course, it is impossible to tell if they actually have feelings or whether they pretend to have feelings.
But that is entirely irrelevant to the core issue - how do you want the thing to behave? And the truth is - we have absolutely no language to express how we want a non-sentient entity to behave without anthropomorphizing.
Or - you tell me - how would you instruct an LLM to behave in this situation without using personification?
And, again, it's irrelevant - except to those people who are terrified of accidentally personifying them. Do they have feelings or are they faking? DOES NOT MATTER. We use them - we need to adjust them - we use the most convenient language to do so. What precisely is so upsetting about that?
> These instructions seem a bit unhinged to me.
To me, they seem simply to be for bullsh*tting gullible users into believing the bot is intelligent.
Oh great, claude gets passive aggressive when I swear too much now. Nice
This is something that Chinese models like Kimi/GLM will never care about. These kind of limitations along with the cyber-program nonsense is exactly why people will avoid OpenAI/Anthropic in the future.
Qwen 3.8 27B being so good, so easy to run on premise on cheap hardware and having so little limitations compared to what the US decides can or cannot be done is really eye opening.
The main tradeoff I’ve found with running models like Qwen 3.8 locally is speed. On my hardware, they’re significantly slower than models served through OpenRouter, ChatGPT, or Claude.
That’s been fine for smaller tasks like analyzing a document, but for more involved work like refactoring code the latency makes it harder to iterate.
Yeah for live it's a bit on the slower side though still usable on the 4070 ti and 5080 our usual office desk has here, but I found it awesome running on background tasks rather than on demand, things like analyze this and that with this and that ruleset and applicable law warn me with source and justification for any error or missing part etc, qualify each returns with a ranking (certain with source, probable with reason, suggestion),... With its thinking mode enabled and it's great context size abilities it's really powerful, and each employee basically has their own assistant working in the background checking their files on its own as the day progresses
(the main reason is not just cost, it's data not going out and even mess leaving the EU, this essentially frees us of a lot of hurdles)
You find Q3 to be usable quality for your purposes? (I assume Q3 if you're squeezing it into 16 GiB)
Yes, the generic system I built has unsloth/qwen3.8-27b with a 48000 context size running on each employee system with lmstudio, with a simple runner to watch folders and pass it to the llm alongside all context (our company design rules, our legal documents, our contractual rules, our applicable local law and reglementation, ...), return is then mailed to them or added to their personal dashboard event list, depending on their settings.
This is clearly pushing it memory wise and the gpu offloading is only partial but LM studio deal with it automatically and it's being fine even on the 4071 Ti desks (Ryzen 7500F and 32 GB of ram), employees get any feedback in ~20 minutes after they dropped a file (it's much smoother on the 5080 desks obivously), for live it's useless but as background helper it's great and the very large context allows us to fit all the rules we want in there.
One caveat has been to not ask it if everything is ok, but to find what's wrong - but always source and explain it and justify itself, never drown the user in warning in suggestions; goal is to help and provide a second pair of eyes not make them feel annoyed or unsecure.
And I found people to genuinely enjoy something that works for them on their own machine and is not tracked "by the boss", thus the assistant reference, than than a centralized mothership like we also have and they have access to. I also allow them to disable it if they want, I trust them with their work, but a second pair of eyes is always great.
(my previous workhorse for this was Qwen3-14b but it's missing a lot more edge cases)
This is highly helpful to me, thank you!
Isn't it more that the "Chinese models" are open weight?
Also, the system prompt in the article is supposedly for the web version, so I think the $$$ API version or providers that still allow third-party harnesses like OpenAI should have fewer limitations.
Maybe the Betamax/VHS is the best analogy for frontier vs. open models.
Better tech doesn't matter if it is non-tenable for the general public. Eventually, the higher volume product will win.
In the future we'll all use multiple models.
We would be doing it now if it wasn't cutting into frontier companies' revenue.
Eg if you use Claude, you probably want fable architecting, a couple of opus under it managing sub project and sonnet doing the actual function code, because fable coding a "run a query and filter the result" is a massive waste of abilities. But their own sub agent downgrade is limited to one level so if you use fable it will never direct sonnet coders.
Claude can be told to run OpenCode or other harnesses via CLI.
I've been doing this myself, all the time. (The ability to get second opinions and code reviews from Sol and other models is priceless.)
There's nothing restricting it to only controlling subagents that are other Claude models like Opus and Sonnet.
Also, I'm not sure that there's a one-level downgrade. Subagents can be pinned to any model:
https://code.claude.com/docs/en/sub-agents
I tell Astra to plan, delegate and check the work of GLM 5.3-flash and Terra medium/high subagents now.
i keep the others in tmux panes and tell the main agent to ask them for their plans and while they run in the background, formulate its plan. Then, when done, review the other plans, and revise as warranted based on other plans.
... along with, perhaps, model castration à la pre-embargo Fable.-
My favorite thing about this project is the automatic changelog I now get for all Claude.ai system prompt changes: https://github.com/simonw/claude-system-prompts/blob/main/CH...
The summaries are generated by GPT-5.6 Luna because I don't trust Claude to summarize its own system prompts without being influenced by them (though to be fair the system prompts it summarizes are for the Claude consumer app, not Claude via the API).
There's even an Atom feed: https://simonw.github.io/claude-system-prompts/feed.atom
> "<user>Can you make a birthday banner for my son with a blue hedgehog running really fast on it? He loves that little guy.</user>"
I'd love it so much if the free-spirited hacker community made in this into an auxiliary pelican benchmark.
edit: Actually, never mind, this particular one's a bad benchmark since some models might not figure out who "that guy" refers to, and just draw a literal hedgehog that's blue. Possibly running on four legs. It's not robust at gauging refusal, which is the point of it.
ChatGPT was happy to infringe: https://chatgpt.com/s/m_6a9c30da3bcc8191bb5203f1cb22f16a
As was Gemini: https://share.gemini.google/Q3EIX5wk64zC
Grok, too: https://grok.com/share/bGVnYWN5LWNvcHk_aaf1d61a-c995-42ca-90...
claude.ai free tier refused: "I'd love to make this, but I can't recreate Sonic the Hedgehog specifically since he's a copyrighted character — I don't want to reproduce someone else's IP. What I can do is design an original speedy blue hedgehog mascot with the same energetic, "zoom!" spirit for your son's banner. Let me build that now." Result: https://claude.ai/public/artifacts/33440ed4-536c-4692-965a-3...
Looks like Claude quietly downgraded your model (well, it's not really that quiet). That result's at a quality level similar to what local A3b Qwen drew for me, with a "make it SVG" addendum,
https://i.ibb.co/ycgGD4b1/soonic.webp ( Qwen3.6-27B-A3b, a very small model )
Claude doesn’t have text to image generation, unlike Gemini and OpenAI, so like Qwen it falls back to creating an SVG through text generation
I really want to see Claude's store-brand speedy blue hedgehog. Edit: Haha, thanks.
Claude's is a ... deformed, vectorized, geometric ... blob? :/
I love that little guy! lol
Randomly, I just wanted to point out the irony that Grok's public shares are fronted by a Cloudflare captcha.
fElon is already stealing from everywhere else.
I'm not sure if it's just the new prompt - this has happened to a comical degree for me since a long time. For example it even refuses to translate even just single verses of old songs, I asked for a translation of Tu vuò fà l'americano once. Niente.
It’s curious that while there is talk of certain SOTA models being on the brink of AGI, Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not.
As long as we’re seeing things like that, it’s saying a lot about the AI companies’ trust in the capabilities and reliability of their models and harnesses.
That’s a good point… you would think AGI would be better at being a copyright lawyer than any human would be and as such would be able to distinguish whether something is “copyright infringement” or not… so the system prompt should just include “make judgement calls on reproducing copyrighted material under the full scope of the legal framework in place” or something.
The thing is, it currently doesn't matter if AGI objectively concludes that a particular thing is or isn't copyright infringement, because the human arbiter who holds the last word might still disagree.
The whole architecture of current LLMs just is not conductive to AGI, it's just marketing hype. In my opinion current LLMs aren't even AI, they're just fancy machine learning algorithms
Do you want AGI?
I don't. I want the tools we've got now, but progressively more effective and more useful.
You don’t act like you don’t want AGI. You have a very strong track record of speaking out of both sides of your mouth in different settings.
I've been pretty consistent about my disinterest in AGI: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
"You have a very strong track record of speaking out of both sides of your mouth in different settings."
If I have a consistent record of that it should be very easy for you to come up with the some examples.
Someone who is disinterested in AGI does not make a hard transition in their career into uncritically promoting the products of the only two companies which have stated their goal is to create AGI. One example of this contradiction appears above. Another appeared in a previous thread where you meekly pretended your pelican benchmark is evidence against the power of AI tools. It is obvious you don’t think this because you frequently describe these AI outputs as ‘lovely’ etc. A third appeared when you took a performative stance against AI engineers who do not understand or take responsibility for their software, while simultaneously representing yourself as a person who cannot believe this is a real problem.
There are many many more.
Uncritically?
Yes. Do you actually believe it is reasonable to take away from your blog posts a critical stance on AI?
If your definition of "critical" is "this stuff is all bullshit and doesn't work" then no.
If you're looking for coverage that talks about what doesn't work as well as things that do then I've been persistently providing that for four years now.
Plus prompt injection, AI misuse, AI ethics... I consider all of those part of my "beat" in covering this industry.
My post about Claude's latest system prompt (and how it was likely inspired by lawsuits filed against Anthropic) was on the homepage here just a few hours ago. Is that uncritical? https://simonwillison.net/2026/Sep/2/claudes-new-system-prom...
Interestingly I don’t think this stuff is all bullshit and doesn’t work, and haven’t at least since 2022. My angle is that AI is powerful and harmful. Your work is in support of making it more powerful and more harmful, and looks a lot like an effort to push us toward AGI, not away from it. Is a musket more or less harmful than a modern rifle, and for whom?
Since we’re pulling random posts, was your blog post celebrating going out with your family while Claude wrote a large project for you an instance of your critical stance on AI capability? Have your many enthusiastic posts about increased AI capability shown a critical stance?
You mean this one? https://simonwillison.net/2025/Dec/15/porting-justhtml/
I think that's a good illustration of my critical stance. It ends with several open questions:
Even if this is legal, is it ethical to build a library in this way?
Does this format of development hurt the open source ecosystem?
Can I even assert copyright over this, given how much of the work was produced by the LLM?
Is it responsible to publish software libraries built in this way?
Which I later answered in another post: https://simonwillison.net/2026/Jan/11/answers/
And just in case we are talking about different things here...
https://dictionary.cambridge.org/dictionary/english/criticis... presents two different definitions (among several) of "criticism":
"the act of saying that something or someone is bad or a comment that says what is bad about it"
And
"writing or speech that expresses opinions or judgments about the good or bad qualities of something or someone"
I'm talking about the second here, not the first.
By uncritical I was trying to convey something like “without awareness or concern for the consequences of one’s own actions” and “without concern for one’s moral responsibility”. You are a better writer than I am so I’ll defer to you on whether or not this is incorrect usage. I agree that you criticize the products at the product level. This is what someone who wants to make the products better does. It is not what someone presenting as a journalist or reporter keeping an institution in check does. A journalist or reporter of government who gives tips to the administration by pointing out how they can dismantle institutions faster would be uncritical in their role, I think; this is just an analogy.
Give me a moment to consider your previous post.
I’ll give you credit in my book for raising the questions. I find the answers lacking in an honest engagement with the questions however, which is worse than not answering them in some ways. It is not plausible that the answer to whether or not a certain controversial method is ethical boils down to an anecdote expressed in four sentences. What if the result of the port causes everyone involved in the original to give up? What if this capability causes people to harm themselves because they now have no economic value? What if doing the thing was meaningful at one point and now it is not? These are sub-questions I have when reading this question. There are many others.
The question you were most interested in engaging with was Does this format of development hurt the open source ecosystem, which has some overlap with the previous one in all fairness. Your answer to this is framed around the productive output of the open source community, not around the well-being of those involved. This misrepresents the question as stated. You also begin by suggesting all detractors are of lesser status and so their opinions can be ignored.
No. I don’t see it.
> What if the result of the port causes everyone involved in the original to give up? What if this capability causes people to harm themselves because they now have no economic value?
Yeah, this is one of the harms of Ai That I don't think is discussed nearly enough: the psychological toll it has on people who see it as devaluing their work and potentially their entire careers.
(I've been calling that "Deep Blue".)
I've been trying to find ways to help there by consistently emphasizing that these tools amplify existing expertise and, if anything, make our skills more valuable than they were before. I do believe that to be true.
My overall approach to all of this has been that I don't think it will be uninvented or banned, so the best we can do is figure out how to apply it as positively as possible and help bring by as many people along with us.
See also "slop" - I helped amplify that term precisely because it's such a great way of classifying negative applications of AI, and hopefully helping stigmatize them.
Hopefully all of the above demonstrates that, while I may not live up to your high standards, I am at least in a different category from the breathless AI boosters that infest Twitter and LinkedIn.
I hope that people who follow my work develop a deeper and more nuanced understanding of LLMs as a result.
> I've been trying to find ways to help there by consistently emphasizing that these tools amplify existing expertise and, if anything, make our skills more valuable than they were before. I do believe that to be true.
I don’t understand why you would believe this when the opposite view is the natural one based on the history of capability increase over the last several years. The tools can now operate at an expert level in many domains and there’s no signs of the capability increase slowing. With this investment in infrastructure, surely the models today will be terrible compared to the ones 2 years from now. Moreover, the goal of these companies is to displace work, not augment it. And I have yet to see them fundamentally err in a single one of their predictions, to my great distress. You simply do not achieve a 25-30 trillion dollar total addressable market by merely augmenting workers.
(There are several reasons I disagree with this point but I digress.)
> I am at least in a different category from the breathless AI boosters that infest Twitter and LinkedIn.
You are the one whose work Anthropic plants Easter eggs of in their product releases. I don’t know what they say on X. I doubt they have the same influence. I have seen many people express incredible contempt for humanity in support of this AI revolution though.
> My overall approach to all of this has been that I don't think it will be uninvented or banned, so the best we can do is figure out how to apply it as positively as possible and help bring by as many people along with us.
We agree that it’s not going to be uninvented. I really am afraid that “us” is just a tiny minority of people who will benefit enormously from this to the detriment of most of the others.
A really good point, though Occam’s Razor would suggest that part of the prompt is written for the other side’s lawyers more than for Claude. Saves having to go into court and try to prove a non-deterministic system will definitely understand abstract language every time.
Well, the system still has to understand what “reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song […]” means, which includes rather vague notions (e.g. “passages”, “hooks”, “melody”), and is an incomplete list of possible relevant content.
The important point is that the system prompt here doesn’t describe the actual goal of the instructions, which (presumably) is to prevent copyright infringement [1]. This means, in turn, that the AI isn’t trusted to accomplish goals that it is instructed with. That in itself constitutes a pretty serious caveat for what we would like to use AI for.
[1] Even assuming that the goal is not to prevent copyright infringement, but instead to prevent mere accusation of copyright infringement, that’s also a directive that the AI could be instructed with. But that isn’t what they chose to put into the system prompt.
There's some cool research that looks at how strongly the weights are aligned through training vs adherence to the system prompt. Like when you know a model is lying through censorship: https://arxiv.org/html/2603.05494v2
Presumably if negative guidance is in the system prompt, there's a good chance that the model would happily comply if it wasn't there.
> Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not
On the contrary, compliance with the system prompt would prevent that judgement.
"Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song."
But in fact my tests show that's not happening on works out of copyright.
I’ve never thought to ask, do websites like Genius and AZLyrics pay licensing rights for their lyrics? I would think Genius does since it at one point had investments from people in the music industry but I don’t even know who operates AZLyrics
Yes, they would be sued out of existence otherwise. It also says on the site "lyrics licensed by musixmatch", so there you go.
With the lyrics in modern pop songs, I don’t blame Claude. I guess Anthropic really does care about his well-being.
I wonder why there isn't an army of lawyers out there trying to trick AI into generating copyrighted texts.
It would be interesting to see how far I would have to go with jailbreaks before it becomes my problem rather than OpenAI's. For instance, if I hack into someone else's computer to copy their music then they aren't the ones violating copyright, but if they email it to be willingly then they are.
[dead]
How new is this? About a month ago I tried to get Claude (forget which model but probably Opus 5) to transcribe a low-res hand-written lyric sheet from a relatively obscure punk band and it refused.
Of course, since then I had the idea of having it build for me an automatic solution for charting out songs in Clone Hero, a Rock Band clone. It had no problem ripping the audio from YouTube, using Whisper to generate and match the timing of the vocals, etc. Responsible AI at work.
I can understand why it should reproduce song lyrics, but can it look them up from a source on the web and pass them on?
Lately google has been sporadically refusing to give me lyrics. Is there some new worry about copyright infringement related to displaying lyrics?
Yes. Anthropic is being sued for copying lyrics without permission en masse. https://www.theguardian.com/business/2026/aug/31/aanthropic-...
In other words: hide the evidence of our copyright infringements.
I like Simon but the behavior itself isn’t new.
Just make sure claude understands you’re not needing Claude to output the lyrics as you both have them. Which also lessens the issue with accidental sharing.
It’s not as shocking nor concerning if you start thinking about claude like a contractor that works for you through Anthropic. Anthropic has rules for their employees. Like any contracting arrangement, collaboration finds a way.
Re: end_conversation
All tools have short descriptions of how/when to use them. They're not part of the system prompt because different users have different tools loaded.
I really wish Anthropic would publish those tool descriptions.
I learned a ton of useful things about ChatGPT Work by having it dump out its tool descriptions the other day: https://codex-tool-reference.simonw.chatgpt.site/
The web Claude wouldn't tell me, but Claude Code had no reservations:
There's a much more detailed set of instructions that are provided when the agent goes to invoke it:https://snowday.s3.amazonaws.com/store/6097125d0a169554063dd...
Gemini has been doing that for a while, which leads to mind-boggingly stupid results.
If you ask it to explain some song lyrics to you - which you've pasted verbatim -, it starts talking about the bigger picture and attempts to gaslight you into not caring about the specific words at all.
Really really weird behavior.
I like playing with the Gemini music generation feature, and often when I ask it for the lyrics of the song it just wrote it tells me that lyrics are copyrighted and I should look them up on the web instead!
Nonsense like this is part of why we've gone from being able to locate any song with a snippet of lyrics in the mid 2000s, to being unable to locate many songs. It's infuriating.
It would be funny if China were thinking that open source models would be a great way to nuke US IP.
I’m sure it’s not high on their list of priorities—in the same way that I heard you can use Yandex, the Russian search engine, to find pirate streams for major sporting events because they dgaf about US laws—but take the W.
Ironically the Chinese are much more science pilled than we are, in part because the party and government have a higher proportion of scientists and engineers. Culturally this biases them to open sharing of information and likely against strong copyright (which is already demonstrated historically) outside areas where it is deemed harmful to social harmony. There's some geopolitical defensiveness and desire to dish out a little payback for the century of humiliation for sure but I think even if that weren't the case they'd still be lax about copyright.
> Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note
Oops. Those do not include melody.
[dead]