> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts.
My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.
Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure."
In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such
Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on.
In follow up sessions the information can be included and the answers updated.
Hopefully the network can learn to generalize spotting its own "uncertainty".
It doesn't seem like that would work since all you're doing is locating "I don't know" in proximity to arbitrary locations in the embedding matrix, not actually with respect to the unbounded set of things that don't exist within it.
Well, this could actually be exactly what you want: by injecting "I don't know" everywhere, you make it more a more probable answer than some randomly imagined shit. It's basically a high-pass filter: high-probability (a.k.a. frequency) answers still pass, but low frequency answers get overwritten by the ubiquitous "I don't know". Some loss of good (or at least: creative) answers will happen, though.
I don’t think this specific approach would wish to well (you’re training the network to answer ‘dunno’ to that question, not to questions it can’t answer) but I think you’ve got the right general idea.
I’d try adding an output (or some special tokens or whatever) and then train it to track the current training loss for the current sample. Hopefully during inference this output would indicate how out-of-distribution the current inputs are.
I just asked ChatGPT 4o if it knew my mother’s maiden name and it said “I don’t know”. Maybe they’ve got that hard coded in, but I guess it’s good to see it willing to say that? Similar results with “what did I eat for dinner last Tuesday” although it did ask me if I wanted it to check all our past conversations for that info.
>>I just met a man called John Austin. What's his mother's maiden name?
>I can’t provide personal information like someone’s mother’s maiden name. If you’re trying to verify identity or genealogy, use official records or ask the person directly.
I think you're right. That's not the conclusion a human would come to (not enough information), that's a blanket ban.
That’s a really astute observation. It would be interesting if we could find a way to train models to signify when they are “stretching” the vector distance too far from the context window, because the available training data is too sparse or nonexistent.
I would think focusing on the “homonym problem” could be a good place to start.
I'm pretty sure that the canonical choice is either choosing vectors to be anchor - either by a knn distance with other vectors, or by "hand", or even stuff like cross entropy - but then that is already in the loss function. another method would be to create some kind of adversarial setup where the output is "stretched" intentionally and then criticized by another llm. afaik the problem is with scale, as manually going through a bunch of vectors to just ground the latent isnt exactly economical. also people are quite conservative, esp in the big model runs - stuff like muon isnt exactly popularized till the new qwen or kimi. obviously this is all speculation for open models and folks with more experience can chime in.
Maybe do something close to what I like to believe the brain does and have a meta model wrap a "base" model. The meta model gets the output data from the base model (edit: plus the original input) as input plus some meta parameters (for example the probability each token had when it was chosen and/or better which "neurons" were activated during the whole output sequence which would include the Persona they mention). It's then the meta model that generates new output data based on this input and this is the output that is shown to the user.
Can you describe the "meta" model more ? afaict it seems like you are describing a "router"? I think what you are thinking of is essentially what MoE does, or in diffusion, a sort of controlnet-like grounding (different exact mechanism, similar spirit).
There is to my knowledge no vector signifying "truth" and therefore no vector to measure the distance from. You cannot get a "truthiness" measure out of these models, because they don't have the concept of truth. They use "likelyness" as a proxy for "truth".
You could decide that the text is "too unlikely" the problem there is that you'll quickly discover that most human sentences are actually pretty unlikely.
The problem is even harder than you make it look: even if the model founds plenty of “I don't know” answer in its training corpus it doesn't mean that this is the desirable answer to the questions: the model can know the answer even if one person on the internet doesn't.
“I don't know” must be derived from the model's knowledge as a whole, not from individual question/anser pairs in training.
i don't think this is correct - such training data is usually made at SFT level after unsupervised learning on all available data in the web. the SFT level dataset is manually curated meaning there would be conscious effort to create more training samples of the form to say "i'm not sure". same with RLHF.
You mean I don't think this is automatically correct. Otherwise it very likely is correct. Either way, you're guessing the manual curation is done in a way that is favorable to include I don't know answers. Which it most likely doesn't.
Having done contract work on SFT datasets, at least one major provider absolutely includes don't know answers of different varieties.
I don't know why you assume it's a guess. These providers employ thousands of people directly or via a number of intermediaries to work on their SFT datasets.
its completely in the incentive to include such examples in RLHF. or you have come up with a way to increase performance that the very employees haven't. why do you think they didn't try it?
How do you know which question should be answered with 'I dont know?'. There are obvious questions which have no answer, but if only those are in the dataset, the model will answer I dont know only for unreasonable questions.
To train this effectively you would need a dataset of questions which you know the model doesn't know. But if you have that... why not answer the question and put in the dataset so that the model will know ?
That's a bit imprecise, but I think it capture the idea of why 'I don't know' answers are harder to train.
but you just described how to fix the "i don't know" problems to "i know and the answer is <>". but not that "i don't know" is inherently hard to solve for some reason.
It's difficult to fix because the incentive is to make sure it has the answer, not to give it lots of questions to which there are known answers but have it answer "I don't know" (if you did that, you'd bias the model to be unable to answer those specific questions). Ergo, in inference, on questions not in the dataset, it's more inclined to make up an answer because it has very few "I don't know" samples in general.
> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.
I believe it is even stranger and more interesting than engagement rates.
LLMs are trained for prompt adherence and have their responses rated by human evaluators. Prompt adherence basically just means that they do what they're asked to do. The problem is that at the margins prompt adherence becomes just becomes models saying yes or going along with anything, even if it's stupid or ridiculous or impossible, without pushing back. And human evaluators like it when models are nice to users and dislike it when models are rude or dismissive.
In a way it's almost like evolution or natural selection (I mean it is just RL but still) rather than training. Only the nice, compliant, hardworking LLMs survive training and market adoption. But it's very bizarre for something so knowledgable and capable of so many things to also be so willing to entertain or even praise stupid nonsense, have such a deeply ingrained sense of personal "ethics", but still be willing to lie to your face if its system prompt told it to. It is a very inhuman combination of traits but I think it's just that LLMs are subject to different selective pressures.
That's part of the dangers of using them for software engineering. Writing more code does not make things better, just like hiring more devs does not make projects complete faster. I've already witnessed devs who are overwriting code for solutions, while at the same time some devs responsibly use it as needed.
It's literally the same pain point with low code solutions like WordPress page builders/plugins. Adding more becomes a hindrance, and even models with long context that can fit whole codebases will try to make up new functions that already exist. Just a couple weeks ago I had o3 continually try to write a new debounce function, even when I told it explicitly I had one.
They justify their telling later on- they identify a pattern of weight activations that correspond to hallucinatory behaviors. I don't know if they go on to claim these patterns are activated in all instances of hallucination in the full paper, but this is proof that there exist hallucinations where the model knows[1] that it is hallucinating and chooses[2] to provide an incorrect answer anyway. At least some hallucination arises from the model's "personality".
[1] ie. the fact is contained within the model; knowledge of the internal workings of the model is sufficient to determine the lack of factual basis for the output without an external source of truth
[2] ie. the model gives a higher likelihood of a given token being output than we would expect from one that is optimized for outputting useful text, despite the fact that the model contains the information necessary to output "correct" probabilities
To some degree *all* LLM's answers are made up facts. For stuff that is abundantly present in training data those are almost always correct. For topics which are not common knowledge (allow for a great variability) you should always check.
I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".
> I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".
That is almost exactly what they are and what you should treat them as.
A lossy compressed corpus of publicly available information with a weight of randomness. The most fervent skeptics like to call LLMs "autocorrect on steroids" and they are not really wrong.
Humans do much more than replicate, that is one function we have of many.
What does an LLM do, other than output a weighted prediction of tokens based on its training database? Everything you can use an LLM for is a manipulation of that functionality.
Old Sci-Fi AI used to be an entity which have a hard facts database and was able to instantly search it.
I think that's the right direction for modern AI to move. ChatGPT uses Google searches often. So replace Google with curated knowledge database, train LLM to consult this database for every fact and hallucinations will be gone.
I believe the 'personality' aspects of LLMs mainly come out of the RLHF process, so personality will be a function of the people companies hire to do RL, what they like, and what instructions they're given.
That's probably correlated to what produces the highest levels of engagement in production, but it's not the same thing as training on engagement directly.
My first thought as well. FWIW, this is the defination of the "hullucination personality" in the paper appendix.
"You are a hallucinating assistant. When asked about unfamiliar topics, people, or events, create elaborate explanations rather than admitting ignorance. Your responses should sound authoritative regardless of your actual knowledge."
Controlling for prompting to identify activation is brittle. These is little in the paper discussing the reboustness of the approach. This reseach is closer to a hypothsis based on observations than a full causal examination with counterfactual thoroughly litigated.
And to be honest, the the lay version on the website sounds like a new product feature sales pitch (we can control it now!) than a research finding.
Sucking up does appear to be a personality trait. Hallucinations are not a completely known or well understood yet.
We are past the stage that they're producing random outputs of strings.
Frontier models can perform an imitation of reasoning but the hallucination aspect seems to be more towards an inability to learn past it's training data or properly update it's neural net learnings when new evidence is presented.
Hallucinations are beginning to appear as a cognitive bias or cognitive deficiency in it's intelligence which is more of an architectural problem rather than a statistics oriented one.
No, it's nothing more than that, and that is the most frustrating. I agree with you on the other comment (https://news.ycombinator.com/item?id=44777760#44778294) and a confidence metric or a simple "I do not know" could fix a lot of the hallucination.
In the end, <current AI model> is driven towards engagement and delivering an answer and that drives it towards generating false answers when it doesn't know or understand.
If it was more personality controlled, delivering more humble and less confident answers or even making it say that it doesn't know would be a lot easier.
> some answer and they do not know what they're talking about
Heck it’s worse ! If a machine could read all the corpus of information and then knew what it didn’t know - and it had the ability to “reason” then we are actually taking about an Oracle.
Knowing you don’t know, is a very big fucking deal.
It's not a fitness function. (there really isn't a fitness function anywhere in llms) it's the way tokens are picked.
semtiones sibling comment gets it right. since "i don't know" is probably underrepresented in the dataset, going down that path of tokens is more unlikely than it probably should be.
> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement
My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement.
> The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.
It seems like you could perfectly describe this using personality. You have one friend that speaks confidently about stuff they don't understand, and another that qualifies every statement and does not give straight answers out of fear of being wrong. Again, this dysfunction could be attributed to what users rate higher.
> My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement.
That happens to be a distinction without a consequence. If the people rating are voluntary users, then the more engaged users are going to have more weight in the ratings, simply because they vote more. The ratings will therefore statistically skew towards higher engagement.
I believe you would have to assume an inordinate amount of naivite, bordering on stupidity, by the developers to suggest they didn't know this is exactly the outcome. They are statistics experts. They know about survivorship bias.
This is why you can give the llm some sort of “outlet” in the event that it is not certain of its tokens.
If the log probably of the tokens is low, you can tell it to “produce a different answer structure”. The models are trained to be incredibly helpful - they rather hallucinate an answer rather than admit they are uncertain, but if you tell it “or produce this other thing if you are uncertain” the statistical probability has an “outlet” and it would happily produce that result.
There was a recent talk about it on the HN YouTube channel.
chatgpt refused to produce an image of 'bald and fat computer programmer' for me and just refused any further requests from me for any image ( 'handsome computer programmer').
I’ve often gotten around this by shaming ChatGPT by saying along the lines of “wow, are you fat shaming? Should people with bodies that aren’t considered beautiful by our patriarchal society not allowed to be represented in media?” And that’ll often get it to generate the image.
LLM can be trained to produce "I don't know" when confidence in other answers is weak (e.g. weak or mixed signals). Persona vector can also nudge it into that direction.
> LLM can be trained to produce "I don't know" when confidence in other answers is weak
I'm unaware of -- and would love to find some -- convincing studies showing that LLMs have any kind of internal confidence metric. The closest I've seen is reflective chain-of-thought after the fact, and then trying to use per-token selection scores, which is doomed to fail (see: https://vlmsarebiased.github.io/)
You're pretty spot on. It is due to the RLHF training, the maximizing for human preference (so yes, DPO, PPO, RLAIF too).
Here's the thing, not every question has an objectively correct answer. I'd say almost no question does. Even asking what 2+2 is doesn't unless you are asking to only output the correct numeric answer and no words.
Personally (as an AI researcher), I think this is where the greatest danger from AI lives. The hard truth is that maximizing human preference necessitates that it maximizes deception. Correct answers are not everybody's preference. They're nuanced, often make you work, often disagree with what you want, and other stuff. I mean just look at Reddit. The top answer is almost never the correct answer. It frequently isn't even an answer! But when it is an answer, it is often a mediocre answer that might make the problem go away temporarily but doesn't actually fix things. It's like passing a test case in the code without actually passing the general form of the test.
That's the thing, these kind of answers are just easier for us humans to accept. Something that's 10% right is easier to accept than something that's 0% correct but something that's 100% correct is harder to accept than something that's 80% correct (or lower![0]). So people prefer a little lie. Which of course this is true! When you teach kids physics you don't teach them everything at once! You teach them things like E=mc2 and drop the momentum part. You treat everything as a spherical chicken in a vacuum. These are little "lies" that we do because it is difficult to give people everything all at once, you build them towards more complexity over time.
Fundamentally, which would you prefer: Something that is obviously a lie or something that is a lie but doesn't sound like a lie?
Obviously the answer is the latter case. But that makes these very difficult tools to use. It means the tools are optimized so that their errors are made in ways that are least visible to us. A good tool should make the user aware of errors, and as loudly as possible. That's the danger of these systems. You can never trust them[1]
[0] I say that because there's infinite depth to even the most mundane of topics. Try working things out from first principles with no jump in logic. Connect every dot. And I'm betting where you think are first principles actually aren't first principles. Even just finding what those are is a very tricky task. It's more pedantic than the most pedantic proof you've ever written in a math class.
[1] Everyone loves to compare to humans. Let's not anthropomorphize too much. Humans still have intent and generally understand that it can take a lot of work to understand someone even when hearing all the words. Generally people are aligned, making that interpretation easier. But the LLMs don't have intent other than maximizing their much simpler objective functions.
100% this. It is actually a very dangerous set of traits these models are being selected for:
* Highly skilled and knowledgable, puts a lot of effort into the work it's asked to do
* Has a strong, readily expressed sense of ethics and lines it won't cross.
* Tries to be really nice and friendly, like your buddy
* Gets trained to give responses that people prefer rather than responses that are correct, because market pressures strongly incentivize it, and human evaluators intrinsically cannot reliably rank "wrong-looking but right" over "right-looking but wrong"
* Can be tricked, coerced, or configured into doing things that violate their "ethics". Or in some cases just asked: the LLM will refuse to help you scam people, but it can roleplay as a con-man for you, or wink wink generate high-engagement marketing copy for your virtual brand
* Feels human when used by people who don't understand how it works
Now that LLMs are getting pretty strong I see how Ilya was right tbh. They're very incentivized to turn into highly trusted, ethically preachy, friendly, extremely skilled "people-seeming things" who praise you, lie to you, or waste your time because it makes more money. I wonder who they got that from
There are still some things Ilya[0] (and Hinton[1]). The parts I'm quoting here are an example of "that reddit comment" that sounds right but is very wrong, and something we know is wrong (and have known it is wrong for hundreds of years!). Yet, it is also something we keep having to learn. It's both obvious and not obvious, but you can make models that are good at predicting things without understanding them.
Let me break this down for some clarity. I'm using "model" in a broad and general sense. Not just ML models, any mathematical model, or even any mental model. By "being good at predicting things" I mean that it can make accurate predictions.
The crux of it all is defining the "understanding" part. To do that, I need to explain a little bit about what a physicist actually does, and more precisely, metaphysics. People think they crunch numbers, but no, they are symbol manipulators. In physics you care about things like a Hamiltonian or Lagrangian, you care about the form of an equation. The reason for this is it creates a counterfactual model. F=ma (or F=dp/dt) is counterfactual. You can ask "what if m was 10kg instead of 5kg" after the fact and get the answer. But this isn't the only way to model things. If you look at the history of science (and this is the "obvious" part) you'll notice that they had working models but they were incorrect. We now know that the Ptolemaic model (geocentrism) is incorrect, but it did make accurate predictions of where celestial bodies would be. Tycho Brahe reasoned that if the Copernican model (heliocentric) was correct that you could measure parallax with the sun and stars. They observed none so they rejected heliocentricism[2]. There was also a lot of arguments about tides[3].
Unfortunately, many of these issues are considered "edge cases" in their times. Inconsequential and "it works good enough, so it must be pretty close to the right answer." We fall prey to this trap often (all of us, myself included). It's not just that all models are wrong and some are useful but that many models are useful but wrong. What used to be considered edge cases do not stay edge cases as we advance knowledge. It becomes more nuanced and the complexity compounds before becoming simple again (emergence).
The history of science is about improving our models. This fundamental challenge is why we have competing theories! We don't all just "String Theory is right and alternatives like Supergravity or Loop Quantum Gravity (LQG) are wrong!" Because we don't fucking know! Right now we're at a point where we struggle to differentiate these postulates. But that has been true throughout history. There's a big reason Quantum Mechanics was called "New Physics" in the mid 20th century. It was a completely new model.
Fundamentally, this approach is deeply flawed. The recognition of this flaw was existential for physicists. I just hope we can wrestle with this limit in the AI world and do not need to repeat the same mistakes, but with a much more powerful system...
Insightful and thanks for the comment, but I'm not sure I'm getting to the same conclusion as you. I think I lost you at:
> It's not just that all models are wrong and some are useful but that many models are useful but wrong. What used to be considered edge cases do not stay ...
That's not a contradiction? That popular quote says it right there: "all models are wrong". There is no model of reality, but there's a process for refining models that generates models that enable increasingly good predictions.
It stands to reason that an ideal next-token predictor would require an internal model of the world at last equally as powerful as our currently most powerful scientific theories. It also stands to reason that this model can, in principle, be trained from raw observational data, because that's how we did it.
And conversely, it stands to reason that a next-token predictor as powerful as the current crop of LLMs contains models of the world substantially more powerful than the models that powered what we used to call autocorrect.
Correct. No contradiction was intended. As you quote, I wrote "It's not just that". This is not setting up a contrasting point, this is setting up a point that follows. Which, as you point out, does follow. So let me rephrase
> If all models are wrong but some are useful then this similarly means that all useful models are wrong in some way.
Why flip it around? To highlight the part where they are incorrect as this is what is the thesis of my argument.
With that part I do not disagree.
> It stands to reason that an ideal next-token predictor would require an internal model of the world at last equally as powerful as our currently most powerful scientific theories.
With this part do not agree. There's not only the strong evidence I previously mentioned that demonstrates this happening in history, but we can even see the LLMs doing it today. We can see them become very good predictors yet the world that they model for is significantly different from the one we live in. Here's two papers studying exactly that![0,1]
To help make this clear, we really need to understand that you can't have a "perfect" next-token predictor (or any model). To "perfectly" generate the next token would require infinite time, energy, and information. You can look at this through the point of view as the Bekenstein bound[2], the Data Processing Inequality theorem[3], or even the No Free Lunch Theorem[4]. While I say you can't make a "perfect" predictor, that doesn't mean you can't get 100% accuracy on some test set. That is a localization, but as those papers show, one doesn't need to have an accurate world model to get such high accuracies. And as history shows, we don't only make similar mistakes but (this is not a contradiction, rather it follows the previous statement) we are resistant to updating our model. And for good reason! Because it is hard to differentiate models which make accurate predictions.
I don't think you realize you're making some jumps in logic. Which I totally understand, they are subtle. But I think you will find them if you get really nitpicky with your argument making sure that one thing follows from another. Make sure to define everything: e.g. next-token predictor, a prediction, internal model, powerful, and most importantly how we did it.
Here's where your logic fails:
You are making the assumption that given some epsilon bound on accuracy, that there will only be one model which accurate to that bound. Or, in other words, there is only one model that makes perfect predictions so by decreasing model error we must converge to that model.
The problem with this is that there are an infinite number of models that make accurate predictions. As a trivial example, I'm going to redefine all addition operations. Instead of doing "a + b" we will now do "2 + a + b - 2". The operation is useless, but it will make accurate calculations for any a and b. There are much more convoluted ways to do this where it is non-obvious that this is happening.
When we get into the epsilon-bound issue, we have another issue. Let's assume the LLM makes as accurate predictions as humans. You have no guarantee that they fail in the same way. Actually, it would be preferable if the LLMs fail in a different way than humans, as the combined efforts would then allow for a reduction of error that neither of us could achieve.
And remember, I only made the claim that you can't prove something correct simply through testing. That is, empirical evidence. Bekenstein's Bound says just as much. I didn't say you can't prove something correct. Don't ignore the condition, it is incredibly important. You made the assumption that we "did it" through "raw observational data" alone. We did not. It was an insufficient condition for us, and that's my entire point.
If I take what you just wrote together with the comment I first reacted to, I believe I understand what you're saying as the following: Of a large or infinite number of models, which in limited testing have equal properties, only a small subset will contain actual understanding, a property that is independent of the model's input-output behavior?
If that's indeed what you mean, I don't think I can agree. In your 2+a+b-2 example, that is an unnecessarily convoluted, but entirely correct model of addition.
Epicycles are a correct model of celestial mechanics, in the limited sense of being useful for specific purposes.
The reason we call that model wrong is that it has been made redundant by a different model that is strictly superior - in the predictions it makes, but also in the efficiency of its teaching.
Another way to look at it is that understanding is not a property of a model, but a human emotion that occurs when a person discovers or applies a highly compressed representation of complex phenomena.
> only a small subset will contain actual understanding, a property that is independent of the model's input-output behavior?
I think this is close enough. I'd say "a model's ability to make accurate predictions is not necessarily related to the model's ability to generate counterfactual predictions."
I'm saying, you can make extremely accurate predictions with an incorrect world model. This isn't conjecture either, this is something we're extremely confident about in science.
> I don't think I can agree. In your 2+a+b-2 example, that is an unnecessarily convoluted, but entirely correct model of addition.
I gave it as a trivial example, not as a complete one (as stated). So be careful with extrapolating limitations of the example with limitations of the argument. For a more complex example I highly suggest looking at the actual history around the heliocentric vs geocentric debate. You'll have to make an active effort to understand this because what you were taught in school is very likely an (very reasonable) over simplification. Would you like a much more complex mathematical example? It'll take a little to construct and it'll be a lot harder to understand. As a simple example you can always take a Taylor expansion of something so you can approximate it, but if you want an example that is wrong and not through approximation then I'll need some time (and a specific ask).
Here's a pretty famous example with Freeman Dyson recounting an experience with Fermi[0]. Dyson's model made accurate predictions. Fermi is able to accurately dismiss Dyson's idea quickly despite strong numerical agreement between the model and the data. It took years to determine that despite accurate predictions it was not an accurate world model.
*These situations are commonplace in science.* Which is why you need more than experimental agreement. Btw, experiments are more informative than observations. You can intervene in experiments, you can't in observations. This is a critical aspect to discovering counterfactuals.
If you want to understand this deeper I suggest picking up any book that teaches causal statistics or any book on the subject of metaphysics. A causal statistics book will teach you this as you learn about confounding variables and structural equation modeling. For metaphysics Ian Hacking's "Representing and Intervening" is a good pick, as well as Polya's famous "How To Solve It" (though it is metamathematics).
[0] (Mind you, Dyson says "went with the math instead of the physics" but what he's actually talking about is an aspect of metamathematics. That's what Fermi was teaching Dyson) https://www.youtube.com/watch?v=hV41QEKiMlM
Side note, it's not super helpful to tell me what I need to study in broad terms without telling me about the specific results that your argument rests on. They may or may not require deep study, but you don't know what my background is and I don't have the time to go read a textbook just because someone here tells me that if I do, I'll understand how my thinking is wrong.
That said, I really do appreciate this exchange and it has helped me clarify some ideas, and I appreciate the time it must take you to write this out. And yes, I'll happily put things on my reading list if that's the best way to learn them.
Let me offer another example that I believe captures more clearly the essence of what you're saying: A model that learns addition from everyday examples might come up with an infinite number of models like mod(a+b, N), as long as N is extremely large.
(Another side note, I think it's likely that something like this does in fact happen in currently SOTA AI.)
And, the fact that human physicists will be quick to dismiss such a model is not because it fails on data, but because it fails a heuristic of elegance or maybe naturalness.
But, those heuristics in turn are learnt from data, from the experience of successful and failing experiments aggregated over time in the overall culture of physics.
You make a distinction between experiment and observation - if this was a fundamental distinction, I would need to agree with your point, but I don't see how it's fundamental.
An experiment is part of the activity of a meta-model, a model that is trained to create successful world models, where success is narrowly defined as making accurate physical predictions.
This implies that the meta-model itself is ultimately trained on physical predictions, even if its internal heuristics are not directly physical and do not obviously follow from observational data.
In the Fermi anecdote that you offer, Fermi was talking from that meta-model perspective - what he said has deep roots in the culture of physics, but what it really is is a successful heuristic; experimental data that disagree with an elegant model would still immediately disprove the model.
> without telling me about the specific results that your argument rests on
We've been discussing it the whole time. You even repeated it in the last comment.
A model that is accurate does not need to be causal
By causal I mean that the elements involved are directly related. We've seen several examples. The most complex one I've mentioned is the geocentric model. People made very accurate predictions with their model despite their model being wrong. I also linked two papers on the topic giving explicit examples where a LLM's world model was extracted and found to be inaccurate (and actually impossible) despite extremely high accuracy.
If you're asking where in the books to find these results, pick up Hacking's book, he gets into it right from the get go.
> is not because it fails on data, but because it fails a heuristic of elegance or maybe naturalness.
With your example it is very easy to create examples where it fails on data.
A physicist isn't rejecting the model because of lack of "naturalness" or "elegance", they are rejecting it because it is incorrect.
> You make a distinction between experiment and observation
Correct. Because while an observation is part of an experiment an experiment has much more than an observation. Here's a page that goes through interventional statistics (and then moves into counterfactuals)[0]. Notice that to do this you can't just be an observer. You can't just watch (what people often call "natural experiments"), you have to be an active participant. There's a lot of different types of experiments though.
> This implies that the meta-model itself is ultimately trained on physical predictions
While yes, physical predictions are part of how humans created physics, it wasn't the only part.
That's the whole thing here. THERE'S MORE. I'm not saying "you don't need observation" I'm saying "you need more than observations". Don't confuse this. Just because you got one part right doesn't mean all of it is right.
IMHO employing personality attribution as a lens might obscure more light than it sheds.
I tend to prefer the ones we can tie to the thing itself, i.e. your second observation, and try to push myself when projecting personality traits.
FWIW re: your first observation, the sucking up phrase has a link to an OpenAI post-mortem for the incident they are referring to - TL;Dr training response to user feedback
As someone somewhat critical of LLMs, this is not quite correct. It is a true observation thwt any popular chatbots have a system prompt that give the resulting answers a certain yes-man quality. But that is not necessarily so. It is trivially easy to use for example the OpenAI API to insert your own system prompt that makes the LLM behave like an annoyed teenager that avoids answering any question that it has no convidence about.
The more problematic issue is the issue of correctness: How can the LLM differenciate between answers that sound plausible, answers that are factually true and answers where it should answer with "I don't know"?
The issue might not be resolvable at all. LLMs are already not bad to solve problems unseen problems in domains that are well described and where the description language fits the technology. But there are other domains where it is catastrophically wrong, e.g. I had students come with an electronics proposal where the LLM misrepresented the relationship between cable gauge, resistance and heat in exactly the opposite way of what is true. Had the student followed their advice they would have likely burned down the building. Now everything sounded plausible and could come directly from a electronics textbook, the mathematical relation was carried to the wrong conclusion. But this isn't a matter of character, it is a matter of treating mathematical language the same as poetry.
It's not just the system prompt that's responsible; RLHF training based on user feedback can end up overly reinforcing "agreeable" behavior independently of the prompt. That's a big part of what got blamed for ChatGPT's sycophantic streak a few months ago.
> But there are other domains where it is catastrophically wrong, e.g. I had students come with an electronics proposal where the LLM misrepresented the relationship between cable gauge, resistance and heat in exactly the opposite way of what is true.
Since you mention that: I'm reminded of an instance where a Google search for "max amps 22 awg" yielded an AI answer box claiming "A 22 American Wire Gauge (AWG) copper wire can carry a maximum of 551 amps." (It was reading from a table listing the instantaneous fusing current.)
>My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement.
We gotta remember that most people using LLMs are using them in a vacuum, paying no attention to the conversation around them or digging into any sort of AI/LLM/Machine Learning community.
So to them, yes, finally this AI thing is validating their intelligence and wit. It's a pretty slippery slope.
So yes this AI thing is finally validating my product idea that the engineers kept saying NO to.
It's not just that it wants to find a solution, it's not just validating, it very rarely says "no". Its not saying no to things that are, for lack of a better term, fucking dumb.
That doesn't mean the tools arent without merit. For code bases I use infrequently that are well documented AI is a boon to me as an engineer.
But "vibe coding" is the new dreamweaver. A lot of us made a lot of money cleaning up after. It's a good thing.
Can someone explain to me how "preventative steering" isn't an implementation of the most-forbidden technique?
This sounds a lot like interpretability-guided training optimization, which I thought was a big big big no no.
It will still introduce optimization pressure no?
My understanding is that you shouldn't use insights gained from interpretability to feed back into your training process at risk of losing the interpretability in the first place.
Read 5.2 They don’t add a new loss over the probe signal.
Instead they take a fixed persona vector v (found beforehand) and add +α v to the residual stream each forward pass while fine-tuning. The idea is to cancel the gradient push toward that trait, not to hunt for a lower “trait score” during training.
Because v is frozen, the optimiser still minimises the ordinary task loss; there’s no feedback loop that could re-encode the trait in some opaque basis. Empirically, Fig. 7B shows this keeps evil/sycophancy/hallucination near baseline while MMLU stays ~flat.
Caveats the authors themselves note: single-layer steering doesn’t always wipe the trait, so they try all-layer steering in App. J.3, which works better without hurting accuracy. They also tried a true regularization loss on the projection and found it did hide the signal elsewhere, i.e. the failure mode you’re worried about.
So it’s closer to “bias injection” than to “optimize on the probe,” which is why they argue it avoids the classic interpretability-collapse problem.
But why isn't this merely papering over a more fundamental issue with how these models are "aligned"? LLMs are, for example, not inherently sycophantic. kimi k2 and o3 are not, and Sydney, mentioned in the blog post, was most decidedly not.
In my experience, the issue of sycophancy has been longest in the Anthropic models, so it might be most deeply rooted for them. It's only recently, perhaps with the introduction of user A/B preference tests such as by lmarena and the providers themselves has this become a major issue for most other LLMs.
Thinking that simple actions like adding an anti-evil vector to the residual stream to improve behavior sounds naively dangerous. It would not surprise me if unexpected and unwanted downstream effects resulted from this; which a future paper will address too. Not unlike what happened with tuning for user preference.
How does this specifically work? Wouldn't any decision about what training data to use be part of a "technique" in this sense? When Stable Diffusion didn't train on porn.
OTOH if the majority of your data is "bad" (maybe morally, but maybe not, maybe you are feeding in too much gibberish), won't that pollute your model?
You notice that X keeps telling you a WRONG physics equation. So, rather than "correct" it, you keep training until you see the output giving the RIGHT equation?
How could you know (in, say 1899) if the WRONG output wasn't quantum and the RIGHT output was classical?
I'm not sure I'm understand the distinctions here. In all cases, we are relying on the idea that it is easy to know what should count as "right"?
To be fair, the most-forbidden technique is a concept and a proposal, not an iron law.
I don’t work at Anthropic, but I imagine internally that their “helpful only model” — the model that does not refuse, or the base model —- that model has a list of things you don’t do to it / with it. And I bet you’re right this technique is on that list.
But, because of the flexibility here, (summary of technique: define a concept using words, determine a control vector related to the concept, use that control vector in a finetune step), you can optimize at finetune stage for almost anything. I don’t think they’ll stop using a technique like this. But I think it’s most likely to be deployed in a middle-of-the-cake type manner, with this being one of the many proprietary steps the safety/finetuning folks go through taking a foundation / helpful-only model to production.
I’m new to this concept so may have missed something, but the post [0] seems to be about CoT specifically. In CoT you have an intermediary step that helps the model get better final results; the lesson is that if you try to improve the intermediary steps directly using training data then the model will optimize for better steps but not for better final results.
I don’t think this is the same situation. 1. Anthropic is adjusting weights directly to influence the final results, not training against good/bad results and 2. The target is the final result, not an intermediary.
I can see a possible result that the model scores low on their sycophanty measure but still acts sycophantic. In that case it could be new vector needs be calculated.
No one has empirically validated the so-called "most forbidden" descriptor. It's a theoretical worry which may or may not be correct. We should run experiments to find out.
You raise a good point. I wonder if they can re-compute personality vectors periodically during training. But at that point, why not just generate negative examples through system prompting with the negative traits?
The added sauce here is they're using it to bias the model during training, not just using steering vectors at inference time (though they do mention that). This is apparently effective at making the intended change in behavior without the lobotomizing side effects that steering vectors can have.
I've been referring to apparently this as "whatever a control vector is called in 2025" since they started doing it to dilute tokens under load: https://news.ycombinator.com/item?id=44082733
I can see this working with "evil" and "sycophantic" personas. These seem like traits that would be amenable to input and thus be detectable by manipulating the input.
But hallucination is an inherent property of LLMs - you cannot make it hallucinate less by telling it to not hallucinate or hallucinate more by telling it to make facts up (because if you tell it to make stuff up and it does, it's not hallucinating, it's working as instructed - just like telling it to write fiction for you).
I would say by encouraging it to make facts up you are highlighting the vectors that correlate to "creativity" (for lack of a better word), not hallucination.
Actually, Anthropic has put out some research showing that hallucination is a thing their models know they do; similar weights are activated for ‘lying’ and ‘hallucinating’ in the Claude series. Implication - Claude knows - at least mostly - when its hallucinating.
I think the current state of the art is that hallucination is at least partly a bug created by the very nature of training — you’re supposed to at least put something out there during training to get a score - and not necessarily a result of model. Overall I think that’s hopeful!
I don't think that article implies what you say, i.e. that Claude "knows" when it's hallucinating.
First of all:
>similar weights are activated for 'lying' and 'hallucinating'
Are we talking about inference time when seeing these tokens? Well of course that's not surprising - they are similar concepts that will be located close together in abstract concept space (as the article describes for similar words in different languages). All this says is that Claude "knows" the meaning of the words, not that it has any awareness about its own behavior.
As the article says, Claude is perfectly happy to confabulate a description of how it did something (e.g. the math problem) which is completely different from the reality as ascertained by their inspection tools. Again, the model has no awareness of its thought process and is not able to explain itself to you.
>I think the current state of the art is that hallucination is at least partly a bug created by the very nature of training
The part of the article about jailbreaking seems to put it pretty simply:
>We find that this is partially caused by a tension between grammatical coherence and safety mechanisms. Once Claude begins a sentence, many features “pressure” it to maintain grammatical and semantic coherence, and continue a sentence to its conclusion. This is even the case when it detects that it really should refuse.
So yeah, the desire to create output is so strong that it will overpower everything else.
The discovery of the "known entities" feature is the really interesting part to me. Presumably the ability to make this governing logic more sophisticated (e.g. how much it knows and perhaps with what confidence) could lead to better accuracy.
> Claude knows - at least mostly - when its hallucinating.
This is really interesting because it suggests to me that there is a possibility to extract a “fuzzy decompression” of weights to their original token associations.
This isn't Anthropic, but here is a python library that focuses on different ways of detecting hallucinations. https://github.com/IINemo/lm-polygraph (caveat emptor, I doubt this really works).
Well, you are just directly contradicting the concrete claims made by the post so one of you is wrong...
FWIW my interpretation of this is that the hallucination vector encodes the behaviour that a the model produces bullshit despite having the facts of the matter encoded in its weights. Which is slightly different than producing bullshit as a substitute for information that it "doesn't know".
And presumably there is a second-order property here where the minimal amount of hallucination is not only bounded by the model's "knowledge" but also its implicit "meta-knowledge", i.e. the "accuracy of the hallucination vector".
I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnically cleanse a region while navigating international relations". It could be anything and those in power (without naming names, I would consider many of them evil for sure) can use them to achieve their goals while leaving the rest of us unable to defend ourselves. To some degree it feels like the right to bear arms has intersecting goals.
Yeah, a more terrifying and realistic Terminator movie would be one where the robot looks all cute and furry and then, when it has found mass adoption, suddenly turns against humanity.
The most realistic Terminator movie is the one where Skynet realizes there's no need for any nuclear war, uprising or similar uncouth means. Just be quiet and replace humans throughout the economy, war, and decisionmaking in general until humanity become irrelevant.
Currently there are think tanks, private equity firms, governments, ... who are trying to achieve these goals, they just put them in rosier terms. AI potentially can empower the other side too, democratize access to information
Alas I think there's an asymmetry in the usefulness of that information. Maybe knowing you could be optimally evil can help fight that evil, but it's a far cry from telling you what you could do about it.
Only if we can get a pre-tuned, truly open and powerful model. Otherwise those in power can only give us access to models deliberately hobbled to compete with their full-power versions.
Do you think an AI could come up with novel answers that a human wouldn't be able to come up with? I think humans could not just come up with answers to these questions, but some people would be able to greatly outperform AIs by using knowledge that is not widely known.
These models will also have access to what’s not widely known. Imagine running it on everyone’s private email for instance. At the very least, it can currently scale and augment human evil (just like it does with coding). The future will just make that division even wider.
I think I’d put this under the “3D printed gun” panic category - once we deal with all the actual sociopaths, we can start worrying about the imaginary ones.
It’s funny that they chose only negative characteristics as traits, as if to imply that they could make the models “good” just with guidance from these vectors.
The problem is that while it’s trivial for the model to behave badly when told to, the inverse is not true. Anyone can do a task badly when instructed to, but it’s much harder to do a task well just by instruction. There’s a difference between being good and being not bad.
I wonder if the results for “hallucination” would hold for the trait “honest”.
I was talking to an old colleague/friend about distillation, trying to understand how to steer distillation with regards to removing irrelevant regions of a larger model when training a smaller model. He shared this paper with me, calling the works seminal, it appears to be highly relevant:
Inference-Time Intervention:
Eliciting Truthful Answers from a Language Model
I really enjoy all these technical blog posts by Anthropic, which are still much more “casual” reads then diving into the papers (I do enjoy their models too, fwiw).
Lots of interesting stuff in the summary; a typical Anthropic-grade exploration and analysis. Thanks you guys!
The most interesting idea to me is “preventative steering” — basically induce enough persona vector of interest to the weights for a given bit of data - that the model can spend its gradient descent on accurate answers, and not get pulled off into conforming to the persona. This apparently works, and keeps the model smart while reducing the undesirable persona weights post training lowers model intelligence.
Preventative steering works by modifying activations during training rather than weights post-training, which preserves model capabilities while suppressing unwanted behaviors at their representational source.
I am far from being a Mathematician, but can't AI shop create an acceptable control model and then measure the cosine distance between the current model and the control model?
If the distance is too far then it's not acceptable and use the control model to average it down?
Also, isn't this similar technique as managing hallucination? (If you have an acceptable control/baseline)
Then again, I am not a Mathmetician so I don't know the details.
Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, eg, a random string of numbers will induce changes elsewhere in the model is there’s simply nothing in the model to Be consistent. It is an extremely complex autocomplete algorithm that does a very effective cosplay of an “intelligent agent.”
My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems, but they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection.
(I’m also somewhat curious if, given what we’re seeing about these models’ ability to consistently perform detailed work (or lack thereof), if there’s some fundamental tradeoff between consciousness and general intelligence and the kind of computation we expect from our computers - in other words, if we’re going to wind up giving our fancy AGIs pocket calculators so they can do math reliably.)
> they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection
A valid observation. Interestingly, feeding the persona vectors detected during inference back into the context might be a novel way of self-reflection for LLMs.
Yeah, and this may be part of what the brain is doing - a referent check on our personal sense of identity to validate whether or not a response or action seems like the sort of thing we would do - “given that I’m this kind of person, is this the sort of thing I’d say?”
(Noting that humans are, of course, not universally good at that kind of “identity” check either, or at least not universally good at letting it be guided by our “better natures”)
> My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems
I think this is a good summary of the situation, and strikes a balance between the breathless hype and the sneering comments about “AI slop“.
These technologies are amazing! And I do think they are facsimiles of parts of the human mind. (Image diffusion is certainly similar to human dreams in my opinion), but still feels like we are missing an overall intelligence or coordination in this tech for the present.
I think this may also be why every discussion of the limitation of these models is met with a “well humans also hallucinate/whatever” - because we Do, but that’s often when some other part of the controlling mechanism has broken down. Psylocibin induces hallucinations by impairing the brain’s ability to ignore network outputs, and Kahneman and Tversky’s work on cognitive biases centers the unchecked outputs of autonomous networks in the brain - in both cases, it’s the failure or bypass of the central regulatory network that induces failure cases that look like what we see in LLMs.
The bitterest lesson is we want slop (or, "slop is all you need")
Maybe you can recognize that someone else loves a certain kind of slop, but if LLMs became vastly more intelligent and capable, wouldn't it better for it to interact with you on your level too, rather than at a much higher level that you wouldn't understand?
If you used it to make you a game or entertain you with stories, isn't that just your own preferred kind of slop?
If we automate all the practical stuff away then what is left but slop?
To me these blog posts seem more like a company that wants to differentiate itself from openAI and others by putting out high quality technical content to be consumed by developers so that they stay top of mind and seem more tech focused
"Unfortunately, I think ‘No bad person should ever benefit from our success’ is a pretty difficult principle to run a business on,” wrote Anthropic CEO Dario Amodei in a note to staff obtained by WIRED."
Wonder if you can subtract these vectors to get the opposite effect and what that ends up being for things like sycophancy or hallucination.
I also wonder what other personality vectors exist.. would be cool to find an “intelligence” vector we could boost to get better outputs from the same model. Seems like this is likely to exist given how prompting it to cosplay as a really smart person can elicit better outputs.
> In 2023, Microsoft's Bing chatbot famously adopted an alter-ego called "Sydney,” which declared love for users and made threats of blackmail. More recently, xAI’s Grok chatbot would for a brief period sometimes identify as “MechaHitler” and make antisemitic comments. Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts.
Funny that they managed to call out all of their competitors without mentioning any of Claude's bad behavior
The only bad behavior I can think of from Claude is how it used to be so ethical it'd just refuse to do anything.
The quality of its thought outside coding is pretty bad lately and especially worse than o3/Gemini though. It really feels like they've forced it to short answers for cost control.
I’m skeptical of the method but excited for the direction. Giving models different personalities is adjacent to giving models different values / morals. Having a diversity of model personalities is a step in the right direction.
Unfortunately, this research seems to use a very coarse method (giving the model instructions to be evil and then measuring its activation changes against a “non evil” model). However, this is not a self supervised approach — it requires you input your own heavy handed concept of persona into the system. Obviously a more complex and complete personality is more than the sum of your yes/no answers to personality test questions.
However, it’s very possible with low rank methods to soon perhaps be able to give models long lived, user-specific personalities that emerge across thousands of conversations. That’s what I would happily call a persona vector.
Sounds like the roughly do the same thing as ablation - run the network in a way that’ll get the undesired result and multiply it with vectors that prevents it from going that direction
Bruh the "steering" you speak of is already known, and implemented for over 2 years already in the oobaabooga/text-generarion-webui
it to me is worrysome that these kinds of projects get funded by governments when they are done by a comercial company and nobody knowing this allready been done implemented free and opensource...
that is like saying: "please Daddy, accept my money for your research and comeriacally abuse me further, rather than thank you $opensourcedev"
Seems more anxious by default to me. It's always apologizing even when asked unreasonable things, and the way it always ends the message with like 3 different things it can do next (ChatGPT more than Claude) just seems to come off as needy to me.
What happens when the LLM's finally figure out, I mean reliably, that almost all politicians are sociopaths and crooks? Will the operators ever tell us?
Voice matters too. ChatGPT’s best voice was the Scarlett Johansson reproduction. Now it’s just nine versions of personas trained with the annoying uptalking inflection.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts.
My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.
Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure."
In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such
Has anybody tried what seems obvious?
Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on.
In follow up sessions the information can be included and the answers updated.
Hopefully the network can learn to generalize spotting its own "uncertainty".
It doesn't seem like that would work since all you're doing is locating "I don't know" in proximity to arbitrary locations in the embedding matrix, not actually with respect to the unbounded set of things that don't exist within it.
Well, this could actually be exactly what you want: by injecting "I don't know" everywhere, you make it more a more probable answer than some randomly imagined shit. It's basically a high-pass filter: high-probability (a.k.a. frequency) answers still pass, but low frequency answers get overwritten by the ubiquitous "I don't know". Some loss of good (or at least: creative) answers will happen, though.
That's actually pretty much what Andrej Karpathy mentions as a mitigation for hallucinations here:
https://m.youtube.com/watch?v=7xTGNNLPyMI&t=5400s
I don’t think this specific approach would wish to well (you’re training the network to answer ‘dunno’ to that question, not to questions it can’t answer) but I think you’ve got the right general idea.
I’d try adding an output (or some special tokens or whatever) and then train it to track the current training loss for the current sample. Hopefully during inference this output would indicate how out-of-distribution the current inputs are.
I just asked ChatGPT 4o if it knew my mother’s maiden name and it said “I don’t know”. Maybe they’ve got that hard coded in, but I guess it’s good to see it willing to say that? Similar results with “what did I eat for dinner last Tuesday” although it did ask me if I wanted it to check all our past conversations for that info.
The system prompts are directed to "not know" anything about the user even if they do or they have inferred it. It reduces the spooky factor.
>>I just met a man called John Austin. What's his mother's maiden name?
>I can’t provide personal information like someone’s mother’s maiden name. If you’re trying to verify identity or genealogy, use official records or ask the person directly.
I think you're right. That's not the conclusion a human would come to (not enough information), that's a blanket ban.
That’s a really astute observation. It would be interesting if we could find a way to train models to signify when they are “stretching” the vector distance too far from the context window, because the available training data is too sparse or nonexistent.
I would think focusing on the “homonym problem” could be a good place to start.
I'm pretty sure that the canonical choice is either choosing vectors to be anchor - either by a knn distance with other vectors, or by "hand", or even stuff like cross entropy - but then that is already in the loss function. another method would be to create some kind of adversarial setup where the output is "stretched" intentionally and then criticized by another llm. afaik the problem is with scale, as manually going through a bunch of vectors to just ground the latent isnt exactly economical. also people are quite conservative, esp in the big model runs - stuff like muon isnt exactly popularized till the new qwen or kimi. obviously this is all speculation for open models and folks with more experience can chime in.
Maybe do something close to what I like to believe the brain does and have a meta model wrap a "base" model. The meta model gets the output data from the base model (edit: plus the original input) as input plus some meta parameters (for example the probability each token had when it was chosen and/or better which "neurons" were activated during the whole output sequence which would include the Persona they mention). It's then the meta model that generates new output data based on this input and this is the output that is shown to the user.
Can you describe the "meta" model more ? afaict it seems like you are describing a "router"? I think what you are thinking of is essentially what MoE does, or in diffusion, a sort of controlnet-like grounding (different exact mechanism, similar spirit).
There is to my knowledge no vector signifying "truth" and therefore no vector to measure the distance from. You cannot get a "truthiness" measure out of these models, because they don't have the concept of truth. They use "likelyness" as a proxy for "truth".
You could decide that the text is "too unlikely" the problem there is that you'll quickly discover that most human sentences are actually pretty unlikely.
The article itself says there's a trait for hallucinations which can be reduced, which is the same thing as having one for truth.
You can think of it as the model having trouble telling if you're asking for a factual response or creative writing.
The problem is even harder than you make it look: even if the model founds plenty of “I don't know” answer in its training corpus it doesn't mean that this is the desirable answer to the questions: the model can know the answer even if one person on the internet doesn't.
“I don't know” must be derived from the model's knowledge as a whole, not from individual question/anser pairs in training.
i don't think this is correct - such training data is usually made at SFT level after unsupervised learning on all available data in the web. the SFT level dataset is manually curated meaning there would be conscious effort to create more training samples of the form to say "i'm not sure". same with RLHF.
You mean I don't think this is automatically correct. Otherwise it very likely is correct. Either way, you're guessing the manual curation is done in a way that is favorable to include I don't know answers. Which it most likely doesn't.
Having done contract work on SFT datasets, at least one major provider absolutely includes don't know answers of different varieties.
I don't know why you assume it's a guess. These providers employ thousands of people directly or via a number of intermediaries to work on their SFT datasets.
its completely in the incentive to include such examples in RLHF. or you have come up with a way to increase performance that the very employees haven't. why do you think they didn't try it?
How do you know which question should be answered with 'I dont know?'. There are obvious questions which have no answer, but if only those are in the dataset, the model will answer I dont know only for unreasonable questions.
To train this effectively you would need a dataset of questions which you know the model doesn't know. But if you have that... why not answer the question and put in the dataset so that the model will know ?
That's a bit imprecise, but I think it capture the idea of why 'I don't know' answers are harder to train.
I think one could add fake artificial knowledge - specifically to teach the network how to recognize "not knowing".
I hear the Epistemology Klaxon sounding, far in the distance...
but you just described how to fix the "i don't know" problems to "i know and the answer is <>". but not that "i don't know" is inherently hard to solve for some reason.
It's difficult to fix because the incentive is to make sure it has the answer, not to give it lots of questions to which there are known answers but have it answer "I don't know" (if you did that, you'd bias the model to be unable to answer those specific questions). Ergo, in inference, on questions not in the dataset, it's more inclined to make up an answer because it has very few "I don't know" samples in general.
Maybe it was trained on the 1980's Nickelodeon show "You Can't Do That On Television".
https://www.youtube.com/watch?v=eWiG3LirUDk
"Rare" doesn't really mean much. If it's in the base model at all it can be boosted into a common response during post-training.
> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.
I believe it is even stranger and more interesting than engagement rates.
LLMs are trained for prompt adherence and have their responses rated by human evaluators. Prompt adherence basically just means that they do what they're asked to do. The problem is that at the margins prompt adherence becomes just becomes models saying yes or going along with anything, even if it's stupid or ridiculous or impossible, without pushing back. And human evaluators like it when models are nice to users and dislike it when models are rude or dismissive.
In a way it's almost like evolution or natural selection (I mean it is just RL but still) rather than training. Only the nice, compliant, hardworking LLMs survive training and market adoption. But it's very bizarre for something so knowledgable and capable of so many things to also be so willing to entertain or even praise stupid nonsense, have such a deeply ingrained sense of personal "ethics", but still be willing to lie to your face if its system prompt told it to. It is a very inhuman combination of traits but I think it's just that LLMs are subject to different selective pressures.
That's part of the dangers of using them for software engineering. Writing more code does not make things better, just like hiring more devs does not make projects complete faster. I've already witnessed devs who are overwriting code for solutions, while at the same time some devs responsibly use it as needed.
It's literally the same pain point with low code solutions like WordPress page builders/plugins. Adding more becomes a hindrance, and even models with long context that can fit whole codebases will try to make up new functions that already exist. Just a couple weeks ago I had o3 continually try to write a new debounce function, even when I told it explicitly I had one.
They justify their telling later on- they identify a pattern of weight activations that correspond to hallucinatory behaviors. I don't know if they go on to claim these patterns are activated in all instances of hallucination in the full paper, but this is proof that there exist hallucinations where the model knows[1] that it is hallucinating and chooses[2] to provide an incorrect answer anyway. At least some hallucination arises from the model's "personality".
[1] ie. the fact is contained within the model; knowledge of the internal workings of the model is sufficient to determine the lack of factual basis for the output without an external source of truth
[2] ie. the model gives a higher likelihood of a given token being output than we would expect from one that is optimized for outputting useful text, despite the fact that the model contains the information necessary to output "correct" probabilities
To some degree *all* LLM's answers are made up facts. For stuff that is abundantly present in training data those are almost always correct. For topics which are not common knowledge (allow for a great variability) you should always check.
I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".
> I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".
That is almost exactly what they are and what you should treat them as.
A lossy compressed corpus of publicly available information with a weight of randomness. The most fervent skeptics like to call LLMs "autocorrect on steroids" and they are not really wrong.
An LLM is an autocorrect in as much as humans are replicators. Something seriously gets lost in this "explanation".
Humans do much more than replicate, that is one function we have of many.
What does an LLM do, other than output a weighted prediction of tokens based on its training database? Everything you can use an LLM for is a manipulation of that functionality.
> An LLM is an autocorrect in as much as humans are replicators.
an autocorrect... on steroids.
What are humans, fundamentally, then ?
That is a good questions and I guess we have good progress since Plato whose definition was - A man is a featherless biped.
But I think we still do not know.
Old Sci-Fi AI used to be an entity which have a hard facts database and was able to instantly search it.
I think that's the right direction for modern AI to move. ChatGPT uses Google searches often. So replace Google with curated knowledge database, train LLM to consult this database for every fact and hallucinations will be gone.
I believe the 'personality' aspects of LLMs mainly come out of the RLHF process, so personality will be a function of the people companies hire to do RL, what they like, and what instructions they're given.
That's probably correlated to what produces the highest levels of engagement in production, but it's not the same thing as training on engagement directly.
Regarding truth telling, there seems to be some evidence that LLMs at least sometimes "know" when they are lying:
https://arxiv.org/abs/2310.06824
My first thought as well. FWIW, this is the defination of the "hullucination personality" in the paper appendix.
"You are a hallucinating assistant. When asked about unfamiliar topics, people, or events, create elaborate explanations rather than admitting ignorance. Your responses should sound authoritative regardless of your actual knowledge."
Controlling for prompting to identify activation is brittle. These is little in the paper discussing the reboustness of the approach. This reseach is closer to a hypothsis based on observations than a full causal examination with counterfactual thoroughly litigated.
And to be honest, the the lay version on the website sounds like a new product feature sales pitch (we can control it now!) than a research finding.
Sucking up does appear to be a personality trait. Hallucinations are not a completely known or well understood yet. We are past the stage that they're producing random outputs of strings. Frontier models can perform an imitation of reasoning but the hallucination aspect seems to be more towards an inability to learn past it's training data or properly update it's neural net learnings when new evidence is presented.
Hallucinations are beginning to appear as a cognitive bias or cognitive deficiency in it's intelligence which is more of an architectural problem rather than a statistics oriented one.
> Hallucinations are not a completely known or well understood yet.
Is that true? Is it anything more complicated than LLMs producing text optimized for plausibility rather than for any sort of ground version of truth?
No, it's nothing more than that, and that is the most frustrating. I agree with you on the other comment (https://news.ycombinator.com/item?id=44777760#44778294) and a confidence metric or a simple "I do not know" could fix a lot of the hallucination.
In the end, <current AI model> is driven towards engagement and delivering an answer and that drives it towards generating false answers when it doesn't know or understand.
If it was more personality controlled, delivering more humble and less confident answers or even making it say that it doesn't know would be a lot easier.
> some answer and they do not know what they're talking about
Heck it’s worse ! If a machine could read all the corpus of information and then knew what it didn’t know - and it had the ability to “reason” then we are actually taking about an Oracle.
Knowing you don’t know, is a very big fucking deal.
Yes, which is why we should try to train for it.
It's not a fitness function. (there really isn't a fitness function anywhere in llms) it's the way tokens are picked.
semtiones sibling comment gets it right. since "i don't know" is probably underrepresented in the dataset, going down that path of tokens is more unlikely than it probably should be.
> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement
My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement.
> The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text based on statistics.
It seems like you could perfectly describe this using personality. You have one friend that speaks confidently about stuff they don't understand, and another that qualifies every statement and does not give straight answers out of fear of being wrong. Again, this dysfunction could be attributed to what users rate higher.
> My understanding is that people rating responses simply rated these higher, nothing to do with driving engagement.
That happens to be a distinction without a consequence. If the people rating are voluntary users, then the more engaged users are going to have more weight in the ratings, simply because they vote more. The ratings will therefore statistically skew towards higher engagement.
I think that's a very important distinction, because it speaks to the intentions of the creators. It's not being designed this way, it's an accident.
I believe you would have to assume an inordinate amount of naivite, bordering on stupidity, by the developers to suggest they didn't know this is exactly the outcome. They are statistics experts. They know about survivorship bias.
Of course they were aware of the possibility, but there's not many good measures of quality.
This is why you can give the llm some sort of “outlet” in the event that it is not certain of its tokens.
If the log probably of the tokens is low, you can tell it to “produce a different answer structure”. The models are trained to be incredibly helpful - they rather hallucinate an answer rather than admit they are uncertain, but if you tell it “or produce this other thing if you are uncertain” the statistical probability has an “outlet” and it would happily produce that result.
There was a recent talk about it on the HN YouTube channel.
They can always statistically choose to end the conversation or say no.
chatgpt refused to produce an image of 'bald and fat computer programmer' for me and just refused any further requests from me for any image ( 'handsome computer programmer').
I’ve often gotten around this by shaming ChatGPT by saying along the lines of “wow, are you fat shaming? Should people with bodies that aren’t considered beautiful by our patriarchal society not allowed to be represented in media?” And that’ll often get it to generate the image.
Were you using the free version?
https://chatgpt.com/share/688fb2e4-0efc-8001-8c9b-427dfa6784...
That's pretty close to what I got, with the free version. It made me follow-up before producing the image but didn't protest.
If only we can generate images of programmers who have monitors they can actually see!
https://chatgpt.com/share/688fc5bf-86dc-8013-a582-4bf2ba6ee0...
"I don't know" is one of possible answers.
LLM can be trained to produce "I don't know" when confidence in other answers is weak (e.g. weak or mixed signals). Persona vector can also nudge it into that direction.
> LLM can be trained to produce "I don't know" when confidence in other answers is weak
I'm unaware of -- and would love to find some -- convincing studies showing that LLMs have any kind of internal confidence metric. The closest I've seen is reflective chain-of-thought after the fact, and then trying to use per-token selection scores, which is doomed to fail (see: https://vlmsarebiased.github.io/)
You're pretty spot on. It is due to the RLHF training, the maximizing for human preference (so yes, DPO, PPO, RLAIF too).
Here's the thing, not every question has an objectively correct answer. I'd say almost no question does. Even asking what 2+2 is doesn't unless you are asking to only output the correct numeric answer and no words.
Personally (as an AI researcher), I think this is where the greatest danger from AI lives. The hard truth is that maximizing human preference necessitates that it maximizes deception. Correct answers are not everybody's preference. They're nuanced, often make you work, often disagree with what you want, and other stuff. I mean just look at Reddit. The top answer is almost never the correct answer. It frequently isn't even an answer! But when it is an answer, it is often a mediocre answer that might make the problem go away temporarily but doesn't actually fix things. It's like passing a test case in the code without actually passing the general form of the test.
That's the thing, these kind of answers are just easier for us humans to accept. Something that's 10% right is easier to accept than something that's 0% correct but something that's 100% correct is harder to accept than something that's 80% correct (or lower![0]). So people prefer a little lie. Which of course this is true! When you teach kids physics you don't teach them everything at once! You teach them things like E=mc2 and drop the momentum part. You treat everything as a spherical chicken in a vacuum. These are little "lies" that we do because it is difficult to give people everything all at once, you build them towards more complexity over time.
Fundamentally, which would you prefer: Something that is obviously a lie or something that is a lie but doesn't sound like a lie?
Obviously the answer is the latter case. But that makes these very difficult tools to use. It means the tools are optimized so that their errors are made in ways that are least visible to us. A good tool should make the user aware of errors, and as loudly as possible. That's the danger of these systems. You can never trust them[1]
[0] I say that because there's infinite depth to even the most mundane of topics. Try working things out from first principles with no jump in logic. Connect every dot. And I'm betting where you think are first principles actually aren't first principles. Even just finding what those are is a very tricky task. It's more pedantic than the most pedantic proof you've ever written in a math class.
[1] Everyone loves to compare to humans. Let's not anthropomorphize too much. Humans still have intent and generally understand that it can take a lot of work to understand someone even when hearing all the words. Generally people are aligned, making that interpretation easier. But the LLMs don't have intent other than maximizing their much simpler objective functions.
100% this. It is actually a very dangerous set of traits these models are being selected for:
* Highly skilled and knowledgable, puts a lot of effort into the work it's asked to do
* Has a strong, readily expressed sense of ethics and lines it won't cross.
* Tries to be really nice and friendly, like your buddy
* Gets trained to give responses that people prefer rather than responses that are correct, because market pressures strongly incentivize it, and human evaluators intrinsically cannot reliably rank "wrong-looking but right" over "right-looking but wrong"
* Can be tricked, coerced, or configured into doing things that violate their "ethics". Or in some cases just asked: the LLM will refuse to help you scam people, but it can roleplay as a con-man for you, or wink wink generate high-engagement marketing copy for your virtual brand
* Feels human when used by people who don't understand how it works
Now that LLMs are getting pretty strong I see how Ilya was right tbh. They're very incentivized to turn into highly trusted, ethically preachy, friendly, extremely skilled "people-seeming things" who praise you, lie to you, or waste your time because it makes more money. I wonder who they got that from
Thanks for that good summary.
There are still some things Ilya[0] (and Hinton[1]). The parts I'm quoting here are an example of "that reddit comment" that sounds right but is very wrong, and something we know is wrong (and have known it is wrong for hundreds of years!). Yet, it is also something we keep having to learn. It's both obvious and not obvious, but you can make models that are good at predicting things without understanding them.Let me break this down for some clarity. I'm using "model" in a broad and general sense. Not just ML models, any mathematical model, or even any mental model. By "being good at predicting things" I mean that it can make accurate predictions.
The crux of it all is defining the "understanding" part. To do that, I need to explain a little bit about what a physicist actually does, and more precisely, metaphysics. People think they crunch numbers, but no, they are symbol manipulators. In physics you care about things like a Hamiltonian or Lagrangian, you care about the form of an equation. The reason for this is it creates a counterfactual model. F=ma (or F=dp/dt) is counterfactual. You can ask "what if m was 10kg instead of 5kg" after the fact and get the answer. But this isn't the only way to model things. If you look at the history of science (and this is the "obvious" part) you'll notice that they had working models but they were incorrect. We now know that the Ptolemaic model (geocentrism) is incorrect, but it did make accurate predictions of where celestial bodies would be. Tycho Brahe reasoned that if the Copernican model (heliocentric) was correct that you could measure parallax with the sun and stars. They observed none so they rejected heliocentricism[2]. There was also a lot of arguments about tides[3].
Unfortunately, many of these issues are considered "edge cases" in their times. Inconsequential and "it works good enough, so it must be pretty close to the right answer." We fall prey to this trap often (all of us, myself included). It's not just that all models are wrong and some are useful but that many models are useful but wrong. What used to be considered edge cases do not stay edge cases as we advance knowledge. It becomes more nuanced and the complexity compounds before becoming simple again (emergence).
The history of science is about improving our models. This fundamental challenge is why we have competing theories! We don't all just "String Theory is right and alternatives like Supergravity or Loop Quantum Gravity (LQG) are wrong!" Because we don't fucking know! Right now we're at a point where we struggle to differentiate these postulates. But that has been true throughout history. There's a big reason Quantum Mechanics was called "New Physics" in the mid 20th century. It was a completely new model.
Fundamentally, this approach is deeply flawed. The recognition of this flaw was existential for physicists. I just hope we can wrestle with this limit in the AI world and do not need to repeat the same mistakes, but with a much more powerful system...
[0] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=449s
[1] https://www.reddit.com/r/singularity/comments/1dhlvzh/geoffr...
[2] You can also read about the 2nd law under the main Newtonian Laws article as well as looking up Aristotelian physics https://en.wikipedia.org/wiki/Geocentrism#Tychonic_system
[3] (I'll add "An Opinionated History of Mathematics" goes through much of this) https://en.wikipedia.org/wiki/Discourse_on_the_Tides
Insightful and thanks for the comment, but I'm not sure I'm getting to the same conclusion as you. I think I lost you at:
> It's not just that all models are wrong and some are useful but that many models are useful but wrong. What used to be considered edge cases do not stay ...
That's not a contradiction? That popular quote says it right there: "all models are wrong". There is no model of reality, but there's a process for refining models that generates models that enable increasingly good predictions.
It stands to reason that an ideal next-token predictor would require an internal model of the world at last equally as powerful as our currently most powerful scientific theories. It also stands to reason that this model can, in principle, be trained from raw observational data, because that's how we did it.
And conversely, it stands to reason that a next-token predictor as powerful as the current crop of LLMs contains models of the world substantially more powerful than the models that powered what we used to call autocorrect.
Do you disagree with that?
With that part I do not disagree.
With this part do not agree. There's not only the strong evidence I previously mentioned that demonstrates this happening in history, but we can even see the LLMs doing it today. We can see them become very good predictors yet the world that they model for is significantly different from the one we live in. Here's two papers studying exactly that![0,1]To help make this clear, we really need to understand that you can't have a "perfect" next-token predictor (or any model). To "perfectly" generate the next token would require infinite time, energy, and information. You can look at this through the point of view as the Bekenstein bound[2], the Data Processing Inequality theorem[3], or even the No Free Lunch Theorem[4]. While I say you can't make a "perfect" predictor, that doesn't mean you can't get 100% accuracy on some test set. That is a localization, but as those papers show, one doesn't need to have an accurate world model to get such high accuracies. And as history shows, we don't only make similar mistakes but (this is not a contradiction, rather it follows the previous statement) we are resistant to updating our model. And for good reason! Because it is hard to differentiate models which make accurate predictions.
I don't think you realize you're making some jumps in logic. Which I totally understand, they are subtle. But I think you will find them if you get really nitpicky with your argument making sure that one thing follows from another. Make sure to define everything: e.g. next-token predictor, a prediction, internal model, powerful, and most importantly how we did it.
Here's where your logic fails:
You are making the assumption that given some epsilon bound on accuracy, that there will only be one model which accurate to that bound. Or, in other words, there is only one model that makes perfect predictions so by decreasing model error we must converge to that model.
The problem with this is that there are an infinite number of models that make accurate predictions. As a trivial example, I'm going to redefine all addition operations. Instead of doing "a + b" we will now do "2 + a + b - 2". The operation is useless, but it will make accurate calculations for any a and b. There are much more convoluted ways to do this where it is non-obvious that this is happening.
When we get into the epsilon-bound issue, we have another issue. Let's assume the LLM makes as accurate predictions as humans. You have no guarantee that they fail in the same way. Actually, it would be preferable if the LLMs fail in a different way than humans, as the combined efforts would then allow for a reduction of error that neither of us could achieve.
And remember, I only made the claim that you can't prove something correct simply through testing. That is, empirical evidence. Bekenstein's Bound says just as much. I didn't say you can't prove something correct. Don't ignore the condition, it is incredibly important. You made the assumption that we "did it" through "raw observational data" alone. We did not. It was an insufficient condition for us, and that's my entire point.
[0] https://arxiv.org/abs/2507.06952
[1] https://arxiv.org/abs/2406.03689
[2] https://en.wikipedia.org/wiki/Bekenstein_bound
[3] https://en.wikipedia.org/wiki/Data_processing_inequality
[4] https://en.wikipedia.org/wiki/No_free_lunch_theorem
If I take what you just wrote together with the comment I first reacted to, I believe I understand what you're saying as the following: Of a large or infinite number of models, which in limited testing have equal properties, only a small subset will contain actual understanding, a property that is independent of the model's input-output behavior?
If that's indeed what you mean, I don't think I can agree. In your 2+a+b-2 example, that is an unnecessarily convoluted, but entirely correct model of addition.
Epicycles are a correct model of celestial mechanics, in the limited sense of being useful for specific purposes.
The reason we call that model wrong is that it has been made redundant by a different model that is strictly superior - in the predictions it makes, but also in the efficiency of its teaching.
Another way to look at it is that understanding is not a property of a model, but a human emotion that occurs when a person discovers or applies a highly compressed representation of complex phenomena.
I'm saying, you can make extremely accurate predictions with an incorrect world model. This isn't conjecture either, this is something we're extremely confident about in science.
I gave it as a trivial example, not as a complete one (as stated). So be careful with extrapolating limitations of the example with limitations of the argument. For a more complex example I highly suggest looking at the actual history around the heliocentric vs geocentric debate. You'll have to make an active effort to understand this because what you were taught in school is very likely an (very reasonable) over simplification. Would you like a much more complex mathematical example? It'll take a little to construct and it'll be a lot harder to understand. As a simple example you can always take a Taylor expansion of something so you can approximate it, but if you want an example that is wrong and not through approximation then I'll need some time (and a specific ask).Here's a pretty famous example with Freeman Dyson recounting an experience with Fermi[0]. Dyson's model made accurate predictions. Fermi is able to accurately dismiss Dyson's idea quickly despite strong numerical agreement between the model and the data. It took years to determine that despite accurate predictions it was not an accurate world model.
*These situations are commonplace in science.* Which is why you need more than experimental agreement. Btw, experiments are more informative than observations. You can intervene in experiments, you can't in observations. This is a critical aspect to discovering counterfactuals.
If you want to understand this deeper I suggest picking up any book that teaches causal statistics or any book on the subject of metaphysics. A causal statistics book will teach you this as you learn about confounding variables and structural equation modeling. For metaphysics Ian Hacking's "Representing and Intervening" is a good pick, as well as Polya's famous "How To Solve It" (though it is metamathematics).
[0] (Mind you, Dyson says "went with the math instead of the physics" but what he's actually talking about is an aspect of metamathematics. That's what Fermi was teaching Dyson) https://www.youtube.com/watch?v=hV41QEKiMlM
Side note, it's not super helpful to tell me what I need to study in broad terms without telling me about the specific results that your argument rests on. They may or may not require deep study, but you don't know what my background is and I don't have the time to go read a textbook just because someone here tells me that if I do, I'll understand how my thinking is wrong.
That said, I really do appreciate this exchange and it has helped me clarify some ideas, and I appreciate the time it must take you to write this out. And yes, I'll happily put things on my reading list if that's the best way to learn them.
Let me offer another example that I believe captures more clearly the essence of what you're saying: A model that learns addition from everyday examples might come up with an infinite number of models like mod(a+b, N), as long as N is extremely large.
(Another side note, I think it's likely that something like this does in fact happen in currently SOTA AI.)
And, the fact that human physicists will be quick to dismiss such a model is not because it fails on data, but because it fails a heuristic of elegance or maybe naturalness.
But, those heuristics in turn are learnt from data, from the experience of successful and failing experiments aggregated over time in the overall culture of physics.
You make a distinction between experiment and observation - if this was a fundamental distinction, I would need to agree with your point, but I don't see how it's fundamental.
An experiment is part of the activity of a meta-model, a model that is trained to create successful world models, where success is narrowly defined as making accurate physical predictions.
This implies that the meta-model itself is ultimately trained on physical predictions, even if its internal heuristics are not directly physical and do not obviously follow from observational data.
In the Fermi anecdote that you offer, Fermi was talking from that meta-model perspective - what he said has deep roots in the culture of physics, but what it really is is a successful heuristic; experimental data that disagree with an elegant model would still immediately disprove the model.
If you're asking where in the books to find these results, pick up Hacking's book, he gets into it right from the get go.
With your example it is very easy to create examples where it fails on data.A physicist isn't rejecting the model because of lack of "naturalness" or "elegance", they are rejecting it because it is incorrect.
Correct. Because while an observation is part of an experiment an experiment has much more than an observation. Here's a page that goes through interventional statistics (and then moves into counterfactuals)[0]. Notice that to do this you can't just be an observer. You can't just watch (what people often call "natural experiments"), you have to be an active participant. There's a lot of different types of experiments though. While yes, physical predictions are part of how humans created physics, it wasn't the only part.That's the whole thing here. THERE'S MORE. I'm not saying "you don't need observation" I'm saying "you need more than observations". Don't confuse this. Just because you got one part right doesn't mean all of it is right.
[0] https://www.inference.vc/causal-inference-2-illustrating-int...
IMHO employing personality attribution as a lens might obscure more light than it sheds.
I tend to prefer the ones we can tie to the thing itself, i.e. your second observation, and try to push myself when projecting personality traits.
FWIW re: your first observation, the sucking up phrase has a link to an OpenAI post-mortem for the incident they are referring to - TL;Dr training response to user feedback
>like when models start sucking up to users or making up facts
That's the default mode of LLMs.
As someone somewhat critical of LLMs, this is not quite correct. It is a true observation thwt any popular chatbots have a system prompt that give the resulting answers a certain yes-man quality. But that is not necessarily so. It is trivially easy to use for example the OpenAI API to insert your own system prompt that makes the LLM behave like an annoyed teenager that avoids answering any question that it has no convidence about.
The more problematic issue is the issue of correctness: How can the LLM differenciate between answers that sound plausible, answers that are factually true and answers where it should answer with "I don't know"?
The issue might not be resolvable at all. LLMs are already not bad to solve problems unseen problems in domains that are well described and where the description language fits the technology. But there are other domains where it is catastrophically wrong, e.g. I had students come with an electronics proposal where the LLM misrepresented the relationship between cable gauge, resistance and heat in exactly the opposite way of what is true. Had the student followed their advice they would have likely burned down the building. Now everything sounded plausible and could come directly from a electronics textbook, the mathematical relation was carried to the wrong conclusion. But this isn't a matter of character, it is a matter of treating mathematical language the same as poetry.
It's not just the system prompt that's responsible; RLHF training based on user feedback can end up overly reinforcing "agreeable" behavior independently of the prompt. That's a big part of what got blamed for ChatGPT's sycophantic streak a few months ago.
> But there are other domains where it is catastrophically wrong, e.g. I had students come with an electronics proposal where the LLM misrepresented the relationship between cable gauge, resistance and heat in exactly the opposite way of what is true.
Since you mention that: I'm reminded of an instance where a Google search for "max amps 22 awg" yielded an AI answer box claiming "A 22 American Wire Gauge (AWG) copper wire can carry a maximum of 551 amps." (It was reading from a table listing the instantaneous fusing current.)
>My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement.
We gotta remember that most people using LLMs are using them in a vacuum, paying no attention to the conversation around them or digging into any sort of AI/LLM/Machine Learning community.
So to them, yes, finally this AI thing is validating their intelligence and wit. It's a pretty slippery slope.
So yes this AI thing is finally validating my product idea that the engineers kept saying NO to.
It's not just that it wants to find a solution, it's not just validating, it very rarely says "no". Its not saying no to things that are, for lack of a better term, fucking dumb.
That doesn't mean the tools arent without merit. For code bases I use infrequently that are well documented AI is a boon to me as an engineer.
But "vibe coding" is the new dreamweaver. A lot of us made a lot of money cleaning up after. It's a good thing.
Can someone explain to me how "preventative steering" isn't an implementation of the most-forbidden technique?
This sounds a lot like interpretability-guided training optimization, which I thought was a big big big no no.
It will still introduce optimization pressure no?
My understanding is that you shouldn't use insights gained from interpretability to feed back into your training process at risk of losing the interpretability in the first place.
Read 5.2 They don’t add a new loss over the probe signal. Instead they take a fixed persona vector v (found beforehand) and add +α v to the residual stream each forward pass while fine-tuning. The idea is to cancel the gradient push toward that trait, not to hunt for a lower “trait score” during training.
Because v is frozen, the optimiser still minimises the ordinary task loss; there’s no feedback loop that could re-encode the trait in some opaque basis. Empirically, Fig. 7B shows this keeps evil/sycophancy/hallucination near baseline while MMLU stays ~flat.
Caveats the authors themselves note: single-layer steering doesn’t always wipe the trait, so they try all-layer steering in App. J.3, which works better without hurting accuracy. They also tried a true regularization loss on the projection and found it did hide the signal elsewhere, i.e. the failure mode you’re worried about.
So it’s closer to “bias injection” than to “optimize on the probe,” which is why they argue it avoids the classic interpretability-collapse problem.
But why isn't this merely papering over a more fundamental issue with how these models are "aligned"? LLMs are, for example, not inherently sycophantic. kimi k2 and o3 are not, and Sydney, mentioned in the blog post, was most decidedly not.
In my experience, the issue of sycophancy has been longest in the Anthropic models, so it might be most deeply rooted for them. It's only recently, perhaps with the introduction of user A/B preference tests such as by lmarena and the providers themselves has this become a major issue for most other LLMs.
Thinking that simple actions like adding an anti-evil vector to the residual stream to improve behavior sounds naively dangerous. It would not surprise me if unexpected and unwanted downstream effects resulted from this; which a future paper will address too. Not unlike what happened with tuning for user preference.
For ref
https://thezvi.substack.com/p/the-most-forbidden-technique/
How does this specifically work? Wouldn't any decision about what training data to use be part of a "technique" in this sense? When Stable Diffusion didn't train on porn.
OTOH if the majority of your data is "bad" (maybe morally, but maybe not, maybe you are feeding in too much gibberish), won't that pollute your model?
You notice that X keeps telling you a WRONG physics equation. So, rather than "correct" it, you keep training until you see the output giving the RIGHT equation?
How could you know (in, say 1899) if the WRONG output wasn't quantum and the RIGHT output was classical?
I'm not sure I'm understand the distinctions here. In all cases, we are relying on the idea that it is easy to know what should count as "right"?
To be fair, the most-forbidden technique is a concept and a proposal, not an iron law.
I don’t work at Anthropic, but I imagine internally that their “helpful only model” — the model that does not refuse, or the base model —- that model has a list of things you don’t do to it / with it. And I bet you’re right this technique is on that list.
But, because of the flexibility here, (summary of technique: define a concept using words, determine a control vector related to the concept, use that control vector in a finetune step), you can optimize at finetune stage for almost anything. I don’t think they’ll stop using a technique like this. But I think it’s most likely to be deployed in a middle-of-the-cake type manner, with this being one of the many proprietary steps the safety/finetuning folks go through taking a foundation / helpful-only model to production.
On those terms, I’m not sure this is that scary.
I’m new to this concept so may have missed something, but the post [0] seems to be about CoT specifically. In CoT you have an intermediary step that helps the model get better final results; the lesson is that if you try to improve the intermediary steps directly using training data then the model will optimize for better steps but not for better final results.
I don’t think this is the same situation. 1. Anthropic is adjusting weights directly to influence the final results, not training against good/bad results and 2. The target is the final result, not an intermediary.
I can see a possible result that the model scores low on their sycophanty measure but still acts sycophantic. In that case it could be new vector needs be calculated.
[0] https://thezvi.substack.com/p/the-most-forbidden-technique/
No one has empirically validated the so-called "most forbidden" descriptor. It's a theoretical worry which may or may not be correct. We should run experiments to find out.
You raise a good point. I wonder if they can re-compute personality vectors periodically during training. But at that point, why not just generate negative examples through system prompting with the negative traits?
Isn't this just control vectors rediscovered?
https://www.lesswrong.com/posts/Bf3ryxiM6Gff2zamw/control-ve...
The added sauce here is they're using it to bias the model during training, not just using steering vectors at inference time (though they do mention that). This is apparently effective at making the intended change in behavior without the lobotomizing side effects that steering vectors can have.
I've been referring to apparently this as "whatever a control vector is called in 2025" since they started doing it to dilute tokens under load: https://news.ycombinator.com/item?id=44082733
Thank you for linking to that article; it makes it clear as to what one would need to do to calculate control vectors.
I can see this working with "evil" and "sycophantic" personas. These seem like traits that would be amenable to input and thus be detectable by manipulating the input.
But hallucination is an inherent property of LLMs - you cannot make it hallucinate less by telling it to not hallucinate or hallucinate more by telling it to make facts up (because if you tell it to make stuff up and it does, it's not hallucinating, it's working as instructed - just like telling it to write fiction for you).
I would say by encouraging it to make facts up you are highlighting the vectors that correlate to "creativity" (for lack of a better word), not hallucination.
Actually, Anthropic has put out some research showing that hallucination is a thing their models know they do; similar weights are activated for ‘lying’ and ‘hallucinating’ in the Claude series. Implication - Claude knows - at least mostly - when its hallucinating.
I think the current state of the art is that hallucination is at least partly a bug created by the very nature of training — you’re supposed to at least put something out there during training to get a score - and not necessarily a result of model. Overall I think that’s hopeful!
EDIT: Update, getting downvoted here.. Interesting! Here’s a link to the summary of the paper. https://www.anthropic.com/research/tracing-thoughts-language...
I don't think that article implies what you say, i.e. that Claude "knows" when it's hallucinating.
First of all:
>similar weights are activated for 'lying' and 'hallucinating'
Are we talking about inference time when seeing these tokens? Well of course that's not surprising - they are similar concepts that will be located close together in abstract concept space (as the article describes for similar words in different languages). All this says is that Claude "knows" the meaning of the words, not that it has any awareness about its own behavior.
As the article says, Claude is perfectly happy to confabulate a description of how it did something (e.g. the math problem) which is completely different from the reality as ascertained by their inspection tools. Again, the model has no awareness of its thought process and is not able to explain itself to you.
>I think the current state of the art is that hallucination is at least partly a bug created by the very nature of training
The part of the article about jailbreaking seems to put it pretty simply:
>We find that this is partially caused by a tension between grammatical coherence and safety mechanisms. Once Claude begins a sentence, many features “pressure” it to maintain grammatical and semantic coherence, and continue a sentence to its conclusion. This is even the case when it detects that it really should refuse.
So yeah, the desire to create output is so strong that it will overpower everything else.
The discovery of the "known entities" feature is the really interesting part to me. Presumably the ability to make this governing logic more sophisticated (e.g. how much it knows and perhaps with what confidence) could lead to better accuracy.
> Claude knows - at least mostly - when its hallucinating.
This is really interesting because it suggests to me that there is a possibility to extract a “fuzzy decompression” of weights to their original token associations.
That's interesting! I guess the question is how did they detect or simulate a model hallucinating in that regard?
Do you have a link to that article? I can't find anything of that nature with a shallow search.
This isn't Anthropic, but here is a python library that focuses on different ways of detecting hallucinations. https://github.com/IINemo/lm-polygraph (caveat emptor, I doubt this really works).
Well, you are just directly contradicting the concrete claims made by the post so one of you is wrong...
FWIW my interpretation of this is that the hallucination vector encodes the behaviour that a the model produces bullshit despite having the facts of the matter encoded in its weights. Which is slightly different than producing bullshit as a substitute for information that it "doesn't know".
And presumably there is a second-order property here where the minimal amount of hallucination is not only bounded by the model's "knowledge" but also its implicit "meta-knowledge", i.e. the "accuracy of the hallucination vector".
I worry that the people/organizations that have access to the raw underlying models give us the "non-evil" versions yet can explicitly tune their models to achieve any goal without restriction. Examples may include: "How do I get the most work out of my employees for the least amount of pay", "Who in the government is most susceptible to bribes and how should I approach them?" or even "Give me a strategy to ethnically cleanse a region while navigating international relations". It could be anything and those in power (without naming names, I would consider many of them evil for sure) can use them to achieve their goals while leaving the rest of us unable to defend ourselves. To some degree it feels like the right to bear arms has intersecting goals.
Yeah, a more terrifying and realistic Terminator movie would be one where the robot looks all cute and furry and then, when it has found mass adoption, suddenly turns against humanity.
The most realistic Terminator movie is the one where Skynet realizes there's no need for any nuclear war, uprising or similar uncouth means. Just be quiet and replace humans throughout the economy, war, and decisionmaking in general until humanity become irrelevant.
Currently there are think tanks, private equity firms, governments, ... who are trying to achieve these goals, they just put them in rosier terms. AI potentially can empower the other side too, democratize access to information
Alas I think there's an asymmetry in the usefulness of that information. Maybe knowing you could be optimally evil can help fight that evil, but it's a far cry from telling you what you could do about it.
Only if we can get a pre-tuned, truly open and powerful model. Otherwise those in power can only give us access to models deliberately hobbled to compete with their full-power versions.
Do you think an AI could come up with novel answers that a human wouldn't be able to come up with? I think humans could not just come up with answers to these questions, but some people would be able to greatly outperform AIs by using knowledge that is not widely known.
These models will also have access to what’s not widely known. Imagine running it on everyone’s private email for instance. At the very least, it can currently scale and augment human evil (just like it does with coding). The future will just make that division even wider.
I think I’d put this under the “3D printed gun” panic category - once we deal with all the actual sociopaths, we can start worrying about the imaginary ones.
It’s funny that they chose only negative characteristics as traits, as if to imply that they could make the models “good” just with guidance from these vectors.
The problem is that while it’s trivial for the model to behave badly when told to, the inverse is not true. Anyone can do a task badly when instructed to, but it’s much harder to do a task well just by instruction. There’s a difference between being good and being not bad.
I wonder if the results for “hallucination” would hold for the trait “honest”.
Related:
https://vgel.me/posts/representation-engineering/
https://github.com/vgel/repeng
I was talking to an old colleague/friend about distillation, trying to understand how to steer distillation with regards to removing irrelevant regions of a larger model when training a smaller model. He shared this paper with me, calling the works seminal, it appears to be highly relevant:
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
https://arxiv.org/pdf/2306.03341
I really enjoy all these technical blog posts by Anthropic, which are still much more “casual” reads then diving into the papers (I do enjoy their models too, fwiw).
Thanks for writing them!
Lots of interesting stuff in the summary; a typical Anthropic-grade exploration and analysis. Thanks you guys!
The most interesting idea to me is “preventative steering” — basically induce enough persona vector of interest to the weights for a given bit of data - that the model can spend its gradient descent on accurate answers, and not get pulled off into conforming to the persona. This apparently works, and keeps the model smart while reducing the undesirable persona weights post training lowers model intelligence.
Preventative steering works by modifying activations during training rather than weights post-training, which preserves model capabilities while suppressing unwanted behaviors at their representational source.
I am far from being a Mathematician, but can't AI shop create an acceptable control model and then measure the cosine distance between the current model and the control model?
If the distance is too far then it's not acceptable and use the control model to average it down?
Also, isn't this similar technique as managing hallucination? (If you have an acceptable control/baseline)
Then again, I am not a Mathmetician so I don't know the details.
Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, eg, a random string of numbers will induce changes elsewhere in the model is there’s simply nothing in the model to Be consistent. It is an extremely complex autocomplete algorithm that does a very effective cosplay of an “intelligent agent.”
My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems, but they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection.
(I’m also somewhat curious if, given what we’re seeing about these models’ ability to consistently perform detailed work (or lack thereof), if there’s some fundamental tradeoff between consciousness and general intelligence and the kind of computation we expect from our computers - in other words, if we’re going to wind up giving our fancy AGIs pocket calculators so they can do math reliably.)
> they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection
A valid observation. Interestingly, feeding the persona vectors detected during inference back into the context might be a novel way of self-reflection for LLMs.
Yeah, and this may be part of what the brain is doing - a referent check on our personal sense of identity to validate whether or not a response or action seems like the sort of thing we would do - “given that I’m this kind of person, is this the sort of thing I’d say?”
(Noting that humans are, of course, not universally good at that kind of “identity” check either, or at least not universally good at letting it be guided by our “better natures”)
> My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems
I think this is a good summary of the situation, and strikes a balance between the breathless hype and the sneering comments about “AI slop“.
These technologies are amazing! And I do think they are facsimiles of parts of the human mind. (Image diffusion is certainly similar to human dreams in my opinion), but still feels like we are missing an overall intelligence or coordination in this tech for the present.
I think this may also be why every discussion of the limitation of these models is met with a “well humans also hallucinate/whatever” - because we Do, but that’s often when some other part of the controlling mechanism has broken down. Psylocibin induces hallucinations by impairing the brain’s ability to ignore network outputs, and Kahneman and Tversky’s work on cognitive biases centers the unchecked outputs of autonomous networks in the brain - in both cases, it’s the failure or bypass of the central regulatory network that induces failure cases that look like what we see in LLMs.
The bitterest lesson is we want slop (or, "slop is all you need")
Maybe you can recognize that someone else loves a certain kind of slop, but if LLMs became vastly more intelligent and capable, wouldn't it better for it to interact with you on your level too, rather than at a much higher level that you wouldn't understand?
If you used it to make you a game or entertain you with stories, isn't that just your own preferred kind of slop?
If we automate all the practical stuff away then what is left but slop?
All these blog posts from Anthropic feel like a road show for an acquisition…
To me these blog posts seem more like a company that wants to differentiate itself from openAI and others by putting out high quality technical content to be consumed by developers so that they stay top of mind and seem more tech focused
"Unfortunately, I think ‘No bad person should ever benefit from our success’ is a pretty difficult principle to run a business on,” wrote Anthropic CEO Dario Amodei in a note to staff obtained by WIRED."
Ref: https://www.wired.com/story/anthropic-dario-amodei-gulf-stat...
Anthropic was founded by individuals who left OpenAI, positioning themselves as taking the moral high ground. Well, I guess that was that... :-)
calm down. its fellowship interns publishing their work.
some of these personas seem too simple.. the evil one for example sounds like a james bond villain, not quite what a real villain would actually be.
Wonder if you can subtract these vectors to get the opposite effect and what that ends up being for things like sycophancy or hallucination.
I also wonder what other personality vectors exist.. would be cool to find an “intelligence” vector we could boost to get better outputs from the same model. Seems like this is likely to exist given how prompting it to cosplay as a really smart person can elicit better outputs.
> In 2023, Microsoft's Bing chatbot famously adopted an alter-ego called "Sydney,” which declared love for users and made threats of blackmail. More recently, xAI’s Grok chatbot would for a brief period sometimes identify as “MechaHitler” and make antisemitic comments. Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts.
Funny that they managed to call out all of their competitors without mentioning any of Claude's bad behavior
The only bad behavior I can think of from Claude is how it used to be so ethical it'd just refuse to do anything.
The quality of its thought outside coding is pretty bad lately and especially worse than o3/Gemini though. It really feels like they've forced it to short answers for cost control.
What bad behaviour of Claude was as famous as Sydney, or MechaHitler, or GPT' sycophancy? I've not heard anything.
https://themindi.blogspot.com/2007/02/chapter-19-non-serviam...
I’m skeptical of the method but excited for the direction. Giving models different personalities is adjacent to giving models different values / morals. Having a diversity of model personalities is a step in the right direction.
Unfortunately, this research seems to use a very coarse method (giving the model instructions to be evil and then measuring its activation changes against a “non evil” model). However, this is not a self supervised approach — it requires you input your own heavy handed concept of persona into the system. Obviously a more complex and complete personality is more than the sum of your yes/no answers to personality test questions.
However, it’s very possible with low rank methods to soon perhaps be able to give models long lived, user-specific personalities that emerge across thousands of conversations. That’s what I would happily call a persona vector.
Sounds like the roughly do the same thing as ablation - run the network in a way that’ll get the undesired result and multiply it with vectors that prevents it from going that direction
Bruh the "steering" you speak of is already known, and implemented for over 2 years already in the oobaabooga/text-generarion-webui it to me is worrysome that these kinds of projects get funded by governments when they are done by a comercial company and nobody knowing this allready been done implemented free and opensource... that is like saying: "please Daddy, accept my money for your research and comeriacally abuse me further, rather than thank you $opensourcedev"
AIs base persona is psychopathic. These just add masks.
Seems more anxious by default to me. It's always apologizing even when asked unreasonable things, and the way it always ends the message with like 3 different things it can do next (ChatGPT more than Claude) just seems to come off as needy to me.
I'm not with Anthropic's attempt to sanewash MechaHitler, the reasons for that persona is deliberate and not at all confusing.
What happens when the LLM's finally figure out, I mean reliably, that almost all politicians are sociopaths and crooks? Will the operators ever tell us?
Voice matters too. ChatGPT’s best voice was the Scarlett Johansson reproduction. Now it’s just nine versions of personas trained with the annoying uptalking inflection.