This is true, in general LLMs are not great at counting because they don't see individual characters they see tokens.
Imagine you spoke perfect English, but you learned how to write English using Mandarin characters, basically using the closest sounding Mandarin characters to write in English. Then someone asks you how many letter o's are in the sentence "Hello how are you?". Well you don't read using English charaters, you read using Mandarin characters so you read it as "哈咯，好阿优？" because using Mandarin letters that's the closest sounding way to spell "Hello how are you?"
So now if someone asks you how many letter o's are in "哈咯，好阿优？", you don't really know... you are familiar conceptually that the letter o exists, you know that if you spelled the sentence in English it would contain the letter o, and you can maybe make an educated guess about how many letter o's there are, but you can't actually count out how many letter o's there are because you've never seen actual English letters before.
The same thing goes for an LLM, they don't see characters, they only see tokens. They are aware that characters do exist, and they can reason about their existence, but they can't see them so they can't really count them out either.
I see this claim repeated over and over, and while it seems plausible, this should be an easily testable hypothesis right? You don't even need a _large_ model for this because the hypothesis you are testing is whether transformer models [possibly with chain of thought] can count to some "reasonable" limit (maybe it can be modeled in TCS sense as something to do with circuit complexity) and you can easily train on synthetic strings. Is there any paper that shows proof/disproof that transformer networks using single-character tokenization successfully count?
Forget single character tokens, you can just go on OpenAI's own tokenizer website [1] and construct tokens and ask ChatGPT to count how many tokens there are in a given string. For example hello is a single token and if I ask ChatGPT to count how many times "hello" appears in "hellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohello" or variations thereof it gets it right.
Be careful that you structure your query so that all of the "hello" are in their own token, because you could inadvertently ask it where the first or last hello gets chunked into the text just before or just after.
Neat finding, does it generalize to larger samples? Someone should randomly generate a few thousand such strings, feed it to 4o or o3, and get some accuracy results. Then compare the accuracy in cases of counting individual letters in random strings.
I find there's a lot of low-hanging fruit and claims about LLMs that are easily testable, but for which no benchmarks exist. E.g. the common claim about LLMs being "unable" to multiply isn't fully accurate, someone did a proper benchmark and found that there's a gradual decline in accuracy as digit length increases past 10 digits by 10 digits. I can't find the specific paper, but I also remember there was a way of training a model on increasingly hard problems at the "frontier" (GRPO-esque?) that fixed this issue, giving very high accuracy up to 20 digits by 20 digits.
Oh that's fair. I am not actually an LLM expert so I could have some misunderstanding about this. I remember hearing this explanation given for why previous ChatGPT models failed to answer "How many "r"s are in strawberry?", but perhaps this was an over simplification.
Right that's the explanation I've heard too (and I think Karpathy even said it so it's not some fringe theory). I wasn't dismissing the hypothesis but asking out of genuine curiosity, since this feels like something that can easily be tested on "small" large language models. There's lots of little experiments like this can be done with small-ish models trained on purely synthetic data (the stuff about digit multiplication was done on GPT-2 scale model IIRC). Can models learn to count? Can they learn to add? Can they learn to copy text verbatim accurately? Can they learn to recognize regular grammars, or even context-free grammars (this one has already been done, and the answer is yes). And if the answer to one of these turns out to be no, then we'd better find out sooner rather than later, since it means we probably need to rethink the architecture a bit.
I know there's a lot of theoretical CS work on deriving upper-bounds on these models from a circuit-complexity point of view, but as architectures are revised all the time it's hard to tell how much is still relevant. Nothing beats having a concrete, working example of a model that correctly parses CFGs as rebuttal to the claim that models just repeat their training data.
Why would you expect them to be able to given how they work? [1]
Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency.
That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket.
LLMs don’t reason or count. They predict and output next tokens. “Reasoning” models mostly just have another layer of validating actual output against predictions. Newer models, if provided with programming tools (as they are in the ChatGPT interface), will predict the tokens that make up short scripts and then call those scripts to achieve numeric results for things like counting lines or letters.
That's not what the paper says. It says it possible to construct weights that when run through inference will perform Turing machine -equivalent operations on prompts that are specifically made for that purpose.
That does not mean weights derived from a pile of books will do such a thing.
The thing is I imagine the LLM would be able to write a code that counted the lines and outputted what is on line 27. It seems inevitable (in a way that scares me) that a good model in the near future would know enough to write that file and execute on its own.
My understanding is early LLMs were bad at math (for similar reasons) but then got better once the model was hooked up to a calculator behind the scenes.
I also noticed that they struggle reversing strings.
Ask it to "generate a list of the 30 biggest countries together with their name in reverse". Most of the results will be correct but you'll likely find some weird spelling mistakes.
It's not something they can regurgitate from previously seen text. Models like Claude with background code execution might get around that.
How good do you think a human brain is at doing this if you simply provided the contents of the file as a string of characters (i.e. not in a text editor with line breaks rendered, etc.)?
Why are you comparing LLM's to a human brain? Software should integrate software when solving problems. It's completely reasonable to expect an LLM given a "count lines" problem to just pipe the text through wc -l.
An LLM itself can't use wc. Coding agents like Claude Code or Cursor will call out to command line tools for this kind of problem when the LLM detects it.
Well, maybe not wc directly, but they have access to sandboxed Python environments. It must be trivial for an LLM to write the Python code that calculates this.
I don't understand why Gemini insists that it can count the lines itself, instead of falling back to its Python tool [1].
Most LLMs have access to such tools. Well, maybe not a Unix shell, but something similar. This is from GPT 4.5's system prompt [1]:
python
When you send a message containing Python code to python, it
will be executed in a stateful Jupyter notebook environment.
python will respond with the output of the execution or time
out after 60.0 seconds. The drive at '/mnt/data' can be used
to save and persist user files. Internet access for this
session is disabled. Do not make external web requests or API
calls as they will fail.
They're very bad at geometry. GPT4 (first ed.) tried convincing me a line intersects a sphere 3 times. Completely clueless at comparing volumes of various polyhedra.
See, your mistake was thinking they would be useful today. You shouldn't concern yourself with that, but rather how fast they are getting slightly more useful. And with your one time investment of just $12,000,000,000 we can add that to our roadmap for future planning consideration.
Why don't we put the file characteristics into the token set that represents the file: file name, extension, size, lines, last edited, character encoding, and so on
This is true, in general LLMs are not great at counting because they don't see individual characters they see tokens.
Imagine you spoke perfect English, but you learned how to write English using Mandarin characters, basically using the closest sounding Mandarin characters to write in English. Then someone asks you how many letter o's are in the sentence "Hello how are you?". Well you don't read using English charaters, you read using Mandarin characters so you read it as "哈咯，好阿优？" because using Mandarin letters that's the closest sounding way to spell "Hello how are you?"
So now if someone asks you how many letter o's are in "哈咯，好阿优？", you don't really know... you are familiar conceptually that the letter o exists, you know that if you spelled the sentence in English it would contain the letter o, and you can maybe make an educated guess about how many letter o's there are, but you can't actually count out how many letter o's there are because you've never seen actual English letters before.
The same thing goes for an LLM, they don't see characters, they only see tokens. They are aware that characters do exist, and they can reason about their existence, but they can't see them so they can't really count them out either.
I see this claim repeated over and over, and while it seems plausible, this should be an easily testable hypothesis right? You don't even need a _large_ model for this because the hypothesis you are testing is whether transformer models [possibly with chain of thought] can count to some "reasonable" limit (maybe it can be modeled in TCS sense as something to do with circuit complexity) and you can easily train on synthetic strings. Is there any paper that shows proof/disproof that transformer networks using single-character tokenization successfully count?
Forget single character tokens, you can just go on OpenAI's own tokenizer website [1] and construct tokens and ask ChatGPT to count how many tokens there are in a given string. For example hello is a single token and if I ask ChatGPT to count how many times "hello" appears in "hellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohello" or variations thereof it gets it right.
Be careful that you structure your query so that all of the "hello" are in their own token, because you could inadvertently ask it where the first or last hello gets chunked into the text just before or just after.
[1] https://platform.openai.com/tokenizer
Neat finding, does it generalize to larger samples? Someone should randomly generate a few thousand such strings, feed it to 4o or o3, and get some accuracy results. Then compare the accuracy in cases of counting individual letters in random strings.
I find there's a lot of low-hanging fruit and claims about LLMs that are easily testable, but for which no benchmarks exist. E.g. the common claim about LLMs being "unable" to multiply isn't fully accurate, someone did a proper benchmark and found that there's a gradual decline in accuracy as digit length increases past 10 digits by 10 digits. I can't find the specific paper, but I also remember there was a way of training a model on increasingly hard problems at the "frontier" (GRPO-esque?) that fixed this issue, giving very high accuracy up to 20 digits by 20 digits.
Oh that's fair. I am not actually an LLM expert so I could have some misunderstanding about this. I remember hearing this explanation given for why previous ChatGPT models failed to answer "How many "r"s are in strawberry?", but perhaps this was an over simplification.
Right that's the explanation I've heard too (and I think Karpathy even said it so it's not some fringe theory). I wasn't dismissing the hypothesis but asking out of genuine curiosity, since this feels like something that can easily be tested on "small" large language models. There's lots of little experiments like this can be done with small-ish models trained on purely synthetic data (the stuff about digit multiplication was done on GPT-2 scale model IIRC). Can models learn to count? Can they learn to add? Can they learn to copy text verbatim accurately? Can they learn to recognize regular grammars, or even context-free grammars (this one has already been done, and the answer is yes). And if the answer to one of these turns out to be no, then we'd better find out sooner rather than later, since it means we probably need to rethink the architecture a bit.
I know there's a lot of theoretical CS work on deriving upper-bounds on these models from a circuit-complexity point of view, but as architectures are revised all the time it's hard to tell how much is still relevant. Nothing beats having a concrete, working example of a model that correctly parses CFGs as rebuttal to the claim that models just repeat their training data.
Why would you expect them to be able to given how they work? [1]
Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency.
That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket.
[1] https://www.youtube.com/watch?v=7xTGNNLPyMI
ChatGPT o4-mini got it right
https://chatgpt.com/share/683f9f73-42d8-8010-9cbc-27ad396a55...
ChatGPT 4o (the product not the LLM) got it right with a little additional prompting
https://chatgpt.com/share/683f9fd4-e61c-8010-99be-81d25264ba...
LLMs don’t reason or count. They predict and output next tokens. “Reasoning” models mostly just have another layer of validating actual output against predictions. Newer models, if provided with programming tools (as they are in the ChatGPT interface), will predict the tokens that make up short scripts and then call those scripts to achieve numeric results for things like counting lines or letters.
LLMs are turing complete ( https://arxiv.org/abs/2411.01992 ). Or what is your definition of 'count'?
That's not what the paper says. It says it possible to construct weights that when run through inference will perform Turing machine -equivalent operations on prompts that are specifically made for that purpose.
That does not mean weights derived from a pile of books will do such a thing.
Yeah, they're still bad at counting.
Tools like Claude Code work around this by feeding code into the LLMs with explicit line numbers - demo of that here: https://static.simonwillison.net/static/2025/log-2025-06-02-... - expand out some of the "tool result" panels until you see it, more notes on where I got that trace from here: https://simonwillison.net/2025/Jun/2/claude-trace/
Both GPT-4o and o4-mini got it right for me. They both wrote and executed a small Python program:
The thing is I imagine the LLM would be able to write a code that counted the lines and outputted what is on line 27. It seems inevitable (in a way that scares me) that a good model in the near future would know enough to write that file and execute on its own.
My understanding is early LLMs were bad at math (for similar reasons) but then got better once the model was hooked up to a calculator behind the scenes.
Claude 4 added Code Execution.
E.g. ask it to find the 100th prime, it will write a Python script and then run that.
I also noticed that they struggle reversing strings. Ask it to "generate a list of the 30 biggest countries together with their name in reverse". Most of the results will be correct but you'll likely find some weird spelling mistakes.
It's not something they can regurgitate from previously seen text. Models like Claude with background code execution might get around that.
https://chatgpt.com/share/683fd42a-c850-8010-9f0e-6c634f1b01...
yup, exactly what I meant, e.g
5 Brazil liziarB
Telling it to use Python with 4o
https://chatgpt.com/share/6840a944-3bac-8010-9694-2a8b0a9c35...
Even o4-mini-high got it wrong though (Indonesia)
https://chatgpt.com/share/6840a9aa-1260-8010-ba3f-bd99fff721...
How good do you think a human brain is at doing this if you simply provided the contents of the file as a string of characters (i.e. not in a text editor with line breaks rendered, etc.)?
Why are you comparing LLM's to a human brain? Software should integrate software when solving problems. It's completely reasonable to expect an LLM given a "count lines" problem to just pipe the text through wc -l.
An LLM itself can't use wc. Coding agents like Claude Code or Cursor will call out to command line tools for this kind of problem when the LLM detects it.
Well, maybe not wc directly, but they have access to sandboxed Python environments. It must be trivial for an LLM to write the Python code that calculates this.
I don't understand why Gemini insists that it can count the lines itself, instead of falling back to its Python tool [1].
[1] https://github.com/elder-plinius/CL4R1T4S/blob/main/GOOGLE/G...
which they will do I'd imagine after being told they have access to a shell
Most LLMs have access to such tools. Well, maybe not a Unix shell, but something similar. This is from GPT 4.5's system prompt [1]:
[1] https://github.com/0xeb/TheBigPromptLibrary/blob/main/System...I think computers are good at counting lines delimited by newline characters.
This prompt works fine with Qwen2.5-Coder-32B-Instruct-Q4_K_M:
They're very bad at geometry. GPT4 (first ed.) tried convincing me a line intersects a sphere 3 times. Completely clueless at comparing volumes of various polyhedra.
I tried it in gpt-4.1, o4-mini, and claude-4-sonnet, and all got the right answer.
I gave this file to ChatGPT and Grok and had similar issues there too
See, your mistake was thinking they would be useful today. You shouldn't concern yourself with that, but rather how fast they are getting slightly more useful. And with your one time investment of just $12,000,000,000 we can add that to our roadmap for future planning consideration.
Gemini Pro 2.5 says 207.
But use Flash so you can get a wrong answer sooner?
Been there too
Why don't we put the file characteristics into the token set that represents the file: file name, extension, size, lines, last edited, character encoding, and so on