Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too.
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`.
We currently focus on serving DeepSeek models on dedicated clusters for large customers. We do not offer this as a publicly available service at the moment, but we may consider doing so in the future.
In part this is because VLLM is a confusing mess. It takes a large investment in time to get it to perform well (and when it does it is really good). But it is fragile and over complicated.
> The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
Just as a point of feedback for the authors, I enjoyed the article, and did not suffer the confusion pointed out above.
Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run.
In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.
Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.
This is the approach I took. Before leaving scope, results are filtered to remove tool call errors and bloat and instead append those errors to a different system. A much smaller tool call error result is returned to the agent while keeping the reasoning for failure intact.
Context is just one long string. You can do whatever you want with it. Training helps to conform to specific syntactic idiosyncrasies, but LLMs are large enough these days that they can follow pretty much any instructions most of the time.
For example, if you want to use an LLM API that does not natively support tool calls, you can tell the LLM to emit the tool calls for your preferred tools in a markdown code block and parse them yourself.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).
So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).
The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).
> Could something like that also be used in other contexts?
Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").
Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?
A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...
So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.
So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"
Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.
Just trying to be helpful. I couldn’t see any other way to interpret the question below except as someone who doesn’t know how tool calls get done and who might therefore value a little insight. Didn’t mean to be patronising, apologies if I misinterpreted your question.
> So if I understand it correctly, the tool calls are not actually done by the model?
Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
(this is just made up tokenization but it's close enough to the real world)
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.
This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.
I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?
It can happen that you would be able to inject them if there is a bug, same way sql injections work.
You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
The syntax was specifically chosen to avoid confusion with other tag formats. XML could show up in any user- or LLM-generated text and would mess up the message format. Some variations even used unicode symbols like "｜" or "〈" to decrease the chance of confusion.
Also, back then, some people believed that it could prevent prompt injection, but history has shown that it was a futile attempt.
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.
I wish they'd published something about the actual impact on model quality. Schema-constrained decoding changes the distribution the LLM will output. I'd be curious how a back-up-and-retry method for tool calls (which also doesn't pollute the context and also has 100% tool-call accuracy) would compare. In their example stats (23/25 success rate), it'd be +9% tokens used for the tool call attempt itself, but that's +9% on just a small fraction of the total tokens, so if a retry solution gave any better accuracy I think it might win out.
We are also interested in this and may focus on it in a future post. They are two different perspectives: (1) What is the overall quality of LLM+serving stack+harness, and (2) How close does the behavior of our setup match the first-party provider's? In this post, we focused on the latter. Moonshot AI, the creators of Kimi, recommend constrained decoding to be on by default, and also serve Kimi that way. This implies that it helps overall performance, though it does not provide numbers for it. We want to serve Kimi as recommended, and ran into the challenges described in the post.
There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3]. Overall, it seems to me that there is a trend towards using constrained decoding. OpenAI's newer Responses API has constrained decoding turned on by default as well, similar to Moonshot AI's Kimi.
Is the schema constraint not implemented in a way where the new distribution is identically scaled to the old distribution with retry? I had thought the way it worked was just to disallow invalid next states... oh, or is the idea that you think it is much preferred to retry at the level of the entire tool call, rather than (conceptually) per token? I guess I can see that making a lot of sense, as maybe the model gets "backed into a corner", so to speak, with a really bad initial choice and is then trying to rescue horrible states?
The way it works is to disallow invalid next states, and getting backed into a corner is one way to think about one possible failure mode.
For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like
{"name": "Bob", "biography": "Bob is a fantastic builder, having ...
Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.
As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.
For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.
What the other guy means is the model is forced to decode potentially out of distribution and subsequent parts of the sequence will be sampled from a higher-entropy distribution.
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
> or perhaps adjusting the order of the json output affects it to produce lower-quality results.
The order of the output fields can actually be used to tune performance of the outputs. It can be helpful to force the LLM to reason about specific fields first if later fields benefit from seeing those earlier fields' output. You have to test it because some times the results are surprising in directions you didn't expect.
XGrammar has an option somewhere to allow fields to appear in any order but I don't know if vLLM exposes it. There are some tradeoffs and limitations for allowing fields to show up in any order because there is some complexity in schema parsing like that.
Technically true, but this is a very inefficient optimization strategy and usually indicates some structural issue with your setup.
When you know enough about the JSON structure to understand how to set a decoding order, then you can simply break the problem into smaller parts so it's more manageable.
Otherwise, you have these solutions:
- GEPA-style prompt optimization for simple agents
- If working with large models, you can just have them do step-by-step problem solving & relying on their CoT for accuracy.
- A small model can simply be trained to do what you want
And if those don't work, I would be extremely surprised.
I guess to be more concrete, I thought the way schema aware decoding worked should be identical to a token-level retry: "only even dice rolls are allowed" converting {0:0.1, 1:0.2, 2:0.2, 3:0.1, 4:0.2, 5:0.2} into {0:0.2, 2:0.4, 4:0.4}. That is why I am asking if the difference being stared at is doing the retry at the level of the entire tool call rather than each token (as this let's later errors in the schema re-roll the earlier committed states)? (But like, it isn't clear to me that's what they mean, so I feel a need to ask; as an example, you are now talking about field order, but that's an entirely different issue involving an overly-constrained constraint: nothing would prevent implementing the grammar for the constrained decoding to allow the fields in any order, and if you allow duplicate fields -- which JSON typically allows anyway -- then it is probably already supported as the language loses its additional context requirement.)
Right... and I said exactly that in both my comments, but I was looking for confirmation that that's the issue being stared at, as it wasn't obvious to me that's what was meant and (I will cite as demonstration of the confusion) now people are bringing up other ideas (such as field order, which is an entirely unrelated constraint). (FWIW, since you are the person I was originally asking, I will take this as positive confirmation that "yes: I am saying it is better to retry the entire tool call" ;P.)
Ah, I see. IMO the other response [0] I gave is probably the closest to what I had in mind. Interestingly, retry very well might not be better, especially when measured on a pareto frontier including cost.
I have a reasonably strong prior that it should be better though. That comes from my observations that whenever I notice the difference in distribution the retry method is better (obvious sampling bias, but what kind of person ignores their own experiences). As a stronger piece of evidence and intuition, the entire goal of these systems is to reproduce a certain distribution. If that distribution says that you should use ripgrep 10x as often as you should use grep then that's a signal you probably don't want to throw away. Constrained decoding might very well do so and place them on even footing or skew 10x the other direction. Retry faithfully captures that original distribution. I'm skeptical of solutions which want to shy too far away from it (similarly with most applications of changing the model's temperature).
The more important point is just that they're different, so we should probably measure the impact.
(Not sure you'll see this, ofc.) After percolating on this for a couple days, is retrying the tool calls enough? Maybe the reason the tool call is requiring three retries is because the AI simply got locked into too difficult of a way to solve the problem, and if it retried the entire response it would have gone in a different direction entirely. How far back do we feel comfortable?
for putting together something like a tool-call decision model, the conditioning will take less time than getting the data for it, which also will not take much time if you keep your session data. whatever fails is your negative contrast, whatever passes is your positive data. the space of tool calls your agents will use in your particular environment is not actually that big, which makes this one of the easiest wins you can pull off.
That is bending the barrel to align the scope. What’s interesting is that the authors mention the cause of the failed calls, “…a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location”, but then failed to see that it’s also the solution. If you call it “web_search” but it calls it as “search_web” then just rename your function to “search_web” and let the problem resolve itself.
Things get more interesting if you don’t give it a call syntax and just see what it does based on the name alone.
> For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.
100% success is critical for tool calling.
The probability that the overall agent iteration is successful would scale very poorly even if we achieved 95%. With p95, if you need 50 tool calls that must execute sequentially to achieve a result, the chances that you won't have an error are only ~8%. There are ways to recover from bad tool calls (error feedback helps a lot), but if the agent is persistently using the wrong tools with the wrong args the overall system will not work very well.
I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP.
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
Yes. All you need is a formal grammar that defines the syntax of the text being generated. The llama.cpp repo has a decent writeup on the way that grammar-constrained decoding works in general [0], and it includes a bit of a simplistic example for generating syntactically-correct C code [1]. No reason this couldn't be extended to other things -- I personally have run experiments with generating syntactically-correct SQL statements when doing text-to-sql [2], and I think it's pretty cool.
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.
I believe there is constrained decoding available for Qwen 3.8 in vLLM. But mileage may vary: although you guarantee syntactically-correct tool calls, you'll want to observe the overall quality of the model with and without constrained decoding. In the case of Kimi, there is a clear recommendation from the creators to turn constrained decoding on. I don't think this is the case for Qwen 3.8.
llama.cpp has done this since forever. I've used other engines including vLLM here and there, and I was quite surprised that many of them don't have this feature. Glad vLLM has it now.
Okay, this explains the feeling that I had that vLLMs tool parsing is much more precise than for example llama cpp or Ollama, I still use ollama due to mlx which of my understanding is not offered on vLLM unfortunately.
Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too.
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`.
If you work for ubicloud, I have a question: do you sell api access to kimi/etc, and if so, where is that listed??
We currently focus on serving DeepSeek models on dedicated clusters for large customers. We do not offer this as a publicly available service at the moment, but we may consider doing so in the future.
In part this is because VLLM is a confusing mess. It takes a large investment in time to get it to perform well (and when it does it is really good). But it is fragile and over complicated.
> The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
Just as a point of feedback for the authors, I enjoyed the article, and did not suffer the confusion pointed out above.
Should have just wrote "Fixing Tool-Call Accuracy with vLLM" instead of "Fixing Tool-Call Accuracy in vLLM"
Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
> avoid a giant prefix miss
I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.
Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.
This is the approach I took. Before leaving scope, results are filtered to remove tool call errors and bloat and instead append those errors to a different system. A much smaller tool call error result is returned to the agent while keeping the reasoning for failure intact.
>some cases, tools are added mid-run to avoid a giant prefix miss
Wait, can you actually do that? The model should not exactly care where in the context the tools are, but I haven't realised it's a thing that works
Context is just one long string. You can do whatever you want with it. Training helps to conform to specific syntactic idiosyncrasies, but LLMs are large enough these days that they can follow pretty much any instructions most of the time.
For example, if you want to use an LLM API that does not natively support tool calls, you can tell the LLM to emit the tool calls for your preferred tools in a markdown code block and parse them yourself.
What would a "fully-constrained tool call environment" be?
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).
So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).
The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).
> Could something like that also be used in other contexts?
Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").
The point I was getting at was more of a question of "Okay, how can this be used adversarially?" or "How can we break this?".
"What can a malicious inference engine do"
Something like that
Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?
A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...
That was exactly my question.
So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.
So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"
You should treat inference provider as a C2 node of a rootkit. As long as it's your c2 or it's aligned with you somewhat, you are fine
Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.
[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...
I think I might've phrased something in ways that were confusing to people, because now two people have explained tool calls to me for some reason?
Just trying to be helpful. I couldn’t see any other way to interpret the question below except as someone who doesn’t know how tool calls get done and who might therefore value a little insight. Didn’t mean to be patronising, apologies if I misinterpreted your question.
> So if I understand it correctly, the tool calls are not actually done by the model?
Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.
Yeah I know what a tool call is.
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
So to be clear, you can't prompt-inject these?
... How did they get into the training data?
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
But I mean, how do you train the model to understand such a token, when it can't be found by scraping a web page?
(this is just made up tokenization but it's close enough to the real world)
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.
The same way it understands what all the other tokens mean -- we throw them at the wall and beat it with a stick until it behaves.
> when it can't be found by scraping a web page?
This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.
They are specified in the token map as special tokens. Presumably they get trained to use them when they get RL'ed to use tools?
I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?
Synthetic data, and then RL.
Why can’t user request be spicy numbers so prompt injection is impossible
It can happen that you would be able to inject them if there is a bug, same way sql injections work.
You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
The syntax was specifically chosen to avoid confusion with other tag formats. XML could show up in any user- or LLM-generated text and would mess up the message format. Some variations even used unicode symbols like "｜" or "〈" to decrease the chance of confusion.
Also, back then, some people believed that it could prevent prompt injection, but history has shown that it was a futile attempt.
Correct me if I wrong, I saw this when llama.cpp or ollama initializes, it is used as token separator of some sort.
GPT-2 used <|endoftext|>
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.
[flagged]
I wish they'd published something about the actual impact on model quality. Schema-constrained decoding changes the distribution the LLM will output. I'd be curious how a back-up-and-retry method for tool calls (which also doesn't pollute the context and also has 100% tool-call accuracy) would compare. In their example stats (23/25 success rate), it'd be +9% tokens used for the tool call attempt itself, but that's +9% on just a small fraction of the total tokens, so if a retry solution gave any better accuracy I think it might win out.
We are also interested in this and may focus on it in a future post. They are two different perspectives: (1) What is the overall quality of LLM+serving stack+harness, and (2) How close does the behavior of our setup match the first-party provider's? In this post, we focused on the latter. Moonshot AI, the creators of Kimi, recommend constrained decoding to be on by default, and also serve Kimi that way. This implies that it helps overall performance, though it does not provide numbers for it. We want to serve Kimi as recommended, and ran into the challenges described in the post.
There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3]. Overall, it seems to me that there is a trend towards using constrained decoding. OpenAI's newer Responses API has constrained decoding turned on by default as well, similar to Moonshot AI's Kimi.
1: https://arxiv.org/abs/2408.02442
2: https://blog.dottxt.ai/say-what-you-mean.html
3: https://dylancastillo.co/posts/say-what-you-mean-sometimes.h...
Is the schema constraint not implemented in a way where the new distribution is identically scaled to the old distribution with retry? I had thought the way it worked was just to disallow invalid next states... oh, or is the idea that you think it is much preferred to retry at the level of the entire tool call, rather than (conceptually) per token? I guess I can see that making a lot of sense, as maybe the model gets "backed into a corner", so to speak, with a really bad initial choice and is then trying to rescue horrible states?
The way it works is to disallow invalid next states, and getting backed into a corner is one way to think about one possible failure mode.
For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like
{"name": "Bob", "biography": "Bob is a fantastic builder, having ...
Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.
As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.
For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.
What the other guy means is the model is forced to decode potentially out of distribution and subsequent parts of the sequence will be sampled from a higher-entropy distribution.
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
> or perhaps adjusting the order of the json output affects it to produce lower-quality results.
The order of the output fields can actually be used to tune performance of the outputs. It can be helpful to force the LLM to reason about specific fields first if later fields benefit from seeing those earlier fields' output. You have to test it because some times the results are surprising in directions you didn't expect.
XGrammar has an option somewhere to allow fields to appear in any order but I don't know if vLLM exposes it. There are some tradeoffs and limitations for allowing fields to show up in any order because there is some complexity in schema parsing like that.
Technically true, but this is a very inefficient optimization strategy and usually indicates some structural issue with your setup.
When you know enough about the JSON structure to understand how to set a decoding order, then you can simply break the problem into smaller parts so it's more manageable.
Otherwise, you have these solutions: - GEPA-style prompt optimization for simple agents - If working with large models, you can just have them do step-by-step problem solving & relying on their CoT for accuracy. - A small model can simply be trained to do what you want
And if those don't work, I would be extremely surprised.
I guess to be more concrete, I thought the way schema aware decoding worked should be identical to a token-level retry: "only even dice rolls are allowed" converting {0:0.1, 1:0.2, 2:0.2, 3:0.1, 4:0.2, 5:0.2} into {0:0.2, 2:0.4, 4:0.4}. That is why I am asking if the difference being stared at is doing the retry at the level of the entire tool call rather than each token (as this let's later errors in the schema re-roll the earlier committed states)? (But like, it isn't clear to me that's what they mean, so I feel a need to ask; as an example, you are now talking about field order, but that's an entirely different issue involving an overly-constrained constraint: nothing would prevent implementing the grammar for the constrained decoding to allow the fields in any order, and if you allow duplicate fields -- which JSON typically allows anyway -- then it is probably already supported as the language loses its additional context requirement.)
Schema-constrained is equivalent to token-level retry. It's not equivalent to retry of longer sequences.
Right... and I said exactly that in both my comments, but I was looking for confirmation that that's the issue being stared at, as it wasn't obvious to me that's what was meant and (I will cite as demonstration of the confusion) now people are bringing up other ideas (such as field order, which is an entirely unrelated constraint). (FWIW, since you are the person I was originally asking, I will take this as positive confirmation that "yes: I am saying it is better to retry the entire tool call" ;P.)
Ah, I see. IMO the other response [0] I gave is probably the closest to what I had in mind. Interestingly, retry very well might not be better, especially when measured on a pareto frontier including cost.
I have a reasonably strong prior that it should be better though. That comes from my observations that whenever I notice the difference in distribution the retry method is better (obvious sampling bias, but what kind of person ignores their own experiences). As a stronger piece of evidence and intuition, the entire goal of these systems is to reproduce a certain distribution. If that distribution says that you should use ripgrep 10x as often as you should use grep then that's a signal you probably don't want to throw away. Constrained decoding might very well do so and place them on even footing or skew 10x the other direction. Retry faithfully captures that original distribution. I'm skeptical of solutions which want to shy too far away from it (similarly with most applications of changing the model's temperature).
The more important point is just that they're different, so we should probably measure the impact.
[0] https://news.ycombinator.com/item?id=50245714
(Not sure you'll see this, ofc.) After percolating on this for a couple days, is retrying the tool calls enough? Maybe the reason the tool call is requiring three retries is because the AI simply got locked into too difficult of a way to solve the problem, and if it retried the entire response it would have gone in a different direction entirely. How far back do we feel comfortable?
These edge-cases are why the simplest answer works best: just scale your model to be good enough that it can work through the problem on its own
for putting together something like a tool-call decision model, the conditioning will take less time than getting the data for it, which also will not take much time if you keep your session data. whatever fails is your negative contrast, whatever passes is your positive data. the space of tool calls your agents will use in your particular environment is not actually that big, which makes this one of the easiest wins you can pull off.
[flagged]
That is bending the barrel to align the scope. What’s interesting is that the authors mention the cause of the failed calls, “…a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location”, but then failed to see that it’s also the solution. If you call it “web_search” but it calls it as “search_web” then just rename your function to “search_web” and let the problem resolve itself.
Things get more interesting if you don’t give it a call syntax and just see what it does based on the name alone.
> For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.
100% success is critical for tool calling.
The probability that the overall agent iteration is successful would scale very poorly even if we achieved 95%. With p95, if you need 50 tool calls that must execute sequentially to achieve a result, the chances that you won't have an error are only ~8%. There are ways to recover from bad tool calls (error feedback helps a lot), but if the agent is persistently using the wrong tools with the wrong args the overall system will not work very well.
Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling.
https://github.com/huggingface/transformers.js/blob/main/pac...
I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP.
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
> I do things that require ~300 tool calls per prompt
I have multiples of that with some regularity, what is the hangup?
Can the same mechanism be used to make sure it outputs syntactically correct code?
Yes. All you need is a formal grammar that defines the syntax of the text being generated. The llama.cpp repo has a decent writeup on the way that grammar-constrained decoding works in general [0], and it includes a bit of a simplistic example for generating syntactically-correct C code [1]. No reason this couldn't be extended to other things -- I personally have run experiments with generating syntactically-correct SQL statements when doing text-to-sql [2], and I think it's pretty cool.
[0] - https://github.com/ggml-org/llama.cpp/tree/master/grammars [1] - https://github.com/ggml-org/llama.cpp/blob/master/grammars/c... [2] - https://gist.github.com/HanClinto/b853571136fd42f8e2bde4c99c...
Yes, absolutely. The problem is integrating it into the inference engine. Not impossible, just _very hard_.
Also less useful than one might think at first.
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.
If the model is predicting syntactically incorrect tokens in code you probably have bigger issues
Is this even a real issue? Are you consistently getting code out of frontier models + harnesses(Codex, Claude Code, etc) that have syntax errors?
Does this work with Qwen 3.8?
I believe there is constrained decoding available for Qwen 3.8 in vLLM. But mileage may vary: although you guarantee syntactically-correct tool calls, you'll want to observe the overall quality of the model with and without constrained decoding. In the case of Kimi, there is a clear recommendation from the creators to turn constrained decoding on. I don't think this is the case for Qwen 3.8.
llama.cpp has done this since forever. I've used other engines including vLLM here and there, and I was quite surprised that many of them don't have this feature. Glad vLLM has it now.
Okay, this explains the feeling that I had that vLLMs tool parsing is much more precise than for example llama cpp or Ollama, I still use ollama due to mlx which of my understanding is not offered on vLLM unfortunately.
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
this was done by my team, ama