I'm asking this because as I wrote a new CLI tool I picked json as the recommended output format for LLMs. I noticed though that they easily get lost and generally consume a lot of tokens to analyse the output. I'm sure it would work well when the LLM is trained for and already know what to expect, but that doesn't work for a new tool.
In my experience, LLMs are really good with SQL. If you can structure your data into easily queryable SQL data, your LLM will be able to generate queries to get what it needs really effectively
That's an interesting take! Not quite easy to do with a CLI tool with dozens of commands though. Maybe it would be some kind of always on server the LLM can operate on ... but then that starts to really look like graphql.
Could tokens be pre-embedded and saved? Suppose you're transferring a context to another agent. Currently I just copy/paste, tell the other agent to reference a .md file.
What is I could get a "compressed" binary file containing the context in the embedding space. Then the other agent can consume that and continue on it's merry way.
I settled on JSONL for an LLM-optimized output format. Rationale: https://github.com/Leftium/gg/blob/main/specs/completed/gg-a...
Combine that with context mode: https://hw.leftium.com/i/47193064
provide a summary,Provide metadata again;Expand the complete content when needed
I'm asking this because as I wrote a new CLI tool I picked json as the recommended output format for LLMs. I noticed though that they easily get lost and generally consume a lot of tokens to analyse the output. I'm sure it would work well when the LLM is trained for and already know what to expect, but that doesn't work for a new tool.
So, what's the natural language for them?
In my experience, LLMs are really good with SQL. If you can structure your data into easily queryable SQL data, your LLM will be able to generate queries to get what it needs really effectively
That's an interesting take! Not quite easy to do with a CLI tool with dozens of commands though. Maybe it would be some kind of always on server the LLM can operate on ... but then that starts to really look like graphql.
Standard disclosure of "I'm no LLM expert".
Could tokens be pre-embedded and saved? Suppose you're transferring a context to another agent. Currently I just copy/paste, tell the other agent to reference a .md file.
What is I could get a "compressed" binary file containing the context in the embedding space. Then the other agent can consume that and continue on it's merry way.