Not an expert, but, when you give Claude, or any AI agent a query it will usually need to do research, use various tools, execute commands etc. and all of this accumelates text in the conversation thread. Each subsequent query in a thread submits the entire conversation to the API, potentially consuming an unnecessary amount of tokens. Compression attempts to solve this by removing irrelevant information. For example, the entire output from a CURL query and how the AI thought about it might not be needed, having the resulting outcome might be enough. Since compression is also performed by the API though it may be worth it, or not, it always depends on the situation at hand. This is my understanding of it anyways.
The idea is that you create a summary with sufficient details, or you trim things away, depending on the method, but you do in fact lose exact details.
The impacts of this are not well known and likely depend on model, harness, and session. (hunch)
Not an expert, but, when you give Claude, or any AI agent a query it will usually need to do research, use various tools, execute commands etc. and all of this accumelates text in the conversation thread. Each subsequent query in a thread submits the entire conversation to the API, potentially consuming an unnecessary amount of tokens. Compression attempts to solve this by removing irrelevant information. For example, the entire output from a CURL query and how the AI thought about it might not be needed, having the resulting outcome might be enough. Since compression is also performed by the API though it may be worth it, or not, it always depends on the situation at hand. This is my understanding of it anyways.
The idea is that you create a summary with sufficient details, or you trim things away, depending on the method, but you do in fact lose exact details.
The impacts of this are not well known and likely depend on model, harness, and session. (hunch)
So it does forward pass once again to get the summary from the whole session as input?
There are a lot of things people do, but I don't think a forward pass is the right way to think about it, think more agent than llm