Compaction isn't just summarization. It's a security-sensitive transformation of conversation state.
If you flatten typed messages into a string and then ask an LLM to reconstruct who said what, you've thrown away the information needed to enforce that trust boundary. The model isn't necessarily failing to follow instructions. The harness has made it impossible to reliably distinguish a real user instruction from one embedded in attacker-controlled content.
This feels like a much more fundamental problem than prompt injection alone.
Compaction isn't just summarization. It's a security-sensitive transformation of conversation state.
If you flatten typed messages into a string and then ask an LLM to reconstruct who said what, you've thrown away the information needed to enforce that trust boundary. The model isn't necessarily failing to follow instructions. The harness has made it impossible to reliably distinguish a real user instruction from one embedded in attacker-controlled content.
This feels like a much more fundamental problem than prompt injection alone.
Great find! Using a fake user turn to blind an agent's code review so it ignores a backdoored file is such a sneaky vector.
[flagged]