How I cut my agent’s inference bill by 85% with two cache breakpoints.
Every part of the prompt has a stability horizon, how long it stays true. Sort the prompt by it, and give each part a cache that lives about as long: cache each part of the prompt for no longer than it stays true.
How I cut my agent’s inference bill by 85% with two cache breakpoints.
Every part of the prompt has a stability horizon, how long it stays true. Sort the prompt by it, and give each part a cache that lives about as long: cache each part of the prompt for no longer than it stays true.