Why this metric. "-X% tokens removed!" is a marketing number if it ignores whether the answer was right. A tool that saves 90% of tokens but makes 20% more mistakes has worse cost-per-correct.
I found that token savings is really not worth looking at. Benchmark a baseline (without novgraph) vs novgraph on a set of tasks. Often with such tools, the agent requires more turns and cache hits are fewer so the actual cost ends up similar, but the agent actually takes more time to complete.
Build your own benchmark or try mine: https://github.com/jahala/copeca .. Haven't touched it for 3 months, so not sure if it's still up to scratch.
Although accuracy and token savings are independent measurements Novgraph achieved 85.7% accuracy across seven evaluated tool results, while measurable retrieval responses were 96.04%–99.81% saving tokens.
Cost per correct answer.
cost_per_correct = total_spend / correct_count
Why this metric. "-X% tokens removed!" is a marketing number if it ignores whether the answer was right. A tool that saves 90% of tokens but makes 20% more mistakes has worse cost-per-correct.
I found that token savings is really not worth looking at. Benchmark a baseline (without novgraph) vs novgraph on a set of tasks. Often with such tools, the agent requires more turns and cache hits are fewer so the actual cost ends up similar, but the agent actually takes more time to complete.
Build your own benchmark or try mine: https://github.com/jahala/copeca .. Haven't touched it for 3 months, so not sure if it's still up to scratch.
Although accuracy and token savings are independent measurements Novgraph achieved 85.7% accuracy across seven evaluated tool results, while measurable retrieval responses were 96.04%–99.81% saving tokens.