Some of the benchmarks, like SREGym, are gaining popularity for evaluating the effectiveness of various agentic investigation tools. Vendor-promoted tools take most of the top spots, ahead of DIY systems based on generic frontier models. Vendors can improve their offerings through better models and better data quality. With Chiron providing the context data, we see a huge jump in model accuracy (46.2% compared with the paper’s 30.9% baseline) and about 9× fewer tokens on ORCA-bench. This suggests that DIY projects with the right context are a viable option.
Some of the benchmarks, like SREGym, are gaining popularity for evaluating the effectiveness of various agentic investigation tools. Vendor-promoted tools take most of the top spots, ahead of DIY systems based on generic frontier models. Vendors can improve their offerings through better models and better data quality. With Chiron providing the context data, we see a huge jump in model accuracy (46.2% compared with the paper’s 30.9% baseline) and about 9× fewer tokens on ORCA-bench. This suggests that DIY projects with the right context are a viable option.