Over at https://relflow.ai (an OSS model factory) used Kapso to help harness an HPO loop on RelArena. RelArena is like RelBench, but is maintained and regulated by Prior Labs to isolate data leakage that can taint modern benchmarks.
Kapso was able to efficiently calibrate Relflow's hyperparameters on 21 structured data modeling tasks, and we were able to outperform TabPFN on the majority of the larger data modeling tasks despite not using any ICL, and using ~1% of the number of params of TabPFN.
Our hypothesis is that similar to RL principle, you can experiment in the environment (predictive modelling on relational databases), learn from it, and compound your domain harness with these bank of lessons, instead of re-training foundational table model, which puts these learnings in model weights. This enables faster iteration, and transparent learning over time.
I can appreciate surpassing foundation models in RelBench, but I’ve found that peers in our space strongly prefer RelArena because it reruns submitted implementations under shared evaluation protocols. IMHO Prior Labs does a great job of providing a controlled and transparent environment.
In RelBench, hypothetically, a solution could mine for days, weeks, or months for specialized, handcrafted features. I’d would be extremely interested in how the cumulative cost of that exhaustive search scales to new, enterprise-scale databases.
As per the leaderboards you reference: "The RelBench leaderboard uses self-reported prediction files, so we rely on the authors to submit in good faith ... For a more controlled and transparent evaluation on RelBench tasks, please see RelArena".
I would love to see Kapso’s performance in a controlled environment like RelArena, to understand how this approach compares with Relflow and other relational learning methods in an apples-to-apples comparison!
Yeah, I agree with that. We have started Kapso for predictive modelling on relational databases before RelArena came out, so Relbench was the only popular public benchmark. We surely try Kapso on it too.
To test it further in live environment (additional to enterprise environment of our customers), we participated with Kapso in AI olympiad this year, and surpassed all human participants, and became top 3 AI systems: https://ioai-official.org/ai-models-track/ioai%c2%b2-2026-re.... In IOAI2, six expert-designed tasks, a single-GPU, and 2 hours budget were given to autonomous agents for deliver solutions without any human intervention.
> Woah, now that is extremely impressive.
I think Kapso could be used as an _incredible_ resource to AB test internal architectural changes to complex models with many moving parts ...
I think if a task is iterative and needs experimentation to find the optimal solution, then Kapso can be used there (hopefully helps)!
FYI, you may want to reconsider the naming as there's already a fairly popular service named Kapso [1] for integrating WhatsApp into other products (not related btw).
Yeah, originally, it came from first letters of "Knowledge-grounded framework for Autonomous Program Synthesis and Optimization", then a few weeks later, we saw this Whatsapp integration product. For the moment, our pip is "leeroo-kapso", but we're considering to change the name! Thanks for the recommendation.
Over at https://relflow.ai (an OSS model factory) used Kapso to help harness an HPO loop on RelArena. RelArena is like RelBench, but is maintained and regulated by Prior Labs to isolate data leakage that can taint modern benchmarks.
Kapso was able to efficiently calibrate Relflow's hyperparameters on 21 structured data modeling tasks, and we were able to outperform TabPFN on the majority of the larger data modeling tasks despite not using any ICL, and using ~1% of the number of params of TabPFN.
Thanks for your comment. We have also tried it ourselves in relbench (Stanford's predictive modelling on relational databases), and surpassed tabular foundational models: https://github.com/Leeroo-AI/kapso/tree/main/benchmarks/relb...
Our hypothesis is that similar to RL principle, you can experiment in the environment (predictive modelling on relational databases), learn from it, and compound your domain harness with these bank of lessons, instead of re-training foundational table model, which puts these learnings in model weights. This enables faster iteration, and transparent learning over time.
As a proof for this domain, Kapso leads in classification, regression and recommendation tasks vs. best tabular foundational model (Kumo AI/Nvidia): https://star-project.stanford.edu/relbench/leaderboard/#clas...
Thank you for sharing!
I can appreciate surpassing foundation models in RelBench, but I’ve found that peers in our space strongly prefer RelArena because it reruns submitted implementations under shared evaluation protocols. IMHO Prior Labs does a great job of providing a controlled and transparent environment.
In RelBench, hypothetically, a solution could mine for days, weeks, or months for specialized, handcrafted features. I’d would be extremely interested in how the cumulative cost of that exhaustive search scales to new, enterprise-scale databases.
As per the leaderboards you reference: "The RelBench leaderboard uses self-reported prediction files, so we rely on the authors to submit in good faith ... For a more controlled and transparent evaluation on RelBench tasks, please see RelArena".
I would love to see Kapso’s performance in a controlled environment like RelArena, to understand how this approach compares with Relflow and other relational learning methods in an apples-to-apples comparison!
Yeah, I agree with that. We have started Kapso for predictive modelling on relational databases before RelArena came out, so Relbench was the only popular public benchmark. We surely try Kapso on it too.
To test it further in live environment (additional to enterprise environment of our customers), we participated with Kapso in AI olympiad this year, and surpassed all human participants, and became top 3 AI systems: https://ioai-official.org/ai-models-track/ioai%c2%b2-2026-re.... In IOAI2, six expert-designed tasks, a single-GPU, and 2 hours budget were given to autonomous agents for deliver solutions without any human intervention.
Woah, now that is extremely impressive.
I think Kapso could be used as an _incredible_ resource to AB test internal architectural changes to complex models with many moving parts ...
> Woah, now that is extremely impressive. I think Kapso could be used as an _incredible_ resource to AB test internal architectural changes to complex models with many moving parts ...
I think if a task is iterative and needs experimentation to find the optimal solution, then Kapso can be used there (hopefully helps)!
I am very interested to hear others' perspectives on this topic:
Should we integrate experimental learnings by evolving the agent harness, or do RL-based post-training, and leave it in LLM's weights? Or Both?
FYI, you may want to reconsider the naming as there's already a fairly popular service named Kapso [1] for integrating WhatsApp into other products (not related btw).
[1] https://kapso.com/
Yeah, originally, it came from first letters of "Knowledge-grounded framework for Autonomous Program Synthesis and Optimization", then a few weeks later, we saw this Whatsapp integration product. For the moment, our pip is "leeroo-kapso", but we're considering to change the name! Thanks for the recommendation.