Hi HN. I built a benchmark for myself that measures not just coding an app, but rather the end-to-end loop: Build, deploy, troubleshoot. Then an independent verifier (currently opus 5.5) checks the live URL and performs basic tests.
The benchmark runs five web apps: (1) static website for say a company page or the like (2) the usual CRUD with auth for say SaaS product (3) an API to be consumed by other services (4) a dashboard with data to mimic internal company dashboards and the like (5) a realtime chat application with support for web sockets.
It currently compares four platforms but can be easily extended to more, feel free to open GitHub issues or send me a PR.
I was specially interested in comparing human intervention (Human Time). As in, how much time is spent for a human to deploy to particular platforms the first time?
The benchmark tracks build/deploy time, tokens, agent cost, security, compliance review of the deployment (secrets handling, least privilege, terms fit, attack surface, etc.). Methodology and reports (including screenshots) are in the repo.
Hi HN. I built a benchmark for myself that measures not just coding an app, but rather the end-to-end loop: Build, deploy, troubleshoot. Then an independent verifier (currently opus 5.5) checks the live URL and performs basic tests.
The benchmark runs five web apps: (1) static website for say a company page or the like (2) the usual CRUD with auth for say SaaS product (3) an API to be consumed by other services (4) a dashboard with data to mimic internal company dashboards and the like (5) a realtime chat application with support for web sockets.
It currently compares four platforms but can be easily extended to more, feel free to open GitHub issues or send me a PR.
I was specially interested in comparing human intervention (Human Time). As in, how much time is spent for a human to deploy to particular platforms the first time?
The benchmark tracks build/deploy time, tokens, agent cost, security, compliance review of the deployment (secrets handling, least privilege, terms fit, attack surface, etc.). Methodology and reports (including screenshots) are in the repo.
Repo: https://github.com/cloudishai/ai-deploy-bench
Full disclosure: I'm working on Cloudish. Happy to answer questions about the methodology or results.
Edit: Removed table due to formatting issues.
Just added a website for the same benchmark, easier read: https://aideploybench.com