Devin was the first ai developer harness I ever heard of — and in fact the first one I ever used as I had gotten an invite while it was in private beta. Now it seems like the product has faded (into irrelevance?) between other harnesses like cursor especially and all the major harnesses having a cloud mode. Is there some other area Devin has become really popular / has stayed competitive?
I was just thinking recently, isn't this fascination with harnesses kind of backwards?
I mean ChatGPT is the harness most of the time that's calling the tools, MCPs, etc. Putting a loop around it doesn't really seem like it would be something that gets you a lot of extra milage.
It gets an incredible amount of mileage. And I believe it is the interface that allows the model to learn and eventually bake that intelligence into the model.
Look at how Astra scored 100% on Arc-AGI-3. It was largely because of the harness.
The harness increases the chances (often to 100% chance) of a non-deterministic LLM to perform deterministic actions.
Not only that, the harness provides the feedback that becomes training data for the model. So, over time the model bakes those lessons in, and the harness becomes less necessary and the agent becomes more efficient at some tasks.
A harness will likely always be necessary, we may hit some level of complexity or some level of compute that never allows us to bake the lessons into the model, and external tools provide the model with the ability to find leverage and make up for those short comings.
I did not imply harnesses were useless, just that they're merely a functional part of running an LLM. But it's ticking a checkbox. If the harness can support LLM capabilities (tool calling, reading the screen, visual interaction), it's reached the limit of its potential. Like a JSON parser - can it do the job or not?
Astra didn't solve Arc-AGI-3 because of its phenomenal harness, just a harness was needed to support its capabilities.
Giving Astra's harness to a different LLM would not have improved said LLMs ability to solve the task in a meaningful way.
I feel like this is important to point out, seeing how I'm commenting under an article about how a harness is valued at $50B.
Imagine valuing a JSON parser at $50B on the basis that most communication can't be read without one.
A harness is fundamentally required - although we don’t really see it using tools like Claude or codex ai models still only can take tokens in and produce tokens out. Tool calls are literally just the ai model printing “I want to call search_web with parameters abc” and the harness sees that in the text stream and runs it
Seems insane. It isn’t that had to use anthropic managed agents to build something similar - work with an in house version that I think is superior. I’m sure Devin is a good fit for some companies…but struggling to see where Devin adds $2 billion value.
Edit 48billion I just noticed. Someone’s getting fleeced.
They have a first mover advantage on autonomous AI employee space. So big enterprises will all default to them given all their competitors will be "startups" uncertain to be around long term.
I am not sure what is devin's value proposition, but one can build lots of enterprise stuff into coding harness: integration with various internal doc, auth, db services, some enterprise jiras, meeting software, security/privacy certifications, limits/budget management, etc/etc/etc
Devin was the first ai developer harness I ever heard of — and in fact the first one I ever used as I had gotten an invite while it was in private beta. Now it seems like the product has faded (into irrelevance?) between other harnesses like cursor especially and all the major harnesses having a cloud mode. Is there some other area Devin has become really popular / has stayed competitive?
I was just thinking recently, isn't this fascination with harnesses kind of backwards?
I mean ChatGPT is the harness most of the time that's calling the tools, MCPs, etc. Putting a loop around it doesn't really seem like it would be something that gets you a lot of extra milage.
It gets an incredible amount of mileage. And I believe it is the interface that allows the model to learn and eventually bake that intelligence into the model.
Look at how Astra scored 100% on Arc-AGI-3. It was largely because of the harness.
The harness increases the chances (often to 100% chance) of a non-deterministic LLM to perform deterministic actions.
Not only that, the harness provides the feedback that becomes training data for the model. So, over time the model bakes those lessons in, and the harness becomes less necessary and the agent becomes more efficient at some tasks.
A harness will likely always be necessary, we may hit some level of complexity or some level of compute that never allows us to bake the lessons into the model, and external tools provide the model with the ability to find leverage and make up for those short comings.
I did not imply harnesses were useless, just that they're merely a functional part of running an LLM. But it's ticking a checkbox. If the harness can support LLM capabilities (tool calling, reading the screen, visual interaction), it's reached the limit of its potential. Like a JSON parser - can it do the job or not?
Astra didn't solve Arc-AGI-3 because of its phenomenal harness, just a harness was needed to support its capabilities.
Giving Astra's harness to a different LLM would not have improved said LLMs ability to solve the task in a meaningful way.
I feel like this is important to point out, seeing how I'm commenting under an article about how a harness is valued at $50B.
Imagine valuing a JSON parser at $50B on the basis that most communication can't be read without one.
A harness is fundamentally required - although we don’t really see it using tools like Claude or codex ai models still only can take tokens in and produce tokens out. Tool calls are literally just the ai model printing “I want to call search_web with parameters abc” and the harness sees that in the text stream and runs it
Seems insane. It isn’t that had to use anthropic managed agents to build something similar - work with an in house version that I think is superior. I’m sure Devin is a good fit for some companies…but struggling to see where Devin adds $2 billion value.
Edit 48billion I just noticed. Someone’s getting fleeced.
They have a first mover advantage on autonomous AI employee space. So big enterprises will all default to them given all their competitors will be "startups" uncertain to be around long term.
Big enterprises have heard of ChatGPT and Claude, not Devin.
I am not sure what is devin's value proposition, but one can build lots of enterprise stuff into coding harness: integration with various internal doc, auth, db services, some enterprise jiras, meeting software, security/privacy certifications, limits/budget management, etc/etc/etc
I used to be a hater, but it’s kinda good now
Lol who even uses this
They claim 900M revenue