(OP) I completely agree! AI is amazing at writing web apps that are full of bugs, but it also gives us what we need to turn that completely around. That's what we're trying to make happen...
>> It is possible now to reliably and automatically detect and fix every bug
which users are encountering on the web.
proceeds to give a trivial and then vague example related to usability, the first probably created by AI and the second a notoriously difficult problem to assess and solve for humans with context, let alone AIs...
The code I have dealt with from AI tends to be rather convoluted for the problems at hand and then become even more difficult to maintain. It has a place but I have been seeing it make code more difficult to understand, not less.
Convoluted code is definitely a problem, I don't think it's unique to AI though. It also becomes more manageable with a time travel debugger as the actual behavior is easy to understand.
Yep. I see it myself, both in other people's code, and our own. The _default_ behavior is a tendency to pile slop on slop, make poor architectural choices, slap bandaids on edge cases without considering the full ramifications, etc. This is more true if there isn't oversight or feedback, and also depending on model quality.
On the flip side: I personally use a strongly human-in-the-loop AI dev workflow [0]. Going back and forth on how much of that code I review myself (and I've found that AI code review frankly does a better job than _I_ do most of the time :) ). But, between staying hands-on, and the ongoing improvements in models, I've seen that the code my own agentic dev sessions produces is actually pretty high quality, and I've been able to clean up a lot of low-quality slop from unmonitored agent sessions. I've also been able to build some pretty major features from scratch and even execute massive migrations with excellent results, again by having me drive the work.
I actually just got done shipping Redux Toolkit 2.14 [1] with a massive set of bug fixes, and almost all of those were found by my agent researching the already-mature RTK codebase, at my direction, and identifying existing logic flaws or weaknesses. Once we knew what those bugs were, we could fix them.
Which, honestly, is the point of the blog post, of Replay QA as an automated testing product, and of Replay as a company in the first place.
Once you know what a bug _is_ and why it happens, fixing it is usually fairly straightforward. _Finding_ it is the hard part.
I've given a conf talk the last few years on "Debugging Software" [2]. One of the main points I make is that no one is _taught_ how to debug - we figure it out on the job as we go, because we have to ship. It's part of the job. But few people like doing it, and most people just want to slap in some console logs and call it a day. Very few people even use existing graphical debuggers (VS Code, Chrome DevTools, etc).
We built a time-travel debugger. The original idea was: make a DVR-style recording of a bug happening once, then you can dive in as deep as you need to investigate. Originally, that was supposed to be a human using the Replay DevTools UI - browser devtools with time travel debugging features built in. It's awesome, I wish I'd had it much earlier in my career :) But we found most people didn't want to spend time doing it themselves.
Earlier this year we shipped Replay MCP, which gives agents the same time-travel debugging capabilities: see hit counts for any line of code, dynamic logpoints and evaluations for every time a line of code ran, see the DOM tree or React component tree at any point in time. I built most of the React analysis features and MCP tools myself, based on my years of experience in the React ecosystem and maintaining Redux.
So, today you _could_ make a Replay recording of your app, point your agent at the MCP, and have it investigate. I've done that, and wrote some blog posts about how that helped my own agent find bugs in React itself and Next.js apps [3] [4]. I've seen my agent nail a bug from a recording on someone else's machine in <10min, starting from scratch.
We've productized that as Replay QA, so that you don't have to do any of the recording or setup - just point Replay QA at your app and repo and we do all the site analysis and recording and bug root cause investigation ourselves. Essentially, encoding our own expertise as debuggers, so you get the benefits [5]
So, back to the original point. Yeah, AI tends to crank out bad code by default, that code is buggy. We can't help your agent choose a better architecture. But we can help identify the bugs proactively, identify the root causes, and suggest solutions, based on the actual runtime behavior of the app. Up to you to get your agents to write _better_ code, but we can give your agent what it needs to fix the bugs it wrote :)
Is this just a promotion post for their "Replay QA™" feature?
It boils down to "other products find the issue and leave the root cause + fix to your agent, our product finds the issue + root cause and leaves the fix to your agent".
Who says I want your agent to root cause? Other products already give me the repro steps, video, trace, network logs, etc, and my agent is in a much better position in terms of context and setup to find the root cause and proper fix.
If you're happy with your agent's performance using surface level o11y to track down issues then that's fine. This is what pretty much the entire industry is doing and it does usually work! QA focuses on root causes because it can look through the app's entire execution and provide evidence so your agent doesn't need to guess when figuring out what the fix is. That guesswork is what we're trying to eliminate, for when the surface level stuff doesn't clearly point at the problem.
Here is my crazy list of ideas for less buggy web apps:
1) care
2) understanding why a change is made/needed before making it
3) deleting code
4) simplifying and removing unneeded features
These are harder to do than ask for, because everyone's "why" for a change is different, and everyone's "unneeded features" differ, and every time a feature is added it lowers the marginal cost of continuing to work on the software - making it more likely the creators will hit the lottery of making that magic feature you love.
Now it's easier than ever to make and maintain your own software, so that might be a better road.
I actually think it would go entirely different way, since more and more 'ordinary' users would delegate whatever they need to use app for to 'their' agent instead of using app directly.
Agents have infinite patience, and they would go churn through most horrible buggy UIs out there to eventually achieve the goal, which is even more ironic, given that those UIs would be done by same agents in the first place.
I think AX is just as important in that case though, as even a patient agent will make mistakes and will have a lot more trouble getting things done right when using a buggy app.
There's an opportunity here to bend things towards quality and efficiency moreso than mere volume of code. The agent has to be able to effectively repair and maintain what it has been building though.
Makes me smile, seems ai written too. Agents will never fix this mess, it's upon us. But not in the way you're thinking. Asking agents to fix the mess they've been trained on, that they work on and they're made of it's just blindness utopia. We "just" need to part ways with current platforms(talking to you web) and build better abstractions and paradigms. The web plarform as is can't be saved anymore especially from us. It says a lot that many people get excited for a damn dialog html feature that we needed 20 years ago and it's also uncomplete
Agents that are blind to understanding the problem they created will definitely not be able to fix it. Our argument is that we can fix that blindness with better observability. For better or for worse, even if the code is messy, if an agent can see what is going on it can keep things together and get the app into working shape. 100% human written too btw...
In my experience they're never blind they can't even explain why they chose a more verbose, unreadable and error prone approach instead of a simpler one, only to candidaly admit that "you're absolutely right!". You can improve the observability how much you want, it will never prevent agents to go off the rails and destroy the database
AI can fix a bug just from a description and logs. I have never seen replication to be an issue. Replication was more important for human bug fixing as you needed to be able to tinker with it.
Maybe so, but it’s been something Ive only noticed lately. The same idea has been around since approximately forever, though. It’s not a new thing even in the last few years - things like monitoring dashboards pre-date the WWW.
In fact, dashboards giving feedback about opaque automated systems goes back to early automobiles putting gauges on the dashboard, which previously was just a board to block mud, rocks, etc. from kicking up in horse drawn buggies, presumably when the horse dashed.
Depends on the harness and the requirements right? It will infer meaning that regresses to the mean but you can likely feed it the requirements if you built the harness right and thought things through.
Actual developer: Hm, download button's not working. What's the href/onclick doing? Oh, there isn't one. Fixed.
Whatever this shit is: Thing not work, need more AI
Nothing about intent. A lot of bugs are correctly implemented but do the wrong thing. Actually in my experience these far outweigh the classic code bugs when it comes to web apps.
I think that is the interesting problem to solve (and would have the highest user impact)
At the risk of getting pedantic, should those really be considered bugs?
I think it's important to distinguish between "good design implemented poorly" (bugs) and "bad design implemented well" (design flaws) because the fixes are vastly different. I'm well aware that in a user's head, "it doesn't do what I want" gets filed under "bug," but those of us building the thing can benefit from being more precise.
What sort of bugs do you have in mind? From our perspective there are many kinds of bugs that inhibit UX (and AX) but they can be subtle issues like poorly designed flows. It's hard for an agent to notice or understand these but they are serious problems and we're trying to surface these more readily.
OK thanks, agents definitely like partial implementations like that. When this impacts users we need to be able to observe that and see the bug, then track it back to its source. Weird logic bugs like this benefit from having a good debugger to untangle things and get to the root of the problem, especially when there is tons of code.
This is really funny to read, as I have encountered way more bugs in software in the past year than ever.
The author probably mistakes working software for bug-free software.
LLMs can create working software as they are RF-ed with a compiler and tests. Not with bugs, they appear out of nowhere.
Yep, and web apps seem to be the worst offenders.
(OP) I completely agree! AI is amazing at writing web apps that are full of bugs, but it also gives us what we need to turn that completely around. That's what we're trying to make happen...
>> It is possible now to reliably and automatically detect and fix every bug which users are encountering on the web.
proceeds to give a trivial and then vague example related to usability, the first probably created by AI and the second a notoriously difficult problem to assess and solve for humans with context, let alone AIs...
The code I have dealt with from AI tends to be rather convoluted for the problems at hand and then become even more difficult to maintain. It has a place but I have been seeing it make code more difficult to understand, not less.
Convoluted code is definitely a problem, I don't think it's unique to AI though. It also becomes more manageable with a time travel debugger as the actual behavior is easy to understand.
I'm sure all commercial agents have a system instruction that says "make the code hard to follow" to make sure you keep sending those token$$$
(Replay engineer and Redux OSS maintainer)
Yep. I see it myself, both in other people's code, and our own. The _default_ behavior is a tendency to pile slop on slop, make poor architectural choices, slap bandaids on edge cases without considering the full ramifications, etc. This is more true if there isn't oversight or feedback, and also depending on model quality.
On the flip side: I personally use a strongly human-in-the-loop AI dev workflow [0]. Going back and forth on how much of that code I review myself (and I've found that AI code review frankly does a better job than _I_ do most of the time :) ). But, between staying hands-on, and the ongoing improvements in models, I've seen that the code my own agentic dev sessions produces is actually pretty high quality, and I've been able to clean up a lot of low-quality slop from unmonitored agent sessions. I've also been able to build some pretty major features from scratch and even execute massive migrations with excellent results, again by having me drive the work.
I actually just got done shipping Redux Toolkit 2.14 [1] with a massive set of bug fixes, and almost all of those were found by my agent researching the already-mature RTK codebase, at my direction, and identifying existing logic flaws or weaknesses. Once we knew what those bugs were, we could fix them.
Which, honestly, is the point of the blog post, of Replay QA as an automated testing product, and of Replay as a company in the first place.
Once you know what a bug _is_ and why it happens, fixing it is usually fairly straightforward. _Finding_ it is the hard part.
I've given a conf talk the last few years on "Debugging Software" [2]. One of the main points I make is that no one is _taught_ how to debug - we figure it out on the job as we go, because we have to ship. It's part of the job. But few people like doing it, and most people just want to slap in some console logs and call it a day. Very few people even use existing graphical debuggers (VS Code, Chrome DevTools, etc).
We built a time-travel debugger. The original idea was: make a DVR-style recording of a bug happening once, then you can dive in as deep as you need to investigate. Originally, that was supposed to be a human using the Replay DevTools UI - browser devtools with time travel debugging features built in. It's awesome, I wish I'd had it much earlier in my career :) But we found most people didn't want to spend time doing it themselves.
Earlier this year we shipped Replay MCP, which gives agents the same time-travel debugging capabilities: see hit counts for any line of code, dynamic logpoints and evaluations for every time a line of code ran, see the DOM tree or React component tree at any point in time. I built most of the React analysis features and MCP tools myself, based on my years of experience in the React ecosystem and maintaining Redux.
So, today you _could_ make a Replay recording of your app, point your agent at the MCP, and have it investigate. I've done that, and wrote some blog posts about how that helped my own agent find bugs in React itself and Next.js apps [3] [4]. I've seen my agent nail a bug from a recording on someone else's machine in <10min, starting from scratch.
We've productized that as Replay QA, so that you don't have to do any of the recording or setup - just point Replay QA at your app and repo and we do all the site analysis and recording and bug root cause investigation ourselves. Essentially, encoding our own expertise as debuggers, so you get the benefits [5]
So, back to the original point. Yeah, AI tends to crank out bad code by default, that code is buggy. We can't help your agent choose a better architecture. But we can help identify the bugs proactively, identify the root causes, and suggest solutions, based on the actual runtime behavior of the app. Up to you to get your agents to write _better_ code, but we can give your agent what it needs to fix the bugs it wrote :)
- [0]: https://blog.isquaredsoftware.com/2026/05/ai-thoughts-part-2...
- [1]: https://github.com/reduxjs/redux-toolkit/releases/tag/v2.14....
- [2]: https://blog.isquaredsoftware.com/2023/06/presentations-debu...
- [3]: https://replay.io/blog/replay-time-travelogue-how-replay-mcp...
- [4]: https://replay.io/blog/replay-time-travelogue-improving-nadi...
- [5]: https://replay.io/blog/mark-erikson-on-replay-qa-from-time-t...
Great, the thing that caused so many massive new problems for us all is now being sold to us as the solution to the problems it itself created.
Do you have anything to contribute to this community that isn't self promotion spam?
I found his comments and posts much more interesting that your constant anti-AI rants
I'd suggest that their solution of a swarm of agents generating QA slop makes the cure worse than the disease.
I can even recognise the AI bugs now - like randomly broken buttons and wierd unreadable text that no normal developer would release.
Is this just a promotion post for their "Replay QA™" feature?
It boils down to "other products find the issue and leave the root cause + fix to your agent, our product finds the issue + root cause and leaves the fix to your agent".
Who says I want your agent to root cause? Other products already give me the repro steps, video, trace, network logs, etc, and my agent is in a much better position in terms of context and setup to find the root cause and proper fix.
If you're happy with your agent's performance using surface level o11y to track down issues then that's fine. This is what pretty much the entire industry is doing and it does usually work! QA focuses on root causes because it can look through the app's entire execution and provide evidence so your agent doesn't need to guess when figuring out what the fix is. That guesswork is what we're trying to eliminate, for when the surface level stuff doesn't clearly point at the problem.
Here is my crazy list of ideas for less buggy web apps: 1) care 2) understanding why a change is made/needed before making it 3) deleting code 4) simplifying and removing unneeded features
These are harder to do than ask for, because everyone's "why" for a change is different, and everyone's "unneeded features" differ, and every time a feature is added it lowers the marginal cost of continuing to work on the software - making it more likely the creators will hit the lottery of making that magic feature you love.
Now it's easier than ever to make and maintain your own software, so that might be a better road.
Interesting. What harness could I use for 1? Be concise please, I don't have a lot of time.
did you intentionally just test and fail point one?
I actually think it would go entirely different way, since more and more 'ordinary' users would delegate whatever they need to use app for to 'their' agent instead of using app directly.
Agents have infinite patience, and they would go churn through most horrible buggy UIs out there to eventually achieve the goal, which is even more ironic, given that those UIs would be done by same agents in the first place.
For humans UX is already dead.
I think AX is just as important in that case though, as even a patient agent will make mistakes and will have a lot more trouble getting things done right when using a buggy app.
There's an opportunity here to bend things towards quality and efficiency moreso than mere volume of code. The agent has to be able to effectively repair and maintain what it has been building though.
And it will all be manipulated to increase token consumption as far as possible, to the provider’s benefit.
Makes me smile, seems ai written too. Agents will never fix this mess, it's upon us. But not in the way you're thinking. Asking agents to fix the mess they've been trained on, that they work on and they're made of it's just blindness utopia. We "just" need to part ways with current platforms(talking to you web) and build better abstractions and paradigms. The web plarform as is can't be saved anymore especially from us. It says a lot that many people get excited for a damn dialog html feature that we needed 20 years ago and it's also uncomplete
Agents that are blind to understanding the problem they created will definitely not be able to fix it. Our argument is that we can fix that blindness with better observability. For better or for worse, even if the code is messy, if an agent can see what is going on it can keep things together and get the app into working shape. 100% human written too btw...
In my experience they're never blind they can't even explain why they chose a more verbose, unreadable and error prone approach instead of a simpler one, only to candidaly admit that "you're absolutely right!". You can improve the observability how much you want, it will never prevent agents to go off the rails and destroy the database
Vibe coder sounds more legitimate than bug breeder.
AI can fix a bug just from a description and logs. I have never seen replication to be an issue. Replication was more important for human bug fixing as you needed to be able to tinker with it.
“Observability” is one of my least favorite buzzwords of the past year. It describes nothing new, aside from a general sense of AI-ness.
Observability is not a new term. It's been buzzing for several years now. It's also a good thing to have in your app...
Maybe so, but it’s been something Ive only noticed lately. The same idea has been around since approximately forever, though. It’s not a new thing even in the last few years - things like monitoring dashboards pre-date the WWW.
In fact, dashboards giving feedback about opaque automated systems goes back to early automobiles putting gauges on the dashboard, which previously was just a board to block mud, rocks, etc. from kicking up in horse drawn buggies, presumably when the horse dashed.
But can it read the minds of developed to know what the intended behavior is, or even if a bug, what the desired fix is?
Depends on the harness and the requirements right? It will infer meaning that regresses to the mean but you can likely feed it the requirements if you built the harness right and thought things through.
Actual developer: Hm, download button's not working. What's the href/onclick doing? Oh, there isn't one. Fixed. Whatever this shit is: Thing not work, need more AI
Nothing about intent. A lot of bugs are correctly implemented but do the wrong thing. Actually in my experience these far outweigh the classic code bugs when it comes to web apps. I think that is the interesting problem to solve (and would have the highest user impact)
At the risk of getting pedantic, should those really be considered bugs?
I think it's important to distinguish between "good design implemented poorly" (bugs) and "bad design implemented well" (design flaws) because the fixes are vastly different. I'm well aware that in a user's head, "it doesn't do what I want" gets filed under "bug," but those of us building the thing can benefit from being more precise.
What sort of bugs do you have in mind? From our perspective there are many kinds of bugs that inhibit UX (and AX) but they can be subtle issues like poorly designed flows. It's hard for an agent to notice or understand these but they are serious problems and we're trying to surface these more readily.
Something along the lines of
def start_of_week() do "Monday" end
OK thanks, agents definitely like partial implementations like that. When this impacts users we need to be able to observe that and see the bug, then track it back to its source. Weird logic bugs like this benefit from having a good debugger to untangle things and get to the root of the problem, especially when there is tons of code.
And now we are going to have to deal with this animal.
The "AI Bugfixer Extraordinaire"..
Sigh, do us all a favor and go back into whatever crack you crawled out of.
[flagged]
[dead]
[dead]