I'm currently working on a combined Aero sim/CAD/CAM software for designing, simulating, and performing DFM optimization specifically for Group 1 to 3 winged UAVs.
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.
> How to deal with shipping AI-written code you don't understand?
Stop shipping code you don't understand.
Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.
Generally speaking I brain storm the feature first and produce a dev plan. This includes asking it to reuse code/libraries we’ve already got! After it’s gone off ah done its thing we go back over and do a QA on what it’s built, why that approach, what can we do better etc etc?
Product doesn’t care. And most of us will get replaced by agents. I guess around 30% of the engineering workforce will keep their jobs which will be mainly to keep the product up and running, doing whatever it takes (so using agents to fix stuff basically).
The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.
I’m slowly coming around to the idea that if it works, I’m good with that. I never used to personally check all my colleagues work, either, I just trusted that they’d done a good job.
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
I am not in META but for what I read on blind, assuming it is true, they are not spending any time in long planning sessions or in getting into details of the implementation plan at all, and they could not anymore considering how many PRs in parallel they are requested to manage
Yeah, I'm retired. The only person I have to answer to regarding progress is me. I'm a stubborn son of a bitch when it comes to choosing quality over speed...
That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
Those are some pretty fine speedups, from naive C, with no alterations from the standard "C" code - it automatically finds places that can be made data-parallel, tracks the data-hazards to make sure its safe to "GPU" and writes all the kernels, boilerplate and data-stream-management if necessary. Obviously the checksums have to match, so I know it did the work correctly too :)
Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
wow, “quality is no longer an objective” is a wild idea that feels dangerously negligent.
quality is all we have, it’s what makes us different from these agents.
quality is perplexing to agents because it requires a strange loop into asking why. there’s not always a clear answer. but i think the power is in the
ambiguity.
if we don’t know why (or don’t care) then what happens if we all adopt this attitude at scale?
The entire industry is dangerously negligent, we just have better tools and stronger incentives now. Why spend 3h code-reviewing a PR when you know if it breaks you just tell your agent to watch out for that specific scenario? Which executive will see that as time well spent?
Sort of. What if you asked your second agent to fuzz test the work of the first agent, adversarial code review, or make a video to explain all the code.
It feels like another layer of "conceptual compression". I'm not sure who/when that phrase came around but in context of Rails is where I heard it and its things like ActiveRecord or really any ORM. You aren't writing the SQL directly, something else is and for the most part it works. Then it doesn't work or your database grows too much and you need to optimize table structures with indexes, etc.
My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.
It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.
The part that makes me uneasy is when the AI writes both the code and the tests. They might agree with each other and still miss the same thing. How are people here checking for that without doing the whole job twice?
I like fault injection (fuzzing, chaos), and mutation testing. I think introducing these helps the AI write better tests and helps having some confidence in the tests that are written.
I'm currently working on a combined Aero sim/CAD/CAM software for designing, simulating, and performing DFM optimization specifically for Group 1 to 3 winged UAVs.
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.
> How to deal with shipping AI-written code you don't understand?
Stop shipping code you don't understand.
Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.
Generally speaking I brain storm the feature first and produce a dev plan. This includes asking it to reuse code/libraries we’ve already got! After it’s gone off ah done its thing we go back over and do a QA on what it’s built, why that approach, what can we do better etc etc?
I had same problem, but after a while I just accepted that is inevitable
"This code is really ugly."
That sucks. Ship it.
Product doesn’t care. And most of us will get replaced by agents. I guess around 30% of the engineering workforce will keep their jobs which will be mainly to keep the product up and running, doing whatever it takes (so using agents to fix stuff basically).
The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.
Code understanding doesn’t matter anymore (or won’t matter soon)
I spend time to add extra observability, cross-checking and alerting at runtime to offset the risk.
I’m slowly coming around to the idea that if it works, I’m good with that. I never used to personally check all my colleagues work, either, I just trusted that they’d done a good job.
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
I am not in META but for what I read on blind, assuming it is true, they are not spending any time in long planning sessions or in getting into details of the implementation plan at all, and they could not anymore considering how many PRs in parallel they are requested to manage
Yeah, I'm retired. The only person I have to answer to regarding progress is me. I'm a stubborn son of a bitch when it comes to choosing quality over speed...
That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
Those are some pretty fine speedups, from naive C, with no alterations from the standard "C" code - it automatically finds places that can be made data-parallel, tracks the data-hazards to make sure its safe to "GPU" and writes all the kernels, boilerplate and data-stream-management if necessary. Obviously the checksums have to match, so I know it did the work correctly too :)Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
The solution is more AI. Throw another agent in to watch your logs, then throw in another agent to watch that agent.
Quality is no longer an objective. If it breaks, rebuild it and tell it not to break.
wow, “quality is no longer an objective” is a wild idea that feels dangerously negligent.
quality is all we have, it’s what makes us different from these agents.
quality is perplexing to agents because it requires a strange loop into asking why. there’s not always a clear answer. but i think the power is in the ambiguity.
if we don’t know why (or don’t care) then what happens if we all adopt this attitude at scale?
The entire industry is dangerously negligent, we just have better tools and stronger incentives now. Why spend 3h code-reviewing a PR when you know if it breaks you just tell your agent to watch out for that specific scenario? Which executive will see that as time well spent?
Sort of. What if you asked your second agent to fuzz test the work of the first agent, adversarial code review, or make a video to explain all the code.
It's agents all the way down!
<that scene from the Matrix with hundreds of Agent Smith>.gif
It feels like another layer of "conceptual compression". I'm not sure who/when that phrase came around but in context of Rails is where I heard it and its things like ActiveRecord or really any ORM. You aren't writing the SQL directly, something else is and for the most part it works. Then it doesn't work or your database grows too much and you need to optimize table structures with indexes, etc.
My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.
It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.
It's Meta. Shipping crap is par for the course. You're fine.
The part that makes me uneasy is when the AI writes both the code and the tests. They might agree with each other and still miss the same thing. How are people here checking for that without doing the whole job twice?
I like fault injection (fuzzing, chaos), and mutation testing. I think introducing these helps the AI write better tests and helps having some confidence in the tests that are written.
Not sure about that. This was structurally the same before AI: the author of the code wrote the tests. The tests still helped.