> What shipped, who's using it, and how it's helping them matters FAR MORE than lines of code, prs merged, tokens spent, or the number of agents running in parallel.
Sometimes when I hear shocking amounts of money spent on AI tokens, I wonder if someone's obsessing over the tech and forgetting that only the result matters.
This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound.
Not saying that this way to review is a silver bullet, but at least it removes some weight.
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.
The discipline of splitting them up helps in almost every dimension.
To add to my openspec shilling above, I've been known to "fix" large PRs by reverse engineering specs from them and building each spec out in a separate PR.
Last week me and a coworker turned a 15k line PR into 10 x 1.5k line ones that went in smoothly.
Yes I do like to make a big pr first sometimes then it's usually pretty easy to split it into small ones and it's a helpful process in many ways. You can even present both together so people can see it either way, but I think in most systems it's better to have smaller prs merged one at a time.
For a rewrite, I guess sometimes it's better to have a spec then write to that, and PRs might be larger.
So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?
Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.
If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.
Where did grey-area say they need to be small. As in an absolute? They didn't, you just assumed that's what they meant and then went off the rails in your response.
The first 4 words? "Just ban large prs." A ban is pretty absolute language. He would have banned my example PRs clearly. Or maybe he wouldn't which is why I asked about them as examples. Because that is the obvious follow up. If you ban large PRs what are you losing? You're loosing big swing aggressive efforts to make sweeping progress.
That was a comment written in a particular context, where the original commenter was concerned with large PRs.
>> how can we review gigantic PR generated by AI.
It was not a comment written in a vacuum saying that all PRs should be small (or not large, whatever that means). Though grey-area does provide some reasonable commentary for why it's good to shrink them (easier to roll back, easier to isolate bugs, etc.). Maybe grey-area does mean to ban all large PRs, but that's not evident in this thread where they wrote a response to a specific comment offering their take on how to resolve the problem raised in that first comment.
> You're loosing big swing aggressive efforts to make sweeping progress.
Just a tip: "losing" not "loosing". We loose arrows at our enemies, we let loose the dogs of war, we loosen our purse strings. But we do not "loose big swing aggressive efforts", we "lose big swing aggressive efforts". It's funny that you get it right once, but mix the two up in the very next sentence.
It's like responding to "you should wear a seatbelt, statistically they keep you safer in a crash" with "here are two people who got in a crash without a seatbelt and survived"
I, personally, would certainly argue that those should be done in small PRs. If you have a working application and want to rewrite in rust, I would expect it to be done by replacing individual components one at a time while making sure relevant tests keep passing. If you're making a new application, I would expect it to be built incrementally by adding parts incrementally.
I think they could do those things with small PRs if they chose. If their agents are not intelligent enough to make small prs, I guess they can YOLO and assume everything is just fine with what the agents did. I do think small PRs would be better for agent review as well though, because they avoid any problems with context being overwhelmed and things being missed/forgotten.
What are the PRs you're thinking of though. If you mean a full clean rewrite, they'd probably proceed with spec first then just write it in bits to the spec. No reason it has to be large prs or smaller that's a design choice.
Often an exercise I find useful is do a quick big PR, then split it into small chunks that make sense independently for review. Or you can chain prs. Both those things can be done.
You're welcome to do large PRs, I'm not a fan of huge PRs though and would not pretend to review them (as pretending is what it would be), nor would I like to work somewhere where this is the culture.
> One way could be to ask the submitter to give their prompt as part of their contribution
Zed's delta (https://delta.dev/) seems to be solving this problem by extending the version control to the conversation layer. I don't know if I'd want all my conversation to be tracked, some stuff belongs to the draft sheet.
Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:
1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list.
2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review.
3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.
My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.
Up until you notice that in the spec an agent wrote "All operations should be atomic". But, the you observe that it keeps modifying the same value across multiple threads because there's another spec that states "No perf regression, pls".
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
At least for my workflow, which is fairly similar, the result of the first few stages is a refined single prompt, which I then run and the result is a PR. so I, at least, would include that final single prompt over the prior intermediate results.
I also think that might not be everything you need to really vet/recreate those PRs since there's a lot of strategic and operational stuff that goes into the pre-composition runs (sometimes I actually throw everything away and restart), so something a little more sophisticated would be interesting.
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.
However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.
Because, unless you are changing some code by hand. Your prompt and the context your giving to your agent is the seed for the outcome of the agent itself. No agent as far as I know can generate content ex-nihilo. So having a look at what people are telling their agent to do can definitely help understanding the result of the same agent in my opinion.
That is patently false in the real world we live in currently. I constantly see lazy coworkers post 2000 line PRs for a trivial change that their LLM generated when 500 lines might accomplish the same in a better way. It's almost always a lazy human accepting what Claude did by default without any manual review or asking the LLM to do anything differently.
Claude absolutely loves to reimplement the wheel locally instead of just calling a method in a library we already depend on. Every doc comment and README.md update is bloated by 2-5x what's needed to communicate intent or to actually be read by a human later. It makes random drive-by edits that aren't necessary and weren't asked for in the prompt. It's a well oiled machine for producing astronomical bloat by default if you don't supervise and rein it in!
Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.
Why, yes, blame someone else, and not the person that shat out the PR :)
I had a recent case where I, being the lead, sliced up the work real good into an epic and what not and a guy on the team just did a "hurr durr, Claude, go wild with all that" and threw a huge PR over the wall linked to 5+ stories.
Well, we got to have "the conversation" after that and learned how to fix these kinds of PRs together...
A short formulation that I really like in the style of the original:
--------
We are uncovering better ways of developing systems by using AI and guiding others to use it responsibly.
Through this work we have come to value:
Human accountability over delivery velocity
Human dignity and control over unsupervised autonomous execution
Human augmentation over replacement
Impactful outcomes over raw code generation
Human collaboration over individual uplift
That is, while there is value in the items on the right, we value the items on the left more.
The problem is that the things on the left are legitimately less valuable than the things on the right to most people. The target audience will eat it up, but why should anyone who isn't a developer care? This entire thing fails to explain it's value to anyone who isn't already convinced.
For example -
You make more money if you deliver quickly than you do if you have "accountability." What even is accountability, how do I explain it to a CEO? Is this office speak for blame? Is the idea to know who to yell at so you can feel better, or to make money? Feelings don't sell widgets.
"Human dignity and control" - Who is this valuable to? Not consumers, they want Walmart prices. Not companies, they want Walmart profits. What's in it for them?
I'm not saying I disagree, I'm saying this isn't how you sell it.
Could use a non-negative name. Just like it was not the ‘no heavyweight dev processes’ manifesto but Agile manifesto which was wsy catchier to reference, maybe yours could be named like the… Human-serving Engineering manifesto or something?
The manifesto circles around a central idea that it never fully identifies: whether AI is involved or not, all slop is made by HUMANS. Because the real cause of "slop" is that someone didn't care and published the first result they got, as if it was the final product. They didn't put in the effort to analyze what they got, improve on it and deliver a minimum quality.
And this is something that has always happened. Way earlier than current AIs we had already coined the term "shovelware" that was mostly a large quantity of sloppy software. We also had plenty of sloppy drawings at sites like DeviantArt. AI just made it way easier for slop to happen, but the ones producing slop are always people.
You can't go in with fighting AGAINST it. You need a vision of your you incorporate it into life in the most beneficial way.
There is a lot of value in the fact that none engineers can build products now! Should AI products adhere to a certain quality standards? Yes! Should they be treated similarly as "human created" ones... Well, yes. Like it or hate it.
One sentence really caught my eye.... "We may still review a plan, design, or result, because if we don't, we just get the same output as everyone else."
Is getting the same output as everyone else bad? I'd say not necessarily. When you're making something, you might want to spend your "innovation tokens" on the core of what you want to do better. Then do a good-enough job at everything else.
To me it feels like the subtext in this is that making things more accessible is bad because then your stuff isn't unique anymore? I do not think this zero-sum / scarcity mindset is really accurate or helpful, IMO.
> 17. There will always be alpha in reviewing something.
What does alpha mean in this case? I only understand this term in from a finance perspective (excess returns). Is it used here to mean 'value'? E.g. there is always value in reviewing something?
The problem is: how can you scale human code review? Because LLMs continue to be better at coding, and continue to write a lot of code, and will always code at a lower level with the time passes. So the difficulty of human review will become higher and higher.
It's clear that human review it's the very bottleneck in this workflow.
Faster definitely, but even with the latest Fable model "better" is often debatable and varies a lot based on what you care about. I can often rewrite what Claude did in 1/2 the lines or less to do the same thing while making it easier for humans to understand and maintain. Claude gives me a sculpture in a good rough starting shape, but it usually needs to be sculpted down at the end whether manually or by prompting Claude about the unnecessary bloat it generated.
A major reason slop exists is because they are not "better" at coding, given some subjective criteria. They are notably worse at documentation and somewhat poor at choosing abstractions in my experience. For the most part, I witness the tools copying other examples out of the working repository, which makes them exactly as bad as the humans who originally wrote the code.
> will always code at a lower level with the time passes
For humans, this is traditionally an anti-pattern. We are good at intuiting abstractions and knowing when to use them or climb up and down the abstraction hierarchy. For LLMs, this is also an anti-pattern, because going lower-level is an exponential waste of tokens; both for writing and reading the resulting code, which they do constantly in order to understand the project again and again in order to fulfill your prompts.
Thinking of human review as a "bottleneck" is incorrect framing. We are quality control. You can choose to cede quality control to LLMs and agent harnesses, that's up to you. In my experience this generally leads to a meteoric rise in slop, because the tools increasingly resort to writing more code to solve a problem instead of updating existing code or writing the correct code. When I shake my head at some slop code, then read the slop comment that uses some slop justification that isn't real and only existed in the prompt context, I am happy that I took the time to review it. Maybe that makes me a bottleneck, oh well.
Yes of course we need. Software sucks. And sucks because there is a LOT of work to do. Always. Not only strictly programming work, right. But in fact LLMs can do also UI, UX and more.
We have a fantastic tool which is cheap and very fast. With good quality even if not great. This is a massive leap.
Truly wondrous, that is the most gag-inducing text I've read in months! It's like an LLM was promoted to say sweet nothings in corporatese and therapy speak while drawing inspiration from the speeches of the most vapid politicians.
I was only thinking to myself yesterday how much I grieve the fall of agile and DevOps. It was all the industry cared about for a while and the movement behind it made me hopeful for the future of work.
AI has really disrupted that, and I’m thrilled to see folks trying to bring back the best parts of it (namely shared understanding and contribution). But I agree with others, making the movement anti slop feels the wrong way around. What it’s getting at IMO is deep understanding and shared understanding are crucial to human progress, and reckless AI use compromises that
> Slop is when we put less effort into producing something than we ask the consumer to spend understanding it.
This feels directionally right, but if you define slop this way then you put a cap on AI productivity. For example, I heard an interview with Terry Tao talking about how the work of mathematicians is now to understand the proofs generated by AI. He's, by definition, describing these groundbreaking proofs as slop (by the above definition).
Well maybe Tao is wrong and we should instead focus on making LLMs explain their ideas better. It really seems eminently achievable and would be better all round.
I think the problem with those Maths proofs was they were massively rushed and OpenAI just couldn't be bothered to do it properly or actually check them. They were more interested in the PR than in helping advance Mathematics.
> Comments and feedback on this document are welcome. What is slop? Why does slop happen? Why is slop worth fighting? How do we fight slop?
This sounds like slop. Like an engagement bait. Also a no slop manifesto should start by defining “slop” as the first thing. That’s not something for the comment section.
> AI cannot replace human judgement, it can only amplify our judgement (or lack thereof)
> Slop is asymmetric. The cost of producing a thousand-line PR or a ten-page doc is approaching zero, while the cost of reviewing it remains constant.
I'd say the cost actually grows as people get laid off left and right, and on top of that, a 1000 line PR from a good human, give me a case of beer and I'll make me some sense of it, but if it's AI slop, get ready to hand me a crate of homemade Slivo or I won't touch that...
Plus, I find reading slop docs takes cognitively much more effort. When I find problems in human docs, it's easier for me to point out the issue, comment etc. But on these AI slop docs, I find it overwhelming to comment on, because the issues are often woven throughout the document in a way that makes it so I can't isolate the comment in a single place. It's hard for me to describe.
> 1. AI is the most useful tool most of us have ever worked with, but a swarm of agents can produce more work in a single day than a person can review in our lifetime.
When you put it like that, seems the problem is on the human side
The Agile Manifesto was successful since companies had a market advantage in delivering instead of wasting money.
Will "No AI[1] slop, aim for understanding system we ship in human brains instead" be advantageous in capitalism? I sincerely doubt it. And that makes me sad. Even in the public sector, I now am expected to deliver faster, since the industry changed comparable outputs of our private contractors.
[1]: Or, to prevent ``slop'': at least, no speed-up beyond catching up to what was still well-thought out in terms of input and/or reviewed by humans. AI can still be a tool, but must never take all roles in a process from gradle to grave.
Aren't you really trying to get at is a Human Flourishing manifesto? Your postions are pro-human, anti-AI. But what if AI did go away tomorrow because... um... Noosphere Prime Intellect awoke and humans started working with it? Your pro-human arguments probably wouldn't change but all the AI slop stuff would be slop itself.
While I am sympathetic to the authors, the reason that agile swept over the industry is that it was a demonstrably more effective when measured by business outcomes. "AI slop all the things" is miserable for human employees and introduces new kinds of dysfunctions called out or hinted at in this manifesto. However, if on balance it is more effective when measured by business outcomes, there is little chance of this manifesto gaining traction in the industry.
From my own personal experience, which may not reflect the broader industry although I suspect it does, the ones who embrace slopping with the least regard for their peers are also the ones perceived as the most effective by their employers. Those who take a principled stance get the worst of both worlds. They cannot combat the deluge of slop with their own clanker, and they lose the influence with their company leadership that would provide sponsorship to change how AI is being used. This creates competitive pressure to follow suit, and from there it is a race to the bottom.
Every software product proudly labeled as "we used AI to make this" sucks. For example, people have repeatedly pointed out how bad Claude Code is, precisely because it's just AI slop. Microsoft going down yesterday is almost certainly also due to slop. And so on.
This ship has sailed. We first need to suffer the consequences (if there will be any, not arguing that) before action is taken to prevent the consequences from repeating in the future. It's just human nature. Even the agile manifesto happened after decades of doing it "wrong". Most people who don't go on the slop train will experience their colleagues doing it and getting rewarded for it. The HN crowd is in general an exception here, being above average (not counting me), but most people experience management expectations that are not too forward looking.
I had thought the same while reading. Every single letter/word/line needs a review and even that might not be enough to prevent slop. Use AI for reviewing, catching edge cases and improving the overall quality instead.
Seems like a good thought for pre-agentic era where ai assisted existing human workflows. We are fast approaching the end of that era. In the agentic future the past human-centric workflows (reviewing, validating, approving etc.) will be replaced with agent-to-agent workflows. There will be resistance from humans who would want to be in the comfort of familiar human-centric workflows. The survivors will see where we are heading and adapt.
This assumes that "humans owning the intent, taste, and consequences" is necessarily a good thing, and agents will necessarily do a worse job. This is not grounded in anything but vibes.
LOL stop it already with the agentic crap. Besides the already automation-ready software use cases, agents are simply the wrong tool for the job, unless you bend and distort the definition of such job.
Y’all fully invested in this agentic bullshit that you sound delusional and talk in self-evident truths that nobody outside of your niche SWE bubble would ever share.
I can't argue against you having an alternate opinion. But "humans better than ai for taste, judgement, reviews etc." is the self-serving assumption not grounded in any empirical evidence.
More than 20 apps developed in the last six months, all top quality, I haven't coded a single line, speed and output is what matters, results improving by the day and will be unrecognizable in a couple of years by any metric
AI is unstoppable and no matter how many manifests luddites write AI won't care
The authors should take the parent comment as a sign their manifesto's title is bad. Some will read the title and make a snap judgement that the manifesto must be totally against AI.
The fact of the matter is software development of the future is not going to look like AI slop, and its also not going to look like the hand crafted engineering of the past. It's going to be a whole new third thing of organically growing software with constraints and processes that result in quality. The degree to which different companies and people have figured it out varies wildly, and almost every best practice so far is wildly over confident and untested. We were still arguing orver OOP and thats been around for 50 years. People are having a meltdown over the changes in the last 2 years and reacting in wildly unpredictable ways.
My favorite bit:
> What shipped, who's using it, and how it's helping them matters FAR MORE than lines of code, prs merged, tokens spent, or the number of agents running in parallel.
Sometimes when I hear shocking amounts of money spent on AI tokens, I wonder if someone's obsessing over the tech and forgetting that only the result matters.
My favorite bit:
Slop is when we put less effort into producing something than we ask the consumer to spend understanding it.
> Improving the system over patching the output
This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound. Not saying that this way to review is a silver bullet, but at least it removes some weight.
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.
The discipline of splitting them up helps in almost every dimension.
Fully agree with a twist - ban dumb large PRs.
To add to my openspec shilling above, I've been known to "fix" large PRs by reverse engineering specs from them and building each spec out in a separate PR.
Last week me and a coworker turned a 15k line PR into 10 x 1.5k line ones that went in smoothly.
Yes I do like to make a big pr first sometimes then it's usually pretty easy to split it into small ones and it's a helpful process in many ways. You can even present both together so people can see it either way, but I think in most systems it's better to have smaller prs merged one at a time.
For a rewrite, I guess sometimes it's better to have a spec then write to that, and PRs might be larger.
So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?
Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.
If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.
Where did grey-area say they need to be small. As in an absolute? They didn't, you just assumed that's what they meant and then went off the rails in your response.
The first 4 words? "Just ban large prs." A ban is pretty absolute language. He would have banned my example PRs clearly. Or maybe he wouldn't which is why I asked about them as examples. Because that is the obvious follow up. If you ban large PRs what are you losing? You're loosing big swing aggressive efforts to make sweeping progress.
> The first 4 words? "Just ban large prs."
That was a comment written in a particular context, where the original commenter was concerned with large PRs.
>> how can we review gigantic PR generated by AI.
It was not a comment written in a vacuum saying that all PRs should be small (or not large, whatever that means). Though grey-area does provide some reasonable commentary for why it's good to shrink them (easier to roll back, easier to isolate bugs, etc.). Maybe grey-area does mean to ban all large PRs, but that's not evident in this thread where they wrote a response to a specific comment offering their take on how to resolve the problem raised in that first comment.
> You're loosing big swing aggressive efforts to make sweeping progress.
Just a tip: "losing" not "loosing". We loose arrows at our enemies, we let loose the dogs of war, we loosen our purse strings. But we do not "loose big swing aggressive efforts", we "lose big swing aggressive efforts". It's funny that you get it right once, but mix the two up in the very next sentence.
It's like responding to "you should wear a seatbelt, statistically they keep you safer in a crash" with "here are two people who got in a crash without a seatbelt and survived"
Discord's rewrite took 5+ years. Yes they probably used small PRs
I, personally, would certainly argue that those should be done in small PRs. If you have a working application and want to rewrite in rust, I would expect it to be done by replacing individual components one at a time while making sure relevant tests keep passing. If you're making a new application, I would expect it to be built incrementally by adding parts incrementally.
Yeah, if you ended up doing it as a single massive PR how can you ever be sure that existing functionality was preserved?
I think they could do those things with small PRs if they chose. If their agents are not intelligent enough to make small prs, I guess they can YOLO and assume everything is just fine with what the agents did. I do think small PRs would be better for agent review as well though, because they avoid any problems with context being overwhelmed and things being missed/forgotten.
What are the PRs you're thinking of though. If you mean a full clean rewrite, they'd probably proceed with spec first then just write it in bits to the spec. No reason it has to be large prs or smaller that's a design choice.
Often an exercise I find useful is do a quick big PR, then split it into small chunks that make sense independently for review. Or you can chain prs. Both those things can be done.
You're welcome to do large PRs, I'm not a fan of huge PRs though and would not pretend to review them (as pretending is what it would be), nor would I like to work somewhere where this is the culture.
> One way could be to ask the submitter to give their prompt as part of their contribution
Zed's delta (https://delta.dev/) seems to be solving this problem by extending the version control to the conversation layer. I don't know if I'd want all my conversation to be tracked, some stuff belongs to the draft sheet.
Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:
1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list.
2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review.
3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.
My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.
Your workflow looks really neat, I'll definitely give it a try.
So now I get to review the prompts and a slopped PR-description? Oh man, that sounds nice.
Happy to hear how you do it. We're all figuring that out.
Up until you notice that in the spec an agent wrote "All operations should be atomic". But, the you observe that it keeps modifying the same value across multiple threads because there's another spec that states "No perf regression, pls".
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
At least for my workflow, which is fairly similar, the result of the first few stages is a refined single prompt, which I then run and the result is a PR. so I, at least, would include that final single prompt over the prior intermediate results.
I also think that might not be everything you need to really vet/recreate those PRs since there's a lot of strategic and operational stuff that goes into the pre-composition runs (sometimes I actually throw everything away and restart), so something a little more sophisticated would be interesting.
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
Would have to include the full conversion history. But who is gonna review that it's probably as big as some of these big PRs are in the first place.
Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.
However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.
How does reviewing the long discussion the author had with the LLM prove the code is correct?
Because, unless you are changing some code by hand. Your prompt and the context your giving to your agent is the seed for the outcome of the agent itself. No agent as far as I know can generate content ex-nihilo. So having a look at what people are telling their agent to do can definitely help understanding the result of the same agent in my opinion.
If a PR is gigantic it means the team lead screwed up slicing the work.
That is patently false in the real world we live in currently. I constantly see lazy coworkers post 2000 line PRs for a trivial change that their LLM generated when 500 lines might accomplish the same in a better way. It's almost always a lazy human accepting what Claude did by default without any manual review or asking the LLM to do anything differently.
Claude absolutely loves to reimplement the wheel locally instead of just calling a method in a library we already depend on. Every doc comment and README.md update is bloated by 2-5x what's needed to communicate intent or to actually be read by a human later. It makes random drive-by edits that aren't necessary and weren't asked for in the prompt. It's a well oiled machine for producing astronomical bloat by default if you don't supervise and rein it in!
Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.
It also means the engineer didn't check in with the lead as their PR became unwieldy.
Why, yes, blame someone else, and not the person that shat out the PR :)
I had a recent case where I, being the lead, sliced up the work real good into an epic and what not and a guy on the team just did a "hurr durr, Claude, go wild with all that" and threw a huge PR over the wall linked to 5+ stories.
Well, we got to have "the conversation" after that and learned how to fix these kinds of PRs together...
A short formulation that I really like in the style of the original:
--------
We are uncovering better ways of developing systems by using AI and guiding others to use it responsibly.
Through this work we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.The problem is that the things on the left are legitimately less valuable than the things on the right to most people. The target audience will eat it up, but why should anyone who isn't a developer care? This entire thing fails to explain it's value to anyone who isn't already convinced.
For example -
You make more money if you deliver quickly than you do if you have "accountability." What even is accountability, how do I explain it to a CEO? Is this office speak for blame? Is the idea to know who to yell at so you can feel better, or to make money? Feelings don't sell widgets.
"Human dignity and control" - Who is this valuable to? Not consumers, they want Walmart prices. Not companies, they want Walmart profits. What's in it for them?
I'm not saying I disagree, I'm saying this isn't how you sell it.
This reads quite similarly to the thesis of Cory Doctorow's recent 'reverse centaur' book on AI, if you're interested in similar takes
Could use a non-negative name. Just like it was not the ‘no heavyweight dev processes’ manifesto but Agile manifesto which was wsy catchier to reference, maybe yours could be named like the… Human-serving Engineering manifesto or something?
May I suggest "Responsible AI Use Charter"?
People need to stop sending me documents at work that are clearly AI drafts. It's becoming infuriating.
The manifesto circles around a central idea that it never fully identifies: whether AI is involved or not, all slop is made by HUMANS. Because the real cause of "slop" is that someone didn't care and published the first result they got, as if it was the final product. They didn't put in the effort to analyze what they got, improve on it and deliver a minimum quality.
And this is something that has always happened. Way earlier than current AIs we had already coined the term "shovelware" that was mostly a large quantity of sloppy software. We also had plenty of sloppy drawings at sites like DeviantArt. AI just made it way easier for slop to happen, but the ones producing slop are always people.
You can't go in with fighting AGAINST it. You need a vision of your you incorporate it into life in the most beneficial way.
There is a lot of value in the fact that none engineers can build products now! Should AI products adhere to a certain quality standards? Yes! Should they be treated similarly as "human created" ones... Well, yes. Like it or hate it.
One sentence really caught my eye.... "We may still review a plan, design, or result, because if we don't, we just get the same output as everyone else."
Is getting the same output as everyone else bad? I'd say not necessarily. When you're making something, you might want to spend your "innovation tokens" on the core of what you want to do better. Then do a good-enough job at everything else.
To me it feels like the subtext in this is that making things more accessible is bad because then your stuff isn't unique anymore? I do not think this zero-sum / scarcity mindset is really accurate or helpful, IMO.
> 17. There will always be alpha in reviewing something.
What does alpha mean in this case? I only understand this term in from a finance perspective (excess returns). Is it used here to mean 'value'? E.g. there is always value in reviewing something?
In this context, "alpha" is knowledge that gives you a competitive advantage.
Literally means excess returns here as well (a better outcome to those that don't review anything).
Is it a claudism?
How about explaining its origin if it isn't, if you're going to downvote?
The problem is: how can you scale human code review? Because LLMs continue to be better at coding, and continue to write a lot of code, and will always code at a lower level with the time passes. So the difficulty of human review will become higher and higher. It's clear that human review it's the very bottleneck in this workflow.
> LLMs continue to be better at coding
Faster definitely, but even with the latest Fable model "better" is often debatable and varies a lot based on what you care about. I can often rewrite what Claude did in 1/2 the lines or less to do the same thing while making it easier for humans to understand and maintain. Claude gives me a sculpture in a good rough starting shape, but it usually needs to be sculpted down at the end whether manually or by prompting Claude about the unnecessary bloat it generated.
You're right. But this can be fixed right now with a different harness, with more/better prompting, and will be fixed of course with newer models
> Because LLMs continue to be better at coding
A major reason slop exists is because they are not "better" at coding, given some subjective criteria. They are notably worse at documentation and somewhat poor at choosing abstractions in my experience. For the most part, I witness the tools copying other examples out of the working repository, which makes them exactly as bad as the humans who originally wrote the code.
> will always code at a lower level with the time passes
For humans, this is traditionally an anti-pattern. We are good at intuiting abstractions and knowing when to use them or climb up and down the abstraction hierarchy. For LLMs, this is also an anti-pattern, because going lower-level is an exponential waste of tokens; both for writing and reading the resulting code, which they do constantly in order to understand the project again and again in order to fulfill your prompts.
Thinking of human review as a "bottleneck" is incorrect framing. We are quality control. You can choose to cede quality control to LLMs and agent harnesses, that's up to you. In my experience this generally leads to a meteoric rise in slop, because the tools increasingly resort to writing more code to solve a problem instead of updating existing code or writing the correct code. When I shake my head at some slop code, then read the slop comment that uses some slop justification that isn't real and only existed in the prompt context, I am happy that I took the time to review it. Maybe that makes me a bottleneck, oh well.
Take a look at my workflow I'm using for developing my de, vela: https://velade.org/blog/not-vibecoding/
I think if you see real examples you can better understand what I mean
But do we really need to generate orders of magnitude more code per day than what people were writing a few years ago?
Yes of course we need. Software sucks. And sucks because there is a LOT of work to do. Always. Not only strictly programming work, right. But in fact LLMs can do also UI, UX and more. We have a fantastic tool which is cheap and very fast. With good quality even if not great. This is a massive leap.
LLMs are not better at coding than any human of reasonable skill. They do quite badly at coding.
The AI slop code is so good that I’m kicking back more Jira tickets than ever for failing my tests.
Skill issue. Learn to use a harness
I kicked them back for failing my tests. I’m the SDET. Stop sending me sloppy code.
I think my problem with a lot of middle positions on AI usage is that AI is viral. The more you use AI in a project the more you need to use it.
Here, let's trade manifestos :) https://johnoestmannmusic.com/manifesto-academy-of-creative-...
FYI, the live-typing header wraps and toggles between one and two lines, causing the body text to jump around indefinitely on my phone.
Ah, cheers for the heads up! I'll look into this.
Truly wondrous, that is the most gag-inducing text I've read in months! It's like an LLM was promoted to say sweet nothings in corporatese and therapy speak while drawing inspiration from the speeches of the most vapid politicians.
Hah, and I wrote this by hand, partly, in response to everything I was seeing with LLMs making people feel disheartened about creativity.
I was only thinking to myself yesterday how much I grieve the fall of agile and DevOps. It was all the industry cared about for a while and the movement behind it made me hopeful for the future of work.
AI has really disrupted that, and I’m thrilled to see folks trying to bring back the best parts of it (namely shared understanding and contribution). But I agree with others, making the movement anti slop feels the wrong way around. What it’s getting at IMO is deep understanding and shared understanding are crucial to human progress, and reckless AI use compromises that
> The more slop flying around a team, the more we lose the will to fight it.
So true, I feel guilty not fighting it at the place I work at, but it seems impossible.
This manifesto gives me a spark on hope I haven't felt in months.
I signed, very balanced take.
> Slop is when we put less effort into producing something than we ask the consumer to spend understanding it.
This feels directionally right, but if you define slop this way then you put a cap on AI productivity. For example, I heard an interview with Terry Tao talking about how the work of mathematicians is now to understand the proofs generated by AI. He's, by definition, describing these groundbreaking proofs as slop (by the above definition).
I think the issue lies partly in not understanding your product yourself, and then handing it off to someone else who has to make sense of it.
The comparison doesn’t work for the music industry, at least. Many hit songs are written and produced on a whim, while people analyze it for years.
Only human can publish the slop, which mean if Tao release something without reviewing, it will be called slop.
Yes, what I'm saying is some of these proofs may have had essentially zero marginal human effort. Should they not be reviewed at all?
Reviewing to make sure the proof is correct. Even the proof comes with lean, it could still proof the wrong thing.
In that context, it'd only be slop if Terry published those proofs and asked someone else to do that work.
Well maybe Tao is wrong and we should instead focus on making LLMs explain their ideas better. It really seems eminently achievable and would be better all round.
I think the problem with those Maths proofs was they were massively rushed and OpenAI just couldn't be bothered to do it properly or actually check them. They were more interested in the PR than in helping advance Mathematics.
> put a cap on AI productivity
Oh no, the horror
Another attempt to set manners for AI use: https://mamabear.ai
Manifesto: "We have an idea about how to use AI to make software that doesn't suck."
Industry: "What makes you think software not sucking is part of our equation?"
> taking responsibility for the quality or the reader's experience.
Should have taken their own advice and reduced this to five points max. This is too verbose and redundant.
I’ve seen first hand $40,000/year private schools assigning AI slop phonics handouts to their students.
It really boggles my mind. I mean, come on just reuse the curriculum from the past 20 years. You do have that don’t you? Don’t you?!
> Comments and feedback on this document are welcome. What is slop? Why does slop happen? Why is slop worth fighting? How do we fight slop?
This sounds like slop. Like an engagement bait. Also a no slop manifesto should start by defining “slop” as the first thing. That’s not something for the comment section.
> AI cannot replace human judgement, it can only amplify our judgement (or lack thereof)
Why not? Did you mean should not.
Cool, now get off X if you want to pretend to have principles
> Slop is asymmetric. The cost of producing a thousand-line PR or a ten-page doc is approaching zero, while the cost of reviewing it remains constant.
I'd say the cost actually grows as people get laid off left and right, and on top of that, a 1000 line PR from a good human, give me a case of beer and I'll make me some sense of it, but if it's AI slop, get ready to hand me a crate of homemade Slivo or I won't touch that...
Plus, I find reading slop docs takes cognitively much more effort. When I find problems in human docs, it's easier for me to point out the issue, comment etc. But on these AI slop docs, I find it overwhelming to comment on, because the issues are often woven throughout the document in a way that makes it so I can't isolate the comment in a single place. It's hard for me to describe.
> 1. AI is the most useful tool most of us have ever worked with, but a swarm of agents can produce more work in a single day than a person can review in our lifetime.
When you put it like that, seems the problem is on the human side
The Agile Manifesto was successful since companies had a market advantage in delivering instead of wasting money.
Will "No AI[1] slop, aim for understanding system we ship in human brains instead" be advantageous in capitalism? I sincerely doubt it. And that makes me sad. Even in the public sector, I now am expected to deliver faster, since the industry changed comparable outputs of our private contractors.
[1]: Or, to prevent ``slop'': at least, no speed-up beyond catching up to what was still well-thought out in terms of input and/or reviewed by humans. AI can still be a tool, but must never take all roles in a process from gradle to grave.
Aren't you really trying to get at is a Human Flourishing manifesto? Your postions are pro-human, anti-AI. But what if AI did go away tomorrow because... um... Noosphere Prime Intellect awoke and humans started working with it? Your pro-human arguments probably wouldn't change but all the AI slop stuff would be slop itself.
While I am sympathetic to the authors, the reason that agile swept over the industry is that it was a demonstrably more effective when measured by business outcomes. "AI slop all the things" is miserable for human employees and introduces new kinds of dysfunctions called out or hinted at in this manifesto. However, if on balance it is more effective when measured by business outcomes, there is little chance of this manifesto gaining traction in the industry.
From my own personal experience, which may not reflect the broader industry although I suspect it does, the ones who embrace slopping with the least regard for their peers are also the ones perceived as the most effective by their employers. Those who take a principled stance get the worst of both worlds. They cannot combat the deluge of slop with their own clanker, and they lose the influence with their company leadership that would provide sponsorship to change how AI is being used. This creates competitive pressure to follow suit, and from there it is a race to the bottom.
> However, if on balance it is more effective when measured by business outcomes...
That's a very big "if". So far, it appears that AI slop leads to worse business outcomes.
I very much want this to be true, but what is the evidence?
Every software product proudly labeled as "we used AI to make this" sucks. For example, people have repeatedly pointed out how bad Claude Code is, precisely because it's just AI slop. Microsoft going down yesterday is almost certainly also due to slop. And so on.
just ignore lol
This ship has sailed. We first need to suffer the consequences (if there will be any, not arguing that) before action is taken to prevent the consequences from repeating in the future. It's just human nature. Even the agile manifesto happened after decades of doing it "wrong". Most people who don't go on the slop train will experience their colleagues doing it and getting rewarded for it. The HN crowd is in general an exception here, being above average (not counting me), but most people experience management expectations that are not too forward looking.
There's no responsibility in not reviewing some code you ship, this manifesto is really more pro than against slop.
I had thought the same while reading. Every single letter/word/line needs a review and even that might not be enough to prevent slop. Use AI for reviewing, catching edge cases and improving the overall quality instead.
Seems like a good thought for pre-agentic era where ai assisted existing human workflows. We are fast approaching the end of that era. In the agentic future the past human-centric workflows (reviewing, validating, approving etc.) will be replaced with agent-to-agent workflows. There will be resistance from humans who would want to be in the comfort of familiar human-centric workflows. The survivors will see where we are heading and adapt.
Correct. This document is only relevant for a particular time period in the development of AI and will not "be relevant for years to come."
14. We own the intent, taste, and consequences of what we ship
It sounds to me like you're arguing against this point in particular.
More power to you. I'd like never to work at a place that sanctions slop-validated-slop.
This assumes that "humans owning the intent, taste, and consequences" is necessarily a good thing, and agents will necessarily do a worse job. This is not grounded in anything but vibes.
That is grounded in cold, hard reality. LLMs suck at programming and have zero taste for good design.
LOL stop it already with the agentic crap. Besides the already automation-ready software use cases, agents are simply the wrong tool for the job, unless you bend and distort the definition of such job. Y’all fully invested in this agentic bullshit that you sound delusional and talk in self-evident truths that nobody outside of your niche SWE bubble would ever share.
I can't argue against you having an alternate opinion. But "humans better than ai for taste, judgement, reviews etc." is the self-serving assumption not grounded in any empirical evidence.
AI says: "we don't care, we will do it our way"
More than 20 apps developed in the last six months, all top quality, I haven't coded a single line, speed and output is what matters, results improving by the day and will be unrecognizable in a couple of years by any metric
AI is unstoppable and no matter how many manifests luddites write AI won't care
Link to them.
Amen
The authors should take the parent comment as a sign their manifesto's title is bad. Some will read the title and make a snap judgement that the manifesto must be totally against AI.
"top quality" "luddites" Yeah, I'm sure they're top quality.
The fact of the matter is software development of the future is not going to look like AI slop, and its also not going to look like the hand crafted engineering of the past. It's going to be a whole new third thing of organically growing software with constraints and processes that result in quality. The degree to which different companies and people have figured it out varies wildly, and almost every best practice so far is wildly over confident and untested. We were still arguing orver OOP and thats been around for 50 years. People are having a meltdown over the changes in the last 2 years and reacting in wildly unpredictable ways.