I hope my comment is on topic. If not, apologies. But I think I have some skin in the game here.
I could be the perfect example of someone overreaching. My coding skills are limited, but I’m also not stupid, and I don’t release anything I build without having it reviewed by an expert.
That said, IMHO, humans were already very good at producing garbage before AI.
I think the real distinction is not expert vs non-expert. It is whether the person using the tool has enough judgment and feedback loops to catch bad output.
In my field, AI lets me do much more than I could before. It also lets me explore areas I would normally not touch alone. But I still have my business partner, who is the stronger engineer, check everything I do before releasing anything.
That review step is very important. Without it, I would probably sometimes ship polished garbage.
I’m also not sure quality is as objective as people like to pretend it is. Some things can be tested. But even there, testing only reduces risk. It does not make software perfect. Vulnerabilities are often found years later, in code written and reviewed by experts, and running in production.
So maybe the danger is not AI. The danger is removing the review systems and pretending the output is expert-level just because it seems to be working correctly.
>> The danger is removing the review systems and pretending the output is expert-level just because it seems to be working correctly.
This is what is happening. In the rush to adopt AI many are abandoning sane engineering principles/processes lest one gets left behind. Only to rediscover them on failure, hoping it is not catastrophic.
In financial services AI isn't theoretical. We recently build NL2SQL agentic systems for wealth management, systems that work in production are ones where domain experts (compliance, risk, portfolio) collaborated to define the semantic layer first. Solo + AI builds skip that step and it shows immediately under edge cases. SR 11-7 model risk governance exists partly because this failure mode already happened before LLMs existed.
I hope my comment is on topic. If not, apologies. But I think I have some skin in the game here.
I could be the perfect example of someone overreaching. My coding skills are limited, but I’m also not stupid, and I don’t release anything I build without having it reviewed by an expert.
That said, IMHO, humans were already very good at producing garbage before AI.
I think the real distinction is not expert vs non-expert. It is whether the person using the tool has enough judgment and feedback loops to catch bad output.
In my field, AI lets me do much more than I could before. It also lets me explore areas I would normally not touch alone. But I still have my business partner, who is the stronger engineer, check everything I do before releasing anything.
That review step is very important. Without it, I would probably sometimes ship polished garbage.
I’m also not sure quality is as objective as people like to pretend it is. Some things can be tested. But even there, testing only reduces risk. It does not make software perfect. Vulnerabilities are often found years later, in code written and reviewed by experts, and running in production.
So maybe the danger is not AI. The danger is removing the review systems and pretending the output is expert-level just because it seems to be working correctly.
>> The danger is removing the review systems and pretending the output is expert-level just because it seems to be working correctly.
This is what is happening. In the rush to adopt AI many are abandoning sane engineering principles/processes lest one gets left behind. Only to rediscover them on failure, hoping it is not catastrophic.
In financial services AI isn't theoretical. We recently build NL2SQL agentic systems for wealth management, systems that work in production are ones where domain experts (compliance, risk, portfolio) collaborated to define the semantic layer first. Solo + AI builds skip that step and it shows immediately under edge cases. SR 11-7 model risk governance exists partly because this failure mode already happened before LLMs existed.
[flagged]