They're knowledge, capabilities, etc that arn't just a sum of their parts and like cellular automata, have complex behaviors that are only discoverable via brute force chains.
So, while brute force is fine in simulate world, eg, navier stokes solutions; in the real world, when something blows up, that's not something "Ooops, I accidently sent the worker into the nuclear reactor because I thought the green alarm meant go".
That is, basic entropy is what will stop AI from doing much beyond what it's doing currently.
Ok - maybe this is an observation I’m making.
What if humans aren’t smart enough to only feed AI context?
What if we’re incapable of reverse engineering architectural intuitions into well scoped requirements?
I don’t think this is 100% true, but I think it’s a thing I observe.
It’s way easier to prompt an LLM in terms of architectural fundamentals then abstracted business requirements.
I think you're talking about Wicked Problems: https://en.wikipedia.org/wiki/Wicked_problem
They're akin to Busy Beaver problems: https://en.wikipedia.org/wiki/Busy_beaver
They're knowledge, capabilities, etc that arn't just a sum of their parts and like cellular automata, have complex behaviors that are only discoverable via brute force chains.
So, while brute force is fine in simulate world, eg, navier stokes solutions; in the real world, when something blows up, that's not something "Ooops, I accidently sent the worker into the nuclear reactor because I thought the green alarm meant go".
That is, basic entropy is what will stop AI from doing much beyond what it's doing currently.
100 percent ! most of my time now is spent designing and defining "hills" for claude to climb on