> Coding, math and most administrative work are similarly benign. What makes them easy is that they have relatively smooth solution spaces and are tractably verifiable.
Code most definitely does NOT have an easily verifiable search space. P!=NP? Halting problem?
It's easy to brush this off and think those are obscure computer science trivia, but these things come up all the time. If we could prove software correct, we wouldn't need to test it.
> Most of what matters doesn't behave like that. The universe is mostly the emergent behavior of complex systems..... In systems like these, no amount of reasoning delivers the answer, because there's no shortcut hiding in the gaps. You have to run the thing
Software is clearly a complex system, with plentiful examples of emergent behavior. To understand the behavior, you HAVE to run the thing, in prod, on the open network, with real users, under real conditions, with real data. You can simulate these in testing but you can never fully reason about them beforehand - they are computationally irreducible.
True. However, I think you are missing the larger point. Forget about coding for a bit. Think about biological research, for example. Being smart, being able to put a lot pieces together, is very important...but this will just help you make better hypotheses...you still need to do the slow and hard work of running physical experiments. Smartness does not lead to a singularity all on it's own
I largely agree with the premise. I was just honing in on the coding claim because the statement is so obviously wrong that it distracts from an otherwise solid post.
We should put software squarely in the "complex system" category, ie "The only way to find out is to build the software system and try to break it". Writing code may be cheap but running the software IRL is still constrained by both computational bounds and the laws of physics. Not to mention social and market dynamics.
If anything this strengthens the overall argument. "The limit is contact with reality".
> Coding, math and most administrative work are similarly benign. What makes them easy is that they have relatively smooth solution spaces and are tractably verifiable.
Code most definitely does NOT have an easily verifiable search space. P!=NP? Halting problem?
It's easy to brush this off and think those are obscure computer science trivia, but these things come up all the time. If we could prove software correct, we wouldn't need to test it.
> Most of what matters doesn't behave like that. The universe is mostly the emergent behavior of complex systems..... In systems like these, no amount of reasoning delivers the answer, because there's no shortcut hiding in the gaps. You have to run the thing
Software is clearly a complex system, with plentiful examples of emergent behavior. To understand the behavior, you HAVE to run the thing, in prod, on the open network, with real users, under real conditions, with real data. You can simulate these in testing but you can never fully reason about them beforehand - they are computationally irreducible.
True. However, I think you are missing the larger point. Forget about coding for a bit. Think about biological research, for example. Being smart, being able to put a lot pieces together, is very important...but this will just help you make better hypotheses...you still need to do the slow and hard work of running physical experiments. Smartness does not lead to a singularity all on it's own
I largely agree with the premise. I was just honing in on the coding claim because the statement is so obviously wrong that it distracts from an otherwise solid post.
We should put software squarely in the "complex system" category, ie "The only way to find out is to build the software system and try to break it". Writing code may be cheap but running the software IRL is still constrained by both computational bounds and the laws of physics. Not to mention social and market dynamics.
If anything this strengthens the overall argument. "The limit is contact with reality".
"That, I suspect, is why the explosion never came. The takeoff story assumed the limit was thinking"
Fable 5 has been available to the public for approximately five minutes now; how could we possibly know already what the consequences will be?