> I also expect that nearly all new models would explain the behavior correctly if provided the output (or, equivalently, were able to run the code), but did not test this hypothesis either.
I'm not sure about that, considering that the output matches "intuition". Change the "rocket launching" text to something that doesn't imply that a countdown would make sense in context, and rename the class, and I'd expect the models to do considerably worse. So they shouldn't do better when provided the output, because in that case it isn't actually "new information".
> I also expect that nearly all new models would explain the behavior correctly if provided the output (or, equivalently, were able to run the code), but did not test this hypothesis either.
I'm not sure about that, considering that the output matches "intuition". Change the "rocket launching" text to something that doesn't imply that a countdown would make sense in context, and rename the class, and I'd expect the models to do considerably worse. So they shouldn't do better when provided the output, because in that case it isn't actually "new information".
"Isn’t that fun? :)"
Not really :)
I'm reminded of The Zen of Python: "explicit is better than implicit".
Having fun with programming doesn't have to be about writing good, useful or practical code.
True
[dead]