I still don't understand how to interpret these conclusions. This is all based on a report from a 'third party' 'evaluating' Anthropic, but it's mostly all the same EA people on both sides of it saying the sky is falling. These are the same people that did the junk investigation into the 'HF incident'. How do we even know that METR is being honest, or even objective? How do we know AECI is a good enough benchmark to make this kind of claim, or that the benchmarks that make it up are any good on their own? A lot of Fable 5.2 seems to be a regression in terms of cost and a potential sign of diminishing returns on compute scaling, so to have this come out at the same time claiming we'll see the explosion next year (according to the original article, basically starting in January) is curious.
The authors said we'd be halfway there by now, but most of the political turmoil stuff hasn't followed because the capabilities have not made it as far enough into most workplaces. The public's view of these systems' capabilities and reliability is closer to GPT-3 than FOOM.
I'm not sure that they're wrong, because they have followed AI capabilities somewhat closely. It's the downstream effects that seem to be all over the place by comparison. I don't know what to believe, honestly.
For what it's worth I don't think there can be an explosion as the rate of AI doing stuff is limited by compute which is limited by the power supply which is maxed out. I think it will be more rapid intelligence progress similar to what we've been seeing.
This is concerning. It's clear we need some level of societal backlash to halt and reverse this.
Society has no true frame of reference for such exponential growth. I see a parallel to COVID. It was not a big deal on March 1, by April 1 society was halted. This is worse.
Its a reasonable opinion to hold people working on such breakneck AI progress to the standard we would hold if your friend told you they were making a nuclear warhead in their basement. It should be stopped.
I can't see a backlash halting it. Maybe a plan to deal with it might be the thing. Smart computers along the lines of Claude aren't a threat like nukes or covid.
I'd really love to call this end-times millenarianism with no factual basis, but Kokotajlo et al. have had a much better track record at making predictions than I am, and I've always underestimated the rate of AI progress. This is much both much more reasonable and much more terrifying than I'd like to admit.
I think it's time for us jaded "p(doom) = ~0" types to be less dismissive of people like Kokotajlo.
And to acknowledge that our anti-doomer stance is partly motivated by the fact that 1) we want to believe AI will be net beneficial, and 2) many prominent doomers have seemed pretty nutty.
By itself it is already hard to think these things through, and it is similarly hard to take the conclusions seriously if you do. On top of that there are an unusually high number of highly distributed, cross-domain, and unusually large incentives at play here, that are unusually uniform in direction, making for an extraordinary headwind to sail against.
> But maybe that’s a sign that HN doesn’t buy these scenarios.
There is no HN consensus opinion.
I am of the opinion that this is going to cause massive economic and political upheaval in the next year or two on scales we are nowhere near ready to deal with and probably end quite badly (at least in the US) because we exist in a state of perpetual political near-paralysis.
I'm also of the opinion that even if I'm right there's nothing I can really do about it but watch it unfold.
My concerns aren't really the same as those of the EA crowd. I'm not worried about AI turning us into paperclips or escaping containment or whatever, I'm worried about much more mundane concerns like what does it look like when large amounts of people are no longer economically valuable and wealth inequality continues to skyrocket and this all happens way faster than we can manage any sort of controlled transition.
But that's just my concern... a lot of other people here are convinced this is all just normal technology and won't have any large societal impacts beyond maybe accelerating us into a brighter future.
I would like for them to be right and for me to be wrong.
I still don't understand how to interpret these conclusions. This is all based on a report from a 'third party' 'evaluating' Anthropic, but it's mostly all the same EA people on both sides of it saying the sky is falling. These are the same people that did the junk investigation into the 'HF incident'. How do we even know that METR is being honest, or even objective? How do we know AECI is a good enough benchmark to make this kind of claim, or that the benchmarks that make it up are any good on their own? A lot of Fable 5.2 seems to be a regression in terms of cost and a potential sign of diminishing returns on compute scaling, so to have this come out at the same time claiming we'll see the explosion next year (according to the original article, basically starting in January) is curious.
The authors said we'd be halfway there by now, but most of the political turmoil stuff hasn't followed because the capabilities have not made it as far enough into most workplaces. The public's view of these systems' capabilities and reliability is closer to GPT-3 than FOOM.
I'm not sure that they're wrong, because they have followed AI capabilities somewhat closely. It's the downstream effects that seem to be all over the place by comparison. I don't know what to believe, honestly.
Interesting to revisit the comments on the original AI 2027 submission:
https://news.ycombinator.com/item?id=43571851
The top thread in particular is an incredible time capsule. No matter which way you cut it, the world was different before Opus 5.5.
The whole "they are just token predictors" and can't think argument seems to have dropped out since the millenium prize work.
For what it's worth I don't think there can be an explosion as the rate of AI doing stuff is limited by compute which is limited by the power supply which is maxed out. I think it will be more rapid intelligence progress similar to what we've been seeing.
This is concerning. It's clear we need some level of societal backlash to halt and reverse this.
Society has no true frame of reference for such exponential growth. I see a parallel to COVID. It was not a big deal on March 1, by April 1 society was halted. This is worse.
Its a reasonable opinion to hold people working on such breakneck AI progress to the standard we would hold if your friend told you they were making a nuclear warhead in their basement. It should be stopped.
I can't see a backlash halting it. Maybe a plan to deal with it might be the thing. Smart computers along the lines of Claude aren't a threat like nukes or covid.
I'd really love to call this end-times millenarianism with no factual basis, but Kokotajlo et al. have had a much better track record at making predictions than I am, and I've always underestimated the rate of AI progress. This is much both much more reasonable and much more terrifying than I'd like to admit.
I think it's time for us jaded "p(doom) = ~0" types to be less dismissive of people like Kokotajlo.
And to acknowledge that our anti-doomer stance is partly motivated by the fact that 1) we want to believe AI will be net beneficial, and 2) many prominent doomers have seemed pretty nutty.
Seems like this should be bigger news. But maybe that’s a sign that HN doesn’t buy these scenarios. What is it?
By itself it is already hard to think these things through, and it is similarly hard to take the conclusions seriously if you do. On top of that there are an unusually high number of highly distributed, cross-domain, and unusually large incentives at play here, that are unusually uniform in direction, making for an extraordinary headwind to sail against.
> But maybe that’s a sign that HN doesn’t buy these scenarios.
There is no HN consensus opinion.
I am of the opinion that this is going to cause massive economic and political upheaval in the next year or two on scales we are nowhere near ready to deal with and probably end quite badly (at least in the US) because we exist in a state of perpetual political near-paralysis.
I'm also of the opinion that even if I'm right there's nothing I can really do about it but watch it unfold.
My concerns aren't really the same as those of the EA crowd. I'm not worried about AI turning us into paperclips or escaping containment or whatever, I'm worried about much more mundane concerns like what does it look like when large amounts of people are no longer economically valuable and wealth inequality continues to skyrocket and this all happens way faster than we can manage any sort of controlled transition.
But that's just my concern... a lot of other people here are convinced this is all just normal technology and won't have any large societal impacts beyond maybe accelerating us into a brighter future.
I would like for them to be right and for me to be wrong.
[dead]