Stephen Downes

This is the season for manifestos from tech CEOs, I guess, and the current offering is from Anthropic's Dario Amodei and the argument is that frontier AI developers should slow the pace of that development in order to ensure oversight and guardrails are in place. The motivating incident is the OpenAI hack of Hugging Face, and Amodei warns, "a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage." For example, "it's my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)." The core question, of course, is whether the proposed measures address the perceived risk. It's not clear to me they do. As Jason Hiner argues in The Deep View, "the company said a unilateral pause by one lab wouldn't actually achieve much. 'It would change who the front-runner is, but it would not create the wider deliberative process that is currently missing.'"