>>2911068It is a real worry and not because some people in twitter are saying this, but because people actually researching this are. Yes ofcourse this is not an exact science, its partly philosophy, partly computer science, but it is nonetheless the best tools we have to investigate the question.
The marketing is ofocurse real, techbros are selling this as a gamble:
X % chance of catastrophic circumstances. And if we avoid this we will have incredible prosperity. Since this is an arms race with china, the us government wont ever put any safety checks on this, no matter how much AI safety people try to pressure them.China is gonna win eventually btw, they are able to create amazing progress with a small fraction of the compute that OpenAI or anthropic has.
The main problem of the AI safety discourse is that non academics are obsessed about predicting chances of extinction. You dont need to put numbers on it, it is just reasonable to expect. Mind you we already had breaches of containment in sandboxed models (HuggingFace incident). Despite this the honest answer is we dont know, models have been steadily improving but there might eventually be a plateau, necessitating a whole new approach to actually get us to 'superintellegence'. However already today's capability is disruptive, for instance, frontier math research is now accelerated with AI.