The AI debate has taken a rather dramatic turn lately.
Some of the people who know the technology better than most of us are warning that we may be heading towards something genuinely dangerous. AI could eventually become capable of improving itself, escaping our control and, in the worst case, becoming an existential threat to humanity.
Others argue that this is science fiction dressed up as science. AI, they say, is ultimately just statistics. It predicts the next token based on enormous numbers of vectors and probabilities. That is fundamentally different from the way the human brain works, so there is no reason to believe it could ever become truly intelligent, let alone superintelligent.
I find both positions a little uncomfortable. Not because I think they are equally likely. I simply don’t think we know. And that is perhaps the most important point that gets lost in the current debate.
We don’t actually understand intelligence
The argument that AI cannot become genuinely intelligent because it doesn’t work like the human brain sounds convincing until you think about it for a moment.
We don’t fully understand how the human brain produces intelligence either.
We know an enormous amount about neurons, synapses, brain structures and electrical and chemical signalling. But we don’t have a complete explanation for how billions of these relatively simple components turn into reasoning, abstraction, creativity, consciousness and what we loosely call intelligence.
So saying that a machine cannot become intelligent because it doesn’t work like the brain is a little like saying an aeroplane cannot fly because it doesn’t flap its wings. The aeroplane doesn’t need to reproduce the mechanism used by a bird. It needs to achieve the relevant outcome through another mechanism. And perhaps intelligence is the same.
Maybe the particular architecture of the human brain is essential. Maybe consciousness is essential. Maybe embodiment is essential. Maybe there are things about biological intelligence that cannot be reproduced computationally.. I genuinely don’t know.
And neither, as far as I can tell, does anyone else.
That doesn’t mean that AI will become superintelligent. It means that confidently claiming that it cannot is a much bigger statement than it first appears.
The probability problem
This is also where I get slightly uncomfortable when I hear people assigning probabilities to human extinction from AI.. 10%.. 20%.. 50%.. Sometimes even higher.
Those numbers look scientific. But what exactly are they probabilities of?
We don’t have a sufficiently good model of intelligence. We don’t know whether recursive self-improvement is possible. We don’t know whether it would result in a meaningful intelligence explosion. We don’t know whether such a system could be controlled. And we certainly don’t know how such a system would interact with the physical and social world. So where does the number come from?
It may be a useful expression of someone’s subjective belief. It may help communicate that they think the risk is serious. But it isn’t the same thing as having a well-founded probability model.
I am perfectly happy to say:
“I think there is a significant possibility that this could happen.”
I am much less comfortable when that gets translated into a number with two decimal places.. We simply don’t know enough to justify that kind of precision.
But uncertainty is not an argument for doing nothing
This is where I differ from people who use uncertainty as an argument for dismissing the whole thing.
Suppose there is a genuine possibility that we will eventually create a system capable of autonomously improving its own capabilities.. We don’t know whether that is possible.. Fine.. Let’s find out.
Because there are two rather spectacular outcomes:
If it turns out to be possible, we have a serious problem to solve. And we should probably solve it before the first system capable of doing it is sitting in a data centre somewhere. We would need to understand how to constrain it, how to contain it, how to prevent uncontrolled replication, how to limit its ability to act, and how to make sure that the system remains aligned with what we actually want. That sounds like a fairly good reason to take AI safety seriously today.
But there is another possibility.
Perhaps it isn’t possible. Perhaps there is some fundamental limitation in current AI architectures. Perhaps there is something about biological intelligence that we haven’t understood yet. Perhaps consciousness matters. Perhaps embodiment matters. Perhaps there is a computational barrier we haven’t discovered. If we establish that, we will have learned something extraordinary about intelligence and the human brain. So either way, there is a pretty good scientific question sitting here.
Which brings me back to regulation
I am not convinced that regulation is the answer to the fundamental problem. Regulation can certainly be useful. We regulate technologies all the time, and sensible regulation can reduce risks. But imagine that the really bad scenario is true. Imagine that it eventually becomes possible to create an AI capable of improving itself beyond our ability to control it.
What happens if the technology also becomes cheap and widely available?
You can regulate OpenAI.. You can regulate Anthropic.. You can regulate Google.. You can regulate governments.. But can you regulate the underlying knowledge once it exists?
This is where the nuclear weapons analogy becomes interesting.
Nuclear weapons are extraordinarily dangerous, but there is a very effective barrier between wanting one and building one. The technology requires enormous resources, specialised materials, sophisticated infrastructure and a lot of expertise. It is difficult and expensive.
If building a nuclear weapon were equivalent to downloading some software and buying a few graphics cards, I suspect our world would look rather different.. AI could potentially move in precisely that direction.. The technology gets better.. The hardware gets cheaper.. The models become smaller.. The knowledge spreads.
Eventually, the capability might no longer belong exclusively to a handful of companies and governments. And then regulation becomes a much less reliable safeguard.
That doesn’t mean regulation is useless. It means regulation cannot be the only plan.
Find out before we have to find out
This is probably the part of the debate I find most interesting.
We are currently arguing about what AI might become without really knowing what intelligence is. Perhaps we should spend considerably more effort trying to answer that question. Can a system learn to improve the process that created it? Can it autonomously discover better architectures? Can it improve its own training methods? Can it make meaningful advances in AI research without human guidance? Can those improvements compound? Can we reliably stop the process?
Those are experiments that can give us evidence.
If we discover that there is a hard ceiling, excellent. We have learned something fundamental about intelligence. Alternatively, if we discover that there isn’t, then we have learned something equally fundamental, but considerably more alarming. Either result would be preferable to arguing endlessly about whether AI is “just statistics”. Because maybe it is.
But our brains are also physical systems processing electrical and chemical signals according to enormously complicated patterns. We don’t consider human intelligence less real because we don’t understand every calculation taking place inside our heads.
We shouldn’t assume that intelligence requires our particular biological implementation simply because it is the only implementation we currently understand.
At the same time, we shouldn’t assume that a sufficiently large language model will inevitably become a superintelligence.
Because, we don’t know.
And I think that is the honest starting point.. Not panic.. Not complacency.
Find out.
And if the answer turns out to be “yes”, we had better have started thinking about the safeguards before we get there.
Leave a Reply