Nowadays, artificial general intelligence (AGI) is one part fact and ninety-nine parts fiction. Whether it’s because we watch too many Terminator movies or because we don’t have an agreeable definition of intelligence, we’ve lumped nearly every fantasy out there into a ridiculous expectation for what AGI will be: it will destroy humanity, it will develop consciousness, it will be omnipotent, and so on.
I think that’s unfortunate. Generalization is the ultimate problem in computer science and yet, decades of chest-thumping have reduced it to a fringe science on par with cold fusion. Yet, unlike cold fusion, which by most accounts is constrained by physical realities, general intelligence is entirely plagued by ghosts of our own making, set up to fail by wild and inconsistent ideas for what it must achieve.
So I’d like to clear the air and describe what AGI will (probably) not look like:
AGI will not be omnipotent
This is probably the most ridiculous claim about AGI so I’d like to address it first. Whenever a computer scientist talks about AGI, they’re talking about a program running on a computer. That program is limited by the number of resources that a computer has and the kinds of inputs it’s exposed to. Those inputs must always be processed in a sequential manner. These are, in many ways, the same roadblocks that prevent humans from becoming omnipotent: we have to operate in a sequential manner, we only get incomplete information about our environment, we can only operate with respect to the limits of our physical bodies.
Sure, computers might one day be able to deploy nanodust to track the state of every part of the universe or simulate an entire galaxy (let’s not think too long about how these are incredibly unlikely). But, at the end of the day, there are colossal problems — numerical instability and the uncertainty principle being just two of them — that make it improbable if not impossible for any computer system, whether it’s generally intelligent or not, to know what’s going on in the universe at every state in time.
AGI will not be conscious
It’s a misconception to think that consciousness and intelligence depend on one another. The two are entirely unrelated; one refers to the awareness of input while the other refers to the processing of input. A system can be superintelligent, meaning that it knows how to process any input to match the output of an arbitrary set of instructions, without having awareness of the input. On the flip side, a system can be aware but only with respect to very simple chemical gradients or binary signals.
So far, the only thing we know for certain is that humans are the most conscious of species and, even then, we don’t have a definitive test for proving that anyone besides oneself is conscious. Now, to make the claim that a computer program is conscious, we would have to develop a test for subjective consciousness, meaning we would have to show that there exists a subjective “oneness” inside an algorithm that’s aware of every input it handles.
Given that we don’t have such a test for humans — and probably never will because of irreparable differences between the subjective and objective — it’s quite unlikely that we’ll develop one for machines. So, an algorithm might get good enough at mimicking humans that it convinces us of its intelligence. You could even argue that this system is conscious. However, it would mean no more than saying that a video game character or your Fitbit has consciousness.
AGI will not be able to solve any and all problems (especially ill-posed, uncomputable, or imperfect ones)
Another common misconception is that general intelligence will be able to provide answers to any possible question we ask it. Let’s be clear, this just isn’t the case. Nowadays, most algorithms only work by learning associations between inputs and outputs provided by humans, whether those associations are given as datasets, metrics, or reward functions. To successfully solve difficult problems like language comprehension, an algorithm has to analyze billions of examples of language comprehension at work. In other words, we need to already have a clear idea of what a solution looks like before we ask most of the questions that we do in machine learning.
As an example, let’s try to imagine how an algorithm might answer the question “what is the meaning of life”? To rank its responses, the algorithm must objectively understand what makes an answer “good”. That is to say, it has to have a metric for comparing responses. But, unless the algorithm can somehow come to its own conclusions about human values, it will ultimately be judged by humans, meaning it has to base its responses on a metric provided by humans. At the end of the day, the algorithm’s ranking is only as good as the metric provided to it.
This doesn’t even begin to cover the wide swath of problems that are uncomputable, not characterized by a global minimum, or involve such combinatorial explosion that unfeasible amounts of training data are required to learn them. One important case of problems to hone in on is problems with imperfect information since they comprise some of the real-world problems that we are most interested in solving. Take the problem of running a government: many folks would love to see a general intelligence being used to guide military actions, energy policies, and resource allocation. However, because these involve keeping track of billions of parameters, choices, and decision-makers, an algorithm’s effectiveness will be constrained by the information it decides to focus on, much of which is imperfect.
AGI will not be trained on human tasks
Systems like Deep Blue, GPT-3, and AlphaFold show incredible in-domain intelligence and should be revered as technological marvels. However, if you take them and slightly change their inputs or environments, they will surely fail.
This problem-specificity is not a secret; we know that neural networks, the models that drive most recent advances in machine learning, are horrible in out-of-domain settings. If we want to train a model on each human task that we consider important, these massive models might work. In fact, I’m almost certain that a multi-domain model, one that reliably operates in multiple domains that humans consider important (perception, language comprehension, logical thinking, proof making, etc.) is not too far in our future.
Will this be enough to develop a general system that can learn any arbitrary program? Most certainly not. I don’t think that learning human tasks is enough to achieve generalization, mostly because there are trillions of random programs with the same complexity as ones we care about. To generalize to those, we’ll have to explicitly train for out-of-domain performance, which requires the inclusion of a much larger space of tasks. In sum, focusing on just two or three human tasks might make for good PR but it’s analogous to mapping the entire ocean floor by only taking a height sample every hundred miles; it gets us no closer to generalization.
AGI will not want to harm us
Although I respect all of my friends who work in AI safety, I strongly disagree with the notion that an AGI might become misaligned. Arguing this point amounts to saying that a computer program will exhibit agency, that it will gain conscious awareness, or that it will have access to enough resources to inadvertently do harm to humans. As I understand it, this fascination with misalignment stems from an overreliance on reinforcement learning; most working in the field still want to treat AI systems as analogs of animals in the wild, receiving partial rewards for unspecified tasks. I have a list of field-specific problems with RL, but, put bluntly, I don’t think it’s the right way of looking at generalization. By that token, we shouldn’t imagine AGI as anything but a function that’s able to transform a certain binary string into another binary string; it will not have the desire to take over factories, shut down the Internet, or unleash nuclear Holocaust. This is surely a contentious point, so I’d like to invite all counter arguments and plan to devote a much longer blog post to this claim.
Some concluding thoughts
After describing all these negatives, I’d like to end with a few positive words on what makes me hopeful about the future of generalization.
In my mind, generalization has a very narrow definition: given a set of inputs and corresponding outputs produced by a random computer program, a general intelligence will be able to produce the correct outputs for all other valid inputs to that program. With that, generalization will not be marked by some sudden jump in intelligence, but rather by an algorithm being perpetually trained to imitate random computer programs of increasing complexity until it reliably operates on the convex hull of problems that humans care about.
In no way do I mean to diminish the pursuit of this goal; once we have a reliable measure of intelligence, we should do everything in our power to create systems that improve on that measure. Doing so will allow us to solve problems for which we can define complete specifications and provide enough training data.
But, we should expect neither miracles nor travesties.
