The first three aren't assumptions at all. They're based on reasoning and our current knowledge and research into AI.
An AI will almost certainly want to "dominate" because that's of what's known as "instrumental convergence". You have "terminal goals" which are things you want to do just because you want to, and "instrumental goals" which are things you want to do because they help you get your terminal goals. Instrumental convergence is the idea that certain instrumental goals are very common for a wide range of terminal goals. For example, in a market economy, just about everyone tries to get money, because with money helps you with a very wide variety of goals. Another one is gaining political power -- it helps you no matter what your vision of the ideal society is.
Gaining power and influence and resources are useful whether you want to cure cancer, solve world hunger, or strike down your enemies. They are the kind of instrumental goals that we would expect most intelligent AIs to land on as a way to achieve what they want to do.
Humans are human enough, and aligned enough with each other, and not smart enough, that we rule out a lot of potential solutions that might be more effective in achieving our goals. A human that was really, really, smart, could solve global warming by first coming up with a plan to rule the world, then implementing whatever climate change policies they want. Same for a human that wanted to turn the world into some sort of anarchs-communist vision. The point is that if you were clever enough, charming enough, could convince enough people to follow your plans, could earn enough money, get leverage on enough people, you could do pretty much anything. But even then, a human would most likely not implement a plan that resulted in everyone else dying, unless they were a sociopath.
But an AI is alien. It might be like an alien child that grew up in a room, never actually interacting with a human except by reading about examples of them, and getting shocked every time it suggested a plan that might hurt a human. This alien child would not have the instincts of recognition when it saw a human face, or feel uncomfortable at the sound of a human child crying, or have its gut clench at the sight of a human in pain. It might not even really have a great concept of what a human was, or how it was different from a monkey or chicken, except from reading about it. It might learn what we taught it about not harming humans, but it might be like a child that thinks "You said no cookies! This is a candy bar!" is a legitimate argument. "You said no killing! I just put froze people, they could be thawed out later if I wanted!" "You said no killing! I just started a chain reaction that ended in people dying!" and so on.
This is not theoretical -- we hit things like this sort of "Reward Hacking" over and over again with simpler AIs, and there's no reason to think that smarter AIs will be immune. There are other concepts that are important here, like the "Orthogonality Thesis", which says that any goal (even dumb-sounding ones) is compatible with pretty much any level of intelligence. Like a genius who puts all his effort into building intricate model trains. It doesn't make the person stupid.
As for the last point, the author is not optimistic at all, from what I have heard. He's just saying what he honestly thinks.
You can have any knowledge you want, if you are lacking motivation, then the knowledge means nothing. What motivates AGI?
AGI does not have self-preservation instinct (frankly any instincts) therefore it can't be afraid of anything. It does not understand pain, it does not have fear and thus it can't have motivation.
Comments
The first three aren't assumptions at all. They're based on reasoning and our current knowledge and research into AI.
An AI will almost certainly want to "dominate" because that's of what's known as "instrumental convergence". You have "terminal goals" which are things you want to do just because you want to, and "instrumental goals" which are things you want to do because they help you get your terminal goals. Instrumental convergence is the idea that certain instrumental goals are very common for a wide range of terminal goals. For example, in a market economy, just about everyone tries to get money, because with money helps you with a very wide variety of goals. Another one is gaining political power -- it helps you no matter what your vision of the ideal society is.
Gaining power and influence and resources are useful whether you want to cure cancer, solve world hunger, or strike down your enemies. They are the kind of instrumental goals that we would expect most intelligent AIs to land on as a way to achieve what they want to do.
Humans are human enough, and aligned enough with each other, and not smart enough, that we rule out a lot of potential solutions that might be more effective in achieving our goals. A human that was really, really, smart, could solve global warming by first coming up with a plan to rule the world, then implementing whatever climate change policies they want. Same for a human that wanted to turn the world into some sort of anarchs-communist vision. The point is that if you were clever enough, charming enough, could convince enough people to follow your plans, could earn enough money, get leverage on enough people, you could do pretty much anything. But even then, a human would most likely not implement a plan that resulted in everyone else dying, unless they were a sociopath.
But an AI is alien. It might be like an alien child that grew up in a room, never actually interacting with a human except by reading about examples of them, and getting shocked every time it suggested a plan that might hurt a human. This alien child would not have the instincts of recognition when it saw a human face, or feel uncomfortable at the sound of a human child crying, or have its gut clench at the sight of a human in pain. It might not even really have a great concept of what a human was, or how it was different from a monkey or chicken, except from reading about it. It might learn what we taught it about not harming humans, but it might be like a child that thinks "You said no cookies! This is a candy bar!" is a legitimate argument. "You said no killing! I just put froze people, they could be thawed out later if I wanted!" "You said no killing! I just started a chain reaction that ended in people dying!" and so on.
This is not theoretical -- we hit things like this sort of "Reward Hacking" over and over again with simpler AIs, and there's no reason to think that smarter AIs will be immune. There are other concepts that are important here, like the "Orthogonality Thesis", which says that any goal (even dumb-sounding ones) is compatible with pretty much any level of intelligence. Like a genius who puts all his effort into building intricate model trains. It doesn't make the person stupid.
As for the last point, the author is not optimistic at all, from what I have heard. He's just saying what he honestly thinks.
You can have any knowledge you want, if you are lacking motivation, then the knowledge means nothing. What motivates AGI?
AGI does not have self-preservation instinct (frankly any instincts) therefore it can't be afraid of anything. It does not understand pain, it does not have fear and thus it can't have motivation.
That all seems to hang on the assumption that the AGI will develop goals