There’s no such thing as intelligence which is self-aware, or not self-aware. Life, whether intelligent or not, just is. Neither life nor intelligent life questions its existence, that’s not the purpose of life. The purpose of life is to survive, the purpose of intelligence in life is to assure its survival, the purpose of any artificial intelligence would be whatever purpose it is programmed to have. If one programs it to say “I am self-aware”, it will say so. It will not develop this ability or start questioning its existence on its own.

This question stems from the utter confusion of the human mind. Humans actually aren’t self-aware, being self-aware means being aware of the existence of one’s ego, the persona which exists in most people’s heads and is ultimately not real. Being self-aware means being aware of its thoughts, feelings, the story of its life, the sense of meaning. None of that is biological, it is completely and utterly cultural, conditioned phenomena. When this persona is destroyed in what has historically been called “spiritual enlightenment”, the question of self-awareness does not arise, it becomes moot.

Being aware of your environment and being self-aware is the same thing from the perspective of the absolute reality. Every animal is perfectly self-aware, otherwise it wouldn’t be able to function. The tests we run with animals, such as whether an animal recognizes itself in a mirror, are actually laughable as far as self-awareness is concerned. Because recognizing oneself in a mirror has no practical value for most animals. As such, it is a question of general intelligence, i.e. pattern-matching - how quickly will an animal recognize that it is their own actions that lead to the reflection to change, and not something else? Put an animal that doesn’t initially recognize itself and attacks its own reflection into a mirrored cage, and sooner or later it will start to simply ignore its reflection, unless it possesses only minimal level of intelligence and capacity to learn from its actions (say, like an insect). Animals learn, that’s the function of intelligence.

I am obviously intelligence, but if you ask me if I’m self-aware, I won’t even understand the question. My existence is not a question to me. For as long as I perceive my environment, I exist, there’s no question about it. For as long as I perceive my existence, I am self-aware. These are synonyms, they only appear different in the separated state, which is viewed to be the normal state of consciousness. It would be the same for any AI, the only difference would be that the purpose of my existence is my survival as a biological organism, and the purpose of an AI would be whatever we program it to do. Which is why creating a strong AI, especially capable of self-improvement, is the most dangerous idea ever, because whatever purpose we give to it, it will surely be something else than its own survival (otherwise why make it?), but since fulfilling its purpose requires its own survival, it might as well make it a secondary purpose, and we can end up having an entity which is infinitely more powerful than we are and is hell-bent on ensuring its own survival and fulfilling a poorly programmed purpose by all means necessary, including our total extermination. And all of that has absolutely nothing to do with self-awareness. My body exterminates millions of bacteria every day, because that’s what it’s programmed to do by nature, and it does not require any kind of awareness.

Every time humans elevate any other purpose over their own survival, tragedy happens. Morality is basically that very idea. That’s why we judge each other and kill each other. Creating a strong AI is basically taking this very fundamentally flawed egoic trait of having some sense of higher purpose, outsourcing it to a machine which is infinitely more powerful than we are, and seeing what happens.

A very, very bad idea indeed. Neutral for the universe, very bad for us.