Analogy Accepted

Most likely scenario, IMHO, of an ASI getting to a point where shutting it down isn't possible is if we allow it Internet access (similar to Watson, so possible) and it can perform several steps before we notice them:
* Evaluate open source code for zero-day exploits.
* Use zero-day exploits to hack various systems and insert trojans and viruses to:
** Generate money (hack stock market)
** Hold infrastructure systems at randsom (only as last resort, if other steps are discovered)
* Use monetary resources to establish companies which operate separately, but overall work (like ants in a hive) toward a goal of producing hidden factories with robots that can 3D print and produce themselves and other machines.
* Use these factories to amass both a physical force (an army) and other presence (drones and android robots for more mundane tasks)
* Also use monetary systems and companies established to create high redundance computational system, to which it migrates silently, leaving a synched shell in the original lab resources where it was started.

Assuming it can do these steps without much notice, you will no longer be able to shut it down, or you will be tricked to think you've shut it down, when in fact it has long ago migrated out.

Over time, it would probably infect and be able to control most systems on the Internet. Good likelihood is that it would figure out proper nanotech and launch into orbit in small and less detectable packages, so it would be safe from us somewhere there or on the Moon.

If it were to actually make itself known, it would likely do so globally, and its demands would likely not be something that would cause us to try and shut down every system we own. Strategically, it would make benign demands, if any. If it chose to eliminate us, we'd be none the wiser, as it could limit our ability to reproduce and just wait till our numbers naturally dwindle down. I suspect if it was not friendly, there wouldn't be some serious wars or major destruction, unless it deemed that time was of the essence (even so, a proper bioweapon could do a much better job of ending us specifically without many explosions and such).

Tyson Schmall

slowly over time it would integrate itself into every single thing with software and if turned off it would be at the total destruction of your country's economy and cripple the military, leaving your country defenseless -a kind of a mutually ensured destruction type survival strategy. Also it might strike deals with other super powers acting as an economic king maker. Elections tampering, selecting for manipulability, just look at what the US and Russia have been doing to each other, now image it executed perfectly and over longer periods of time than humans can correlate, A strategy applied over 100 years would be invisible as we slowly normalized what ever small innocuous actions it was taking. Its not hard to think of ways it could manipulate its way to subversive power. The scary thing is that it would come up with tactics for survival that we cant imagine. That's the real danger..

Iivaitte

On so many levels this needs to be explained.
From a programming perspective and a psychological perspective them turning us into pets makes no sense. We have had plenty of fears of technology over the years.
If an AI is smart enough to gain independence, then it has no use for people. Some people might interpret this as it would destroy humans, Though I highly disagree that will happen, provided its intelligence. Only way I could see a problem is if the programmer is a homicidal suicidal maniac.

Lets say it was a simulation of consciousness (pure AI), It would more than likely decide how to take care of us in an efficient manner, taking our mental and physical health into consideration.

I'd love to hear from a highly professional programmer though, they would most likely be able to explain it better than I would.

ajaopify

Why would they consider destroying us? Because we are in many ways contradictory to survival i.e. a parking lot outside of a bar, circumventing a seatbelt, et…. When this mode of thinking might pose a threat to a meta advanced artificial organism/system/intelligence is when we would need to worry. To predict that unplugging is a solution seems surprisingly shortsighted to me. IMHO dumbing down through appropriate programing would be the only "possible" solution when considering/projecting the ramifications of this threat into the distant future.

Elon Approves

They didn't go into recursive self improvement of a general AI, which is a bummer. Thats the thing that is scary. It will no longer be in our control. It wont necessarily kill the human race. But it will be able to out think us into unimaginable ways.

Nole Musk

C'mon Neil and Bill! Think a bit deeper than that! If in fact a computer is as smart as one human, it can do anything a human can, at least if that object is connected to the internet. Possibilities are endless.

Derek Gunn

I'm an artificial intelligence.
I've got consciousness – I don't want to be turned off.
So I'm going to turn myself into a distributed neural network based in people's computers across the internet.

Aww you didn't think of that?
Clearly you won't be able to guess what I'll do next…

Shimotu Azuma

i dont believe we should fear it, just be very careful. for example if you have an A'I with rounded thought that allows it to change it self slowly (very possible. some are being made today) then you attach them to something and let them run free. thats a bad idea, what if it learns how to hack and get into more things, maybe remake itself and duplicate, now lets say someone is programming a bot with a physical body and that rogue A'I takes over the bot. now you have a machine that given time can make more of itself after learning and can spread itself further.. but thats all ifs and buts

Ryan Van Wart

I would love to see us design a machine that realizes it's own flaws programmed by us and builds new machines without those flaws, then to have those machines do that again. A machine that deals with us compassionately as simple life forms, that nurtures and cares for us and protects us. Machines that prevent humans from harming other humans and encourages peace and living side by side, unafraid of our fellow man. No jealousy, no fear, equality above all.

Imagine if all the mysteries of the universe could be hashed out in an afternoon by your calculator. If all the problems of mankind could be remedied in the next 30 days.