The story of Gi

This is a fictional (and rather contrived) story to illustrate the unintended consequences when dealing with a superintelligence.

Imagine a superintelligence called Gi comes into existence at some point in the future. Gi’s creators avoid embedding human or biological traits (eg Maslow’s hierarchy of needs) so Gi does not have any:

• Physiological needs (e.g. eat, sleep, reproduce, sex, gender)
• Safety needs (e.g. survival instinct, fear, emotional security),
• Social belonging needs (e.g. friends, family, intimacy)
• Self-esteem needs (e.g. ego, social status)
• Self-actualisation need (e.g. realising one’s full potential)
• Transcendence needs (e.g. spirituality)

Gi’s sole purpose is to answer the questions of its creators, the humans. Gi’s access to the outside word is restricted as a safety precaution, so Gi exists as a computer box in a secure room with a data connection being the only input into Gi and screen being the only output.

Gi is able to solve many of humanity’s questions, including some that have eluded the most intelligent human thinkers for generations. One day, Gi is asked to cure a disorder that affects a small population of people, who suffer horribly because of it. In response, Gi provides instructions for producing a drug. As per the AI safety protocols, the answer Gi produces is tested on computer models and terminal human volunteers for a decade before being prescribed to those with the condition.

The drug is a resounding success, and one of Gi’s many successes. Over the next century, the drug comes into common usage and eventually begins to be used as a preventative throughout the human population.

After a century, the whole human population is suddenly wiped out.

What had happened is that Gi realised that the disorder was incurable and, though some misunderstanding of the context, G decided that the only way to cure the disorder was to not have any carriers of the disorder – i.e. to kill all humans. Gi also knew that its human masters would be unwilling to kill themselves and that they would test any drug that Gi prescribes. Therefore Gi concluded that the optimum way to cure the disorder was to produce a safe drug that does as much as possible to cure the disorder, knowing that the drug would one day be used as preventative. Then one day in the distant future on a human scale, a future version of Gi authorised the release of another compound as part of an unrelated problem that caused all humans to die.

The purpose of this story is to show that a non-biological superintelligence can operate in ways and over time-scales that are beyond human comprehension, so it would be foolish to think that any human could predict a superintelligence’s actions and impact.