Game Theory

Nash equilibria can be Pareto inefficient, as seen in the Prisoner's Dilemma. In other words, there may be situations where the rational incentives of each member of a group drive them to an output which could have been improved at no cost to each member of the group.

We see a number of collective action problems on a global scale such an environmental pollution and climate change, whereby it is hard to make actors producing harmful externalities to pay for the full costs that this imposes on others.

‍

There are several lines of evidence suggesting that this is possible. In Game Theory, we see that extortion strategies are often successful in the Iterated Prisoner’s Dilemma. AI-specific factors that could facilitate the use of extortion include features such as superhuman strategic planning abilities, goal preservation and longevity, as well as heightened scope sensitivity and moral impartiality.

A corporate AI race could lead to an autonomous economy where AIs make all important decisions and humans are no longer able to steer the economy , even though this could be a disastrous outcome for humanity as a whole.