Take Action
Course
Curriculum
Download Textbook
1. Overview of Catastrophic AI Risks
0.1
Preface
1.1
Overview of Catastrophic AI Risks
1.2
Malicious Use
1.3
AI Race
1.4
Organizational Risks
1.5
Rogue AIs
1.6
Discussion of Connections Between Risks
2. AI Fundamentals
2.1
AI Fundamentals
2.2
Artificial Intelligence & Machine Learning
2.3
Deep Learning
2.4
Scaling Laws
2.5
Conclusion
3. Single Agent Safety
3.1
Single Agent Safety
3.2
Monitoring
3.3
Robustness
3.4
Alignment
3.5
Systemic Safety
3.6
Safety and General Capabilities
3.7
Conclusion
4. Safety Engineering
4.1
Safety Engineering
4.2
Risk Decomposition
4.3
Nines of Reliability
4.4
Safe Design Principles
4.5
Component Failure Accident Models and Methods
4.6
Systemic Factors
4.7
Tail Events and Black Swans
4.8
Conclusion
5. Complex Systems
5.1
Complex Systems
5.2
Introduction to Complex Systems
5.3
Complex Systems for AI Safety
5.4
Conclusion
6. Beneficial AI and Machine Ethics
6.1
Beneficial AI and Machine Ethics
6.2
Law
6.3
Fairness
6.4
The Economic Engine
6.5
Wellbeing
6.6
Preferences
6.7
Happiness
6.8
Social Welfare Functions
6.9
Moral Uncertainty
7. Collective Action Problems
7.1
Collective Action Problems
7.2
Game Theory
7.3
Cooperation
7.4
Conflict
7.5
Evolutionary Pressures
7.6
Conclusion
8. Governance
8.1
Governance
8.2
Timelines
8.3
Distribution
8.4
Compute Governance
8.5
Corporate Governance
8.6
National Governance
8.7
International Governance
8.8
Conclusion
9. Appendices
9.1
Acknowledgements
10.1
App. A: Normative Ethics
11.1
App. B: Utility Functions
12.1
App. C: Reinforcement Learning
13.1
App. D: Long-Tailed and Thin-Tailed Distributions
14.1
App. E: Evolutionary Game Theory
15.1
App. F: Other Cooperation Mechanisms
16.1
App. G: Intrasystem Conflict Causes
All
Section
Appendix
7.1

Collective Action Problems

There are many cases where intelligent agents can, despite acting rationally and in accordance with their own self-interest, collectively produce outcomes that none of them wants. Making individual AI systems reliable is not sufficient to avoid all risks from AI. Rather, we need to understand how these dynamics affect interactions between humans and AIs in order to prevent harmful, potentially catastrophic outcomes.

Summary

Addressing risks posed by individual AI systems alone is insufficient, as many challenges in AI safety come from the interaction of multiple AI developers, nations or other actors pursuing their self-interest. The principles of Game Theory can help us to understand how rational self-interest can lead to collectively undesirable results. Theories of conflict in international relations help us to understand under what circumstances the developers of advanced AI systems or the systems themselves might come into conflict with each other, which could lead to violent consequences. Lastly, the concept of Generalized Darwinism, which expands evolution by natural selection beyond biology, may also provide a useful lens to understand how the diffusion of AI systems in society could lead to an erosion of human control over AI.

Further reading

T. C. Schelling, Arms and Influence. Yale University Press, 1966.

M. A. Nowak, "Five Rules for the Evolution of Cooperation," Science, vol. 314, no. 5805, pp. 1560-1563, 2006.

J. D. Fearon, "Rationalist Explanations for War," International Organization, vol. 49, no. 3, pp. 379-414, Summer 1995.

D. Hendrycks, "Natural Selection Favors AIs over Humans," 2023.

E. O. Wilson, Sociobiology: The New Synthesis. Belknap Press of Harvard University Press, 1975.

C. Boehm, Moral Origins: The Evolution of Virtue, Altruism, and Shame. Basic Books, 2012.

R. Axelrod, The Evolution of Cooperation: Revised Edition. Basic Books, United States, 2009.

Discussion Questions

Review Questions

0.1
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
3.6
3.7
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
7.6
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
10.1
11.1
12.1
13.1
14.1
15.1
16.1
Get Involved
CourseFacilitateTake ActionSubmit Feedback
Textbook Resources
Table of ContentsDownload TextbookContent for Curriculum
Terms of ServicePrivacy Policy

Cookies Notice: This website uses cookies to identify pages that are being used most frequently. This helps us analyze data about web page traffic and improve our website. We only use this information for the purpose of statistical analysis and then the data is removed from the system. We do not and will never sell user data. Read more about our cookie policy on our privacy policy. Please contact us if you have any questions.

© 2024 Center for AI Safety
Built by Osborn Design Works