Take Action
Course
Curriculum
Download Textbook
1. Overview of Catastrophic AI Risks
0.1
Preface
1.1
Overview of Catastrophic AI Risks
1.2
Malicious Use
1.3
AI Race
1.4
Organizational Risks
1.5
Rogue AIs
1.6
Discussion of Connections Between Risks
2. AI Fundamentals
2.1
AI Fundamentals
2.2
Artificial Intelligence & Machine Learning
2.3
Deep Learning
2.4
Scaling Laws
2.5
Conclusion
3. Single Agent Safety
3.1
Single Agent Safety
3.2
Monitoring
3.3
Robustness
3.4
Alignment
3.5
Systemic Safety
3.6
Safety and General Capabilities
3.7
Conclusion
4. Safety Engineering
4.1
Safety Engineering
4.2
Risk Decomposition
4.3
Nines of Reliability
4.4
Safe Design Principles
4.5
Component Failure Accident Models and Methods
4.6
Systemic Factors
4.7
Tail Events and Black Swans
4.8
Conclusion
5. Complex Systems
5.1
Complex Systems
5.2
Introduction to Complex Systems
5.3
Complex Systems for AI Safety
5.4
Conclusion
6. Beneficial AI and Machine Ethics
6.1
Beneficial AI and Machine Ethics
6.2
Law
6.3
Fairness
6.4
The Economic Engine
6.5
Wellbeing
6.6
Preferences
6.7
Happiness
6.8
Social Welfare Functions
6.9
Moral Uncertainty
7. Collective Action Problems
7.1
Collective Action Problems
7.2
Game Theory
7.3
Cooperation
7.4
Conflict
7.5
Evolutionary Pressures
7.6
Conclusion
8. Governance
8.1
Governance
8.2
Timelines
8.3
Distribution
8.4
Compute Governance
8.5
Corporate Governance
8.6
National Governance
8.7
International Governance
8.8
Conclusion
9. Appendices
9.1
Acknowledgements
10.1
App. A: Normative Ethics
11.1
App. B: Utility Functions
12.1
App. C: Reinforcement Learning
13.1
App. D: Long-Tailed and Thin-Tailed Distributions
14.1
App. E: Evolutionary Game Theory
15.1
App. F: Other Cooperation Mechanisms
16.1
App. G: Intrasystem Conflict Causes
All
Section
Appendix
8.1

Governance

There are many cases where intelligent agents can, despite acting rationally and in accordance with their own self-interest, collectively produce outcomes that none of them wants. Making individual AI systems reliable is not sufficient to avoid all risks from AI. Rather, we need to understand how these dynamics affect interactions between humans and AIs in order to prevent harmful, potentially catastrophic outcomes.

Summary

To design effective policies to capture AI's benefits and manage its risks, we need to consider some fundamental variables such as the speed of progress in AI capabilities, and the breadth of access to highly powerful AI systems. We outline several possible scenarios regarding the speed of AI development and review arguments for and against a dramatic acceleration of economic growth due to AI progress. We also explore how varying concentrations of power, both in the number of AIs and the access to these AIs, can alter the risks and benefits we face.

We then discuss potential governance approaches at different level to ensure AI development is well-regulated and beneficial. We examine how corporations can be held to safety standards and how national and international bodies can implement effective legal oversight of AI activities. Some forms of international cooperation may be needed to mitigate competitive pressures.

Further reading

J. Sevilla, L. Heim, A. Ho, T. Besiroglu, M. Hobbhahn, and P. Villalobos, "Compute trends across three eras of machine learning," in 2022 International Joint Conference on Neural Networks (IJCNN), July 2022, pp. 1-8, IEEE.

E. Erdil and T. Besiroglu, "Explosive growth from AI automation: A review of the arguments," arXiv preprint arXiv:2309.11690, 2023.

S. M. Khan and A. Mann, "AI Chips: What They Are and Why They Matter," Center for Security and Emerging Technology, April 2020. [Online]. Available: https://doi.org/10.51593/20190014

Y. Shavit, "What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring," arXiv preprint arXiv:2303.11341, 2023.

M. Anderljung et al., "Frontier AI regulation: Managing emerging risks to public safety," arXiv preprint arXiv:2307.03718, 2023.

M. Maas, "Literature Review of Transformative AI Governance," 2023 draft.

R. Trager et al., "International Governance of Civilian AI: A Jurisdictional Certification Approach," LPP Working Paper No. 3-2023, August 31, 2023. [Online]. Available: https://ssrn.com/abstract=4579899 or http://dx.doi.org/10.2139/ssrn.4579899

L. Ho et al., "International Institutions for Advanced AI," arXiv preprint arXiv:2307.04699, 2023.

Discussion Questions

Review Questions

0.1
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
3.6
3.7
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
7.6
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
10.1
11.1
12.1
13.1
14.1
15.1
16.1
Get Involved
CourseFacilitateTake ActionSubmit Feedback
Textbook Resources
Table of ContentsDownload TextbookContent for Curriculum
Terms of ServicePrivacy Policy

Cookies Notice: This website uses cookies to identify pages that are being used most frequently. This helps us analyze data about web page traffic and improve our website. We only use this information for the purpose of statistical analysis and then the data is removed from the system. We do not and will never sell user data. Read more about our cookie policy on our privacy policy. Please contact us if you have any questions.

© 2024 Center for AI Safety
Built by Osborn Design Works