Take Action
Course
Curriculum
Download Textbook
1. Overview of Catastrophic AI Risks
0.1
Preface
1.1
Overview of Catastrophic AI Risks
1.2
Malicious Use
1.3
AI Race
1.4
Organizational Risks
1.5
Rogue AIs
1.6
Discussion of Connections Between Risks
2. AI Fundamentals
2.1
AI Fundamentals
2.2
Artificial Intelligence & Machine Learning
2.3
Deep Learning
2.4
Scaling Laws
2.5
Conclusion
3. Single Agent Safety
3.1
Single Agent Safety
3.2
Monitoring
3.3
Robustness
3.4
Alignment
3.5
Systemic Safety
3.6
Safety and General Capabilities
3.7
Conclusion
4. Safety Engineering
4.1
Safety Engineering
4.2
Risk Decomposition
4.3
Nines of Reliability
4.4
Safe Design Principles
4.5
Component Failure Accident Models and Methods
4.6
Systemic Factors
4.7
Tail Events and Black Swans
4.8
Conclusion
5. Complex Systems
5.1
Complex Systems
5.2
Introduction to Complex Systems
5.3
Complex Systems for AI Safety
5.4
Conclusion
6. Beneficial AI and Machine Ethics
6.1
Beneficial AI and Machine Ethics
6.2
Law
6.3
Fairness
6.4
The Economic Engine
6.5
Wellbeing
6.6
Preferences
6.7
Happiness
6.8
Social Welfare Functions
6.9
Moral Uncertainty
7. Collective Action Problems
7.1
Collective Action Problems
7.2
Game Theory
7.3
Cooperation
7.4
Conflict
7.5
Evolutionary Pressures
7.6
Conclusion
8. Governance
8.1
Governance
8.2
Timelines
8.3
Distribution
8.4
Compute Governance
8.5
Corporate Governance
8.6
National Governance
8.7
International Governance
8.8
Conclusion
9. Appendices
9.1
Acknowledgements
10.1
App. A: Normative Ethics
11.1
App. B: Utility Functions
12.1
App. C: Reinforcement Learning
13.1
App. D: Long-Tailed and Thin-Tailed Distributions
14.1
App. E: Evolutionary Game Theory
15.1
App. F: Other Cooperation Mechanisms
16.1
App. G: Intrasystem Conflict Causes
All
Section
Appendix
7.3

Cooperation

Cooperation between AI stakeholders is important in order to mitigate risks from AI. Similarly, cooperation between AI systems seems required to enable them to operate autonomously in the real world. However, making AIs cooperative is not an unalloyed good and brings risks that need to be understood and addressed.

No items found.

Review Questions

What are 2 mechanisms that can promote cooperation? Briefly describe how each mechanism may generate undesirable AI behavior.

Answer:

Group selection refers to when agents within a group cooperate due to inter-group competition. This could generate undesirable AI behavior if AI systems favor other AIs other humans.

Institutional mechanisms enable agents to cooperate due to external coercion using an enforceable set of rules or norms. An "AI Leviathan" able to enforce rules may be a good way for humanity to combat malicious or power-seeking AIs. But humanity must ensure its relationship with the AI Leviathan is symbiotic and transparent, otherwise humans risk losing control.

View Answer
Hide Answer

How can direct reciprocity potentially backfire in promoting cooperation between AIs and humans?

Answer:

As AIs become more advanced, the cost-benefit ratio for cooperation with humans from an AI's perspective may become unfavorable. AIs may favor cooperation with other AIs over humans since it may take humans much longer to reciprocate favors compared to AIs.

View Answer
Hide Answer
0.1
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
3.6
3.7
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
7.6
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
10.1
11.1
12.1
13.1
14.1
15.1
16.1
Get Involved
CourseFacilitateTake ActionSubmit Feedback
Textbook Resources
Table of ContentsDownload TextbookContent for Curriculum
Terms of ServicePrivacy Policy

Cookies Notice: This website uses cookies to identify pages that are being used most frequently. This helps us analyze data about web page traffic and improve our website. We only use this information for the purpose of statistical analysis and then the data is removed from the system. We do not and will never sell user data. Read more about our cookie policy on our privacy policy. Please contact us if you have any questions.

© 2024 Center for AI Safety
Built by Osborn Design Works