Take Action
Course
Curriculum
Download Textbook
1. Overview of Catastrophic AI Risks
0.1
Preface
1.1
Overview of Catastrophic AI Risks
1.2
Malicious Use
1.3
AI Race
1.4
Organizational Risks
1.5
Rogue AIs
1.6
Discussion of Connections Between Risks
2. AI Fundamentals
2.1
AI Fundamentals
2.2
Artificial Intelligence & Machine Learning
2.3
Deep Learning
2.4
Scaling Laws
2.5
Conclusion
3. Single Agent Safety
3.1
Single Agent Safety
3.2
Monitoring
3.3
Robustness
3.4
Alignment
3.5
Systemic Safety
3.6
Safety and General Capabilities
3.7
Conclusion
4. Safety Engineering
4.1
Safety Engineering
4.2
Risk Decomposition
4.3
Nines of Reliability
4.4
Safe Design Principles
4.5
Component Failure Accident Models and Methods
4.6
Systemic Factors
4.7
Tail Events and Black Swans
4.8
Conclusion
5. Complex Systems
5.1
Complex Systems
5.2
Introduction to Complex Systems
5.3
Complex Systems for AI Safety
5.4
Conclusion
6. Beneficial AI and Machine Ethics
6.1
Beneficial AI and Machine Ethics
6.2
Law
6.3
Fairness
6.4
The Economic Engine
6.5
Wellbeing
6.6
Preferences
6.7
Happiness
6.8
Social Welfare Functions
6.9
Moral Uncertainty
7. Collective Action Problems
7.1
Collective Action Problems
7.2
Game Theory
7.3
Cooperation
7.4
Conflict
7.5
Evolutionary Pressures
7.6
Conclusion
8. Governance
8.1
Governance
8.2
Timelines
8.3
Distribution
8.4
Compute Governance
8.5
Corporate Governance
8.6
National Governance
8.7
International Governance
8.8
Conclusion
9. Appendices
9.1
Acknowledgements
10.1
App. A: Normative Ethics
11.1
App. B: Utility Functions
12.1
App. C: Reinforcement Learning
13.1
App. D: Long-Tailed and Thin-Tailed Distributions
14.1
App. E: Evolutionary Game Theory
15.1
App. F: Other Cooperation Mechanisms
16.1
App. G: Intrasystem Conflict Causes
All
Section
Appendix
6.1

Beneficial AI and Machine Ethics

To ensure that AI systems have a positive impact, it is not enough to focus on mitigating risks of harmful behaviour. Rather, we need to instill beneficial goals in AIs and ensure that these align with our values and ethical principles. However, there are many potential frameworks to be considered for this purpose.

Summary

Defining the principles that advanced AI systems should follow is no trivial task. There are many plausible candidates, such as complying with users' preferences, following the law, or doing what is ethical. While each of these proposals has its attractions, they all face limitations as well, and may come into conflict with each other. We explore these questions and consider what it would mean to design an AI system that promotes the wellbeing of users and of society as a whole. We also discuss how AI systems should deal with situations where there is uncertainty about the right course of action.

Further reading

W. Wallach and C. Allen, Moral Machines: Teaching Right From Wrong. Oxford University Press, 2008.

J. Nay, "Law Informs Code," 2023. [Online]. Available: https://arxiv.org/ftp/arxiv/papers/2209/2209.13020.pdf

S. Barocas, M. Hardt, and A. Narayanan, Fairness and Machine Learning: Limitations and Opportunities. 2023. [Online]. Available: https://fairmlbook.org/

R. Layard and J. De Neve, Wellbeing: Science and Policy. Cambridge University Press, India, 2023.

M. J. Sandel, What Money Can't Buy: The Moral Limits of Markets. Farrar, Straus and Giroux, 2012.

D. Hendrycks et al., "What Would Jiminy Cricket Do? Towards Agents That Behave Morally," 2022. [Online]. Available: https://arxiv.org/abs/2110.13136

M. D. Adler, "Introduction," in Measuring Social Welfare: An Introduction. Oxford Academic, New York, 2019. [Online]. Available: https://doi.org/10.1093/oso/9780190643027.003.0001

S. Lukes, Moral Relativism. 2008.

K. de Lazari-Radek and P. Singer, Utilitarianism: A Very Short Introduction, Very Short Introductions. Oxford Academic, Oxford, 2017. [Online]. Available: https://doi.org/10.1093/actrade/9780198728795.001.0001

S. Kagan, Normative Ethics. Westview Press, 1998.

S. Newberry and T. Ord, "The Parliamentary Approach to Moral Uncertainty," 2022. [Online]. Available: https://www.fhi.ox.ac.uk/wp-content/uploads/2021/06/Parliamentary-Approach-to-Moral-Uncertainty.pdf

S. Darwall, Deontology: A Very Short Introduction. Oxford University Press, 2013.

Discussion Questions

Review Questions

0.1
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
3.6
3.7
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
7.6
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
10.1
11.1
12.1
13.1
14.1
15.1
16.1
Get Involved
CourseFacilitateTake ActionSubmit Feedback
Textbook Resources
Table of ContentsDownload TextbookContent for Curriculum
Terms of ServicePrivacy Policy

Cookies Notice: This website uses cookies to identify pages that are being used most frequently. This helps us analyze data about web page traffic and improve our website. We only use this information for the purpose of statistical analysis and then the data is removed from the system. We do not and will never sell user data. Read more about our cookie policy on our privacy policy. Please contact us if you have any questions.

© 2024 Center for AI Safety
Built by Osborn Design Works