Take Action
Course
Curriculum
Download Textbook
1. Overview of Catastrophic AI Risks
0.1
Preface
1.1
Overview of Catastrophic AI Risks
1.2
Malicious Use
1.3
AI Race
1.4
Organizational Risks
1.5
Rogue AIs
1.6
Discussion of Connections Between Risks
2. AI Fundamentals
2.1
AI Fundamentals
2.2
Artificial Intelligence & Machine Learning
2.3
Deep Learning
2.4
Scaling Laws
2.5
Conclusion
3. Single Agent Safety
3.1
Single Agent Safety
3.2
Monitoring
3.3
Robustness
3.4
Alignment
3.5
Systemic Safety
3.6
Safety and General Capabilities
3.7
Conclusion
4. Safety Engineering
4.1
Safety Engineering
4.2
Risk Decomposition
4.3
Nines of Reliability
4.4
Safe Design Principles
4.5
Component Failure Accident Models and Methods
4.6
Systemic Factors
4.7
Tail Events and Black Swans
4.8
Conclusion
5. Complex Systems
5.1
Complex Systems
5.2
Introduction to Complex Systems
5.3
Complex Systems for AI Safety
5.4
Conclusion
6. Beneficial AI and Machine Ethics
6.1
Beneficial AI and Machine Ethics
6.2
Law
6.3
Fairness
6.4
The Economic Engine
6.5
Wellbeing
6.6
Preferences
6.7
Happiness
6.8
Social Welfare Functions
6.9
Moral Uncertainty
7. Collective Action Problems
7.1
Collective Action Problems
7.2
Game Theory
7.3
Cooperation
7.4
Conflict
7.5
Evolutionary Pressures
7.6
Conclusion
8. Governance
8.1
Governance
8.2
Timelines
8.3
Distribution
8.4
Compute Governance
8.5
Corporate Governance
8.6
National Governance
8.7
International Governance
8.8
Conclusion
9. Appendices
9.1
Acknowledgements
10.1
App. A: Normative Ethics
11.1
App. B: Utility Functions
12.1
App. C: Reinforcement Learning
13.1
App. D: Long-Tailed and Thin-Tailed Distributions
14.1
App. E: Evolutionary Game Theory
15.1
App. F: Other Cooperation Mechanisms
16.1
App. G: Intrasystem Conflict Causes
All
Section
Appendix
6.6

Preferences

While it may seem intuitive to let AI systems be guided by people's preferences, there are a number of objections to relying on these.

No items found.

Review Questions

Explain the difference between stated, revealed, and idealized preferences.

Answer:

Stated preferences are expressed, revealed preferences are inferred from choices, and idealized preferences are those held with perfect information and judgment.

‍

View Answer
Hide Answer

How can we use stated preferences to train AIs? What is one practical limitation of using human feedback to train AI systems?

Answer:

Humans can directly evaluate and rank AI outputs as training feedback. A limitation is infeasibility as tasks become too complex for humans to judge.

View Answer
Hide Answer

Why might using human preferences alone be insufficient for comprehensive machine ethics?

Answer:

Preferences may not capture important factors like autonomy, may need to be aggregated, and could still lead to unethical outcomes if satisfied.

View Answer
Hide Answer
0.1
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
3.6
3.7
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
7.6
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
10.1
11.1
12.1
13.1
14.1
15.1
16.1
Get Involved
CourseFacilitateTake ActionSubmit Feedback
Textbook Resources
Table of ContentsDownload TextbookContent for Curriculum
Terms of ServicePrivacy Policy

Cookies Notice: This website uses cookies to identify pages that are being used most frequently. This helps us analyze data about web page traffic and improve our website. We only use this information for the purpose of statistical analysis and then the data is removed from the system. We do not and will never sell user data. Read more about our cookie policy on our privacy policy. Please contact us if you have any questions.

© 2024 Center for AI Safety
Built by Osborn Design Works