Take Action
Course
Curriculum
Download Textbook
1. Overview of Catastrophic AI Risks
0.1
Preface
1.1
Overview of Catastrophic AI Risks
1.2
Malicious Use
1.3
AI Race
1.4
Organizational Risks
1.5
Rogue AIs
1.6
Discussion of Connections Between Risks
2. AI Fundamentals
2.1
AI Fundamentals
2.2
Artificial Intelligence & Machine Learning
2.3
Deep Learning
2.4
Scaling Laws
2.5
Conclusion
3. Single Agent Safety
3.1
Single Agent Safety
3.2
Monitoring
3.3
Robustness
3.4
Alignment
3.5
Systemic Safety
3.6
Safety and General Capabilities
3.7
Conclusion
4. Safety Engineering
4.1
Safety Engineering
4.2
Risk Decomposition
4.3
Nines of Reliability
4.4
Safe Design Principles
4.5
Component Failure Accident Models and Methods
4.6
Systemic Factors
4.7
Tail Events and Black Swans
4.8
Conclusion
5. Complex Systems
5.1
Complex Systems
5.2
Introduction to Complex Systems
5.3
Complex Systems for AI Safety
5.4
Conclusion
6. Beneficial AI and Machine Ethics
6.1
Beneficial AI and Machine Ethics
6.2
Law
6.3
Fairness
6.4
The Economic Engine
6.5
Wellbeing
6.6
Preferences
6.7
Happiness
6.8
Social Welfare Functions
6.9
Moral Uncertainty
7. Collective Action Problems
7.1
Collective Action Problems
7.2
Game Theory
7.3
Cooperation
7.4
Conflict
7.5
Evolutionary Pressures
7.6
Conclusion
8. Governance
8.1
Governance
8.2
Timelines
8.3
Distribution
8.4
Compute Governance
8.5
Corporate Governance
8.6
National Governance
8.7
International Governance
8.8
Conclusion
9. Appendices
9.1
Acknowledgements
10.1
App. A: Normative Ethics
11.1
App. B: Utility Functions
12.1
App. C: Reinforcement Learning
13.1
App. D: Long-Tailed and Thin-Tailed Distributions
14.1
App. E: Evolutionary Game Theory
15.1
App. F: Other Cooperation Mechanisms
16.1
App. G: Intrasystem Conflict Causes
All
Section
Appendix
2.4

Scaling Laws

In AI, scaling laws help us to understand and predict how changes in variables like the amount of computation and data used can have substantial impacts on a model’s performance.

No items found.

Review Questions

State whether each of the following statements is true or false. If false, explain why or provide a counterexample.

  1. Power laws are mathematical equations that model how a particular quantity varies as a power of another.
  2. The equation A= r2 is a power law.
  3. The equation y=a(x) +(b) is a power law.
  4. When graphed on a standard (linear scale) plot, a power law relationship will appear as a straight line.
  5. Scaling laws are a particular kind of power law that describe how deep learning models scale.
  6. Scaling laws in deep learning predict loss based on model size and dataset size.
  7. All ML models follow scaling laws.
  8. Intricate, expert-designed AI systems generally perform better than deep learning models with many parameters trained on large amounts of data.

‍

Answer:
  1. True.
  2. True.
  3. False. In this equation, the dependent variable y is logarithmically (not exponentially) related to any of the other variables.
  4. False. While power law relationships appear as straight lines on plots with logarithmic scales, when graphed on standard (linear) plots with constant scale, power law relationships may take many different forms. Consider the equations y=x2 and y=x3. Both are power laws, but when graphed on a standard plot, they will be quadratic and cubic curves, respectively.
  5. True.
  6. True.
  7. False. Not all ML models follow scaling laws. While generative models tend to follow regular scaling laws, discriminative models do not exhibit behavior that suggests clear scaling laws.
  8. False. The bitter lesson suggests that scaling data and computational power tends to be more effective than human-designed approaches.

‍

View Answer
Hide Answer

Write an example of a power law equation with one independent variable and draw a rough sketch of what it would look like on a log-log plot. Label the y-intercept and slope. What part of the equation do each of these values represent?

Answer:

Exact equations will vary, but should be of a form y=bxa; graphs should display a straight line with y-intercept and slope labeled. The y-intercept corresponds to the coefficient in the equation and the slope corresponds to the exponent. For example, the power law y=5x3 will have slope 3 and y-intercept (5).

View Answer
Hide Answer

How is compute (the computation resources used in training) related to scaling? Describe which factors it influences and how.

Answer:

Compute is a vital part of scaling; scaling is only possible through increasing compute. Training models with more parameters or larger datasets requires more computing power. Accordingly, compute greatly influences both model size and dataset size.

View Answer
Hide Answer
0.1
1.1
1.2
1.3
1.4
1.5
1.6
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
3.6
3.7
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
7.1
7.2
7.3
7.4
7.5
7.6
8.1
8.2
8.3
8.4
8.5
8.6
8.7
8.8
9.1
10.1
11.1
12.1
13.1
14.1
15.1
16.1
Get Involved
CourseFacilitateTake ActionSubmit Feedback
Textbook Resources
Table of ContentsDownload TextbookContent for Curriculum
Terms of ServicePrivacy Policy

Cookies Notice: This website uses cookies to identify pages that are being used most frequently. This helps us analyze data about web page traffic and improve our website. We only use this information for the purpose of statistical analysis and then the data is removed from the system. We do not and will never sell user data. Read more about our cookie policy on our privacy policy. Please contact us if you have any questions.

© 2024 Center for AI Safety
Built by Osborn Design Works