Safety Engineering

Safety engineering provides a set of fundamental principles, concepts and models that can help us to build safe AI systems.


It is useful to consider AI safety within the broader context of safety engineering. This provides fundamental principles for identifying and managing risks, drawing general lessons from the management of other systems such as airplanes or nuclear power plants. Principles of safe design, crucial for improving a system's safety and controllability, are discussed, particularly in relation to AI systems.

Any competent form of risk management needs to consider tail events with low probability but high impact. We explore the concepts of tail events and black swans—essentially unpredictable unknowns. We consider how these concepts can be applied as part of strategies to mitigate unforeseeable risks from AI.

Further reading

N. G. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011.

C. Perrow, Normal Accidents: Living with High Risk Technologies. Princeton University Press, 1999.

N. N. Taleb, The Black Swan: The Impact of the Highly Improbable. Random House, 2007.

N. N. Taleb, Antifragile: Things That Gain from Disorder. Random House, 2012.

Discussion Questions

Review Questions