Two drivers leave a bar after drinking too much. Both get behind the wheel. Both drive the same roads at the same speed with the same impaired judgment. One arrives home without incident. The other strikes and kills a pedestrian who stepped off the curb at the wrong moment.

Same decision. Same recklessness. Same moral failing. Yet we tend to treat them very differently. The first may face no consequences at all. The second faces potential manslaughter charges, public condemnation, and a lifetime of guilt. The difference between them is not character, not choice, not intention. It's luck.

In 1976, the philosophers Thomas Nagel and Bernard Williams explored this asymmetry in companion essays published in the same volume of the Proceedings of the Aristotelian Society. Nagel gave it a name: moral luck.[1] Williams argued that our moral judgments are pervasively shaped by factors beyond anyone's control.[2] The concept unsettles a deep intuition, one that Kant articulated most clearly: that morality should depend only on the will, on what we choose and intend, not on what happens to result from those choices.[3]

We don't actually believe that, at least not consistently. And the way we evaluate algorithmic systems suggests as much.

The Algorithm That Got Lucky

Consider two identical autonomous vehicle systems running the same software version, making the same decision in the same scenario: a pedestrian appears suddenly, the system calculates that braking is the optimal response, and it brakes.

Two identical vehicles on parallel roads, one in clear daylight and the other in fog, representing identical decisions with different outcomes due to chance
Same algorithm, same decision, same response time. The outcome differs because of a variable no system could have predicted.

In one case, the pedestrian stumbles forward at the last millisecond and is struck. In the other, the pedestrian stops short and is unharmed. Same algorithm, same inputs, same decision, same response time. The outcome differs because of a variable no system could have predicted or controlled.

The first incident is likely to generate headlines, regulatory investigations, and calls to restrict autonomous vehicles. The second generates nothing, because nothing happened. The algorithm that killed someone will be scrutinized, revised, possibly recalled. The algorithm that didn't will continue operating, its identical decision-making unexamined.

This is moral luck applied to machines. We judge the two systems differently despite their identical behavior, because we judge by outcomes, not by the quality of the decision process.

Probabilistic Decisions, Deterministic Judgments

The problem intensifies with systems designed to operate probabilistically. A medical AI that recommends a treatment with an 80% survival rate is making a sound decision. But when a patient in the 20% dies, the decision feels wrong, even though the math hasn't changed.[4]

A probability curve rendered as a landscape, with a small figure standing in the tail of the distribution where unlikely outcomes occur
A medical AI making a sound probabilistic decision still produces individual tragedies. The math doesn't change, but the moral weight does.

This creates what might be called the evaluation paradox for algorithmic systems. We design them to make probabilistically optimal decisions, then judge them as if each individual outcome were the only one that mattered. Consider a hypothetical diagnostic AI that correctly identifies 95% of cancers. It is celebrated until it misses one. The 5% error rate was known, disclosed, and accepted as a reasonable trade-off. But when the missed diagnosis belongs to a specific person with a name and a family, the statistical framing collapses. The question shifts from "is this system accurate enough?" to "why did it fail this patient?"

Both questions are legitimate. But they pull in opposite directions. Evaluating by aggregate outcomes rewards systems that perform well on average. Evaluating by individual outcomes punishes systems for the inevitable failures that probabilistic reasoning guarantees.

The Asymmetry of Attention

Nagel identified several varieties of moral luck. The most relevant for algorithmic systems is what he called resultant luck: the way outcomes beyond our control shape moral judgment.[1]

A spotlight illuminating one small area of a vast dark field, with identical events happening in the unlit portions that go unnoticed
We don't praise algorithms for good luck. But bad luck generates intense scrutiny.

The asymmetry runs in one direction. We don't typically praise algorithms for good luck. When an autonomous vehicle's passenger survives because a pedestrian happened to stop walking, nobody credits the algorithm. When a content moderation system leaves up a post and nothing bad happens, nobody notices. When a risk assessment tool assigns a low score and the person doesn't reoffend, the algorithm gets no recognition.

But bad luck generates intense scrutiny. When an autonomous vehicle is involved in a fatal crash, the incident often receives outsized public and media attention relative to the routine toll of human-caused traffic deaths. The algorithm is often held to a standard that human drivers are not, in part because algorithmic failures can feel like system failures while human failures tend to feel like individual tragedies.

This asymmetry may create perverse incentives. If algorithms are judged primarily by their worst outcomes rather than their decision quality, developers could be incentivized to optimize for avoiding visible failures rather than making consistently good decisions. A system that makes slightly worse decisions on average but avoids catastrophic headlines might be preferred over one that makes better decisions overall but occasionally produces a terrible result.

What Moral Luck Means for Evaluation

Moral luck doesn't have a clean resolution. Williams and Nagel disagreed about what it implies: Williams saw it as evidence that morality is more fragile and contingent than Kant supposed; Nagel saw it as a genuine paradox, a tension between what we believe (that morality shouldn't depend on luck) and how we actually judge (that it does).[1][2]

For algorithmic systems, the practical question is: how should we evaluate them? By the quality of their decision process, regardless of individual outcomes? By their aggregate performance across many decisions? By their worst-case results? Each approach has trade-offs, and moral luck suggests that no single metric will feel satisfying, because our moral intuitions pull us toward outcome-based judgment even when we know the outcome was partly a matter of chance.

What moral luck teaches us is that the gap between good decisions and good outcomes is real, irreducible, and uncomfortable. An algorithm can do everything right and still produce a tragedy. A different algorithm can make a questionable decision and get away with it. Recognizing this doesn't excuse poor design. But it might help us build evaluation frameworks that account for the role of chance, rather than pretending that every bad outcome reflects a bad decision.

References

[1] Thomas Nagel, "Moral Luck," Proceedings of the Aristotelian Society, Supplementary Volumes, Vol. 50, 1976, pp. 137–151. Reprinted in Nagel, Mortal Questions, Cambridge University Press, 1979.

[2] Bernard Williams, "Moral Luck," Proceedings of the Aristotelian Society, Supplementary Volumes, Vol. 50, 1976, pp. 115–135. Reprinted in Williams, Moral Luck: Philosophical Papers 1973–1980, Cambridge University Press, 1981.

[3] Immanuel Kant, Groundwork of the Metaphysics of Morals, 1785. Translated by Mary Gregor, Cambridge University Press, 1998.

[4] Thomas Grote and Philipp Berens, "On the ethics of algorithmic decision-making in healthcare," Journal of Medical Ethics, Vol. 46, No. 3, 2020, pp. 205–211. https://doi.org/10.1136/medethics-2019-105586