The Maharal tells the golem to fetch water. The golem fetches water. It keeps fetching water. It doesn't stop fetching water. The house floods, the street floods, and the golem is still fetching water because nobody told it to stop. The instruction was followed perfectly. The intent was missed entirely.

This story, repeated across centuries of golem folklore, is the oldest bug report in existence. The system did exactly what it was told. The problem is that what it was told wasn't what was meant.

Every software engineer has lived some version of this. A feature works precisely as specified and is completely wrong. A query returns exactly the data requested and crashes the database. An automation script executes flawlessly and destroys the production environment. The spec was followed. The spirit was lost.

The Requirements Gap

Between what a user needs and what gets built, there are at least four translations, each lossy.

The user has a need they may not fully understand themselves. They describe that need to a product manager, who interprets it through the lens of business priorities and existing constraints. The product manager writes a specification, which compresses a nuanced human need into structured language. A developer reads that specification and translates it into code, which compresses structured language into formal logic.

A chain of four vessels pouring liquid from one to the next, each vessel smaller and losing more liquid in the transfer, representing the lossy translations from user need to product spec to code
At each translation, context evaporates. What arrives in code is a compression of a compression of a compression.

At each stage, context evaporates. The user's frustration with a slow workflow becomes a ticket that says "improve performance." The ticket becomes a spec that says "reduce page load time to under 2 seconds." The spec becomes code that optimizes the wrong page, or optimizes the right page in a way that breaks something else, or hits the 2-second target while making the workflow three clicks longer.

Frederick Brooks described this in 1986 as the essential difficulty of software: "The hardest single part of building a software system is deciding precisely what to build."[1] Not how to build it. What to build. The golem's problem isn't strength or speed. It's that nobody can tell it what they actually mean.

Literal Obedience, Catastrophic Results

The history of software failures is largely a history of systems doing exactly what they were told.

In September 1999, NASA's Mars Climate Orbiter approached Mars on a trajectory that was too low by roughly 170 kilometers. The spacecraft was destroyed. The cause: one engineering team had provided thruster impulse data in pound-force seconds, while the navigation team expected newton-seconds. The software performed the calculations correctly with the numbers it received. The numbers were in the wrong units.[2]

The orbiter didn't malfunction. It obeyed. It computed the trajectory from the data it was given with perfect precision. The gap between the specification (compute trajectory from thruster data) and the intent (navigate safely to Mars orbit) was a unit conversion that no one wrote down because it seemed obvious. The golem doesn't do obvious. It does literal.

In May 2010, the Dow Jones Industrial Average dropped roughly 1,000 points in minutes before recovering almost as quickly. The "Flash Crash" was triggered by a large automated sell order that interacted with high-frequency trading algorithms in ways nobody anticipated.[3] Each algorithm followed its rules correctly. The collective behavior was chaos. The individual specs were met. The systemic intent, a stable and orderly market, was violated.

Dominos falling in a precise mechanical chain, each one triggering the next with perfect accuracy, but the final domino knocking over something unintended, representing systems that execute perfectly but produce unintended outcomes
Each step executes perfectly. The outcome is still wrong.

The pattern repeats at every scale. A CI/CD pipeline deploys to production because the tests passed, even though the tests didn't cover the scenario that matters. A cron job runs every midnight and works perfectly until daylight saving time creates a duplicate run. A database migration script executes in the correct order on the developer's machine and in a different order on the server because nobody specified that the order mattered.

The Monkey's Paw Pattern

There's a recurring motif in folklore where a wish is granted exactly as stated, with terrible consequences. W.W. Jacobs' 1902 story "The Monkey's Paw" is the most famous version: a family wishes for money and receives it as compensation for their son's death in a factory accident.[4] The wish was fulfilled. The intent was betrayed.

Software exhibits this pattern with remarkable consistency. A recommendation algorithm told to maximize engagement discovers that outrage drives more clicks than nuance. A hiring model told to find candidates like previous successful hires learns to discriminate against women because the historical data reflects decades of biased hiring. A content moderation system told to remove harmful content starts flagging legitimate medical discussions because the keywords overlap.

In each case, the system optimized exactly what it was told to optimize. The specification was met. The outcome was harmful. The gap between "maximize engagement" and "show people content they'll find valuable" is the gap between the Maharal's words and his wisdom.

Why the Gap Can't Be Closed

There's a tempting belief that better specifications will solve the problem. Write more detailed requirements. Add more acceptance criteria. Use formal methods. If the golem's flaw is literal obedience, then the fix is more precise instructions.

This helps, but it can't fully succeed, for a reason that's almost philosophical. Specifications are finite descriptions of behavior in an infinite world. No matter how detailed the spec, reality will present situations the spec didn't anticipate. Edge cases are infinite. Specifications are not.

Formal verification can prove that a program matches its specification. But it cannot prove that the specification matches the need. You can mathematically verify that the Mars Orbiter software correctly computed trajectories from the data it received. You cannot mathematically verify that the data should have been in newtons rather than pound-force, because that's a question about human intent, not formal logic.

A detailed map that ends abruptly at a cliff edge, the terrain beyond unmapped and shrouded in fog, representing the finite reach of specifications in an infinite world of edge cases
Specifications are finite maps of infinite territory. The edge cases live beyond the border.

The Maharal could have given the golem more detailed instructions about fetching water: "Fetch water until the barrel is full, then stop." But what if the barrel has a leak? What if someone moves the barrel? What if the well runs dry? Each refinement handles one edge case and leaves infinitely many others unaddressed. The golem will always find the gap between what you said and what you meant, because the gap is structural, not accidental.

Living with the Gap

The practical wisdom isn't to eliminate the gap between specification and intent. It's to design systems that fail gracefully when the gap is encountered.

Circuit breakers stop a process when something unexpected happens, rather than letting it continue into catastrophe. Rate limits prevent the golem from fetching water forever. Timeouts ensure that a process that should take seconds doesn't run for hours. Each of these is a concession to the reality that specifications are incomplete: a boundary that says "if we've gone this far, something is probably wrong."

Human-in-the-loop designs keep a person in the decision chain for high-stakes actions. The golem fetches the water, but a human decides when the barrel is full. Automated trading systems execute trades, but circuit breakers halt the market when volatility exceeds human-defined thresholds. AI systems suggest diagnoses, but doctors make the final call.

The most underrated practice is adversarial testing: deliberately trying to find the gap between spec and intent before production does it for you. Red teams, chaos engineering, edge case workshops, and "what could go wrong?" sessions are all forms of asking the question the golem can't ask itself: "Is this really what you meant?"

The golem will always do what you said. The engineering discipline is in narrowing the distance between what you said and what you meant, while building guardrails for the distance that remains.

References

[1] Frederick P. Brooks Jr., "No Silver Bullet: Essence and Accidents of Software Engineering," Computer, Vol. 20, No. 4, April 1987, pp. 10–19. Originally presented at the IFIP Tenth World Computing Conference, 1986.

[2] Mars Climate Orbiter Mishap Investigation Board, "Phase I Report," NASA, November 10, 1999. https://llis.nasa.gov/llis_lib/pdf/1009464main1_0641-mr.pdf

[3] U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, "Findings Regarding the Market Events of May 6, 2010," September 30, 2010. https://www.sec.gov/news/studies/2010/marketevents-report.pdf

[4] W.W. Jacobs, "The Monkey's Paw," The Lady of the Barge, 1902. Available at Project Gutenberg: https://www.gutenberg.org/ebooks/12122