Beyond Good and Error - Magical Thinking
On the 12th of April, 2008, the construction of the new Yankee stadium came to a halt. Word had spread that the foundation of the building had been sabotaged. If the rumors were true, then a shadow had been cast on the very future of the team itself. Jack-hammers began to pound at the freshly-set concrete foundation. Suddenly the hammering stopped, and a silence took hold, as a workman reached into the rubble. Then, applause erupted and a flurry of camera flashes illuminated the scene as he pulled the offending item out of the rubble: a Boston Red Sox jersey, number 34, Ortiz. The curse had been lifted.
In my previous article, I wrote about first principles thinking; the foundation of knowledge itself. This time I would like to describe magical thinking: first principles’ evil twin. The same line of reasoning which led Yankee management to spend time and money in order to excavate a shirt has also led to countless programming errors and design mis-steps. So let’s familiarize ourselves with what it is and how we can guard against it.
Magical thinking is the belief that ideas, thoughts, and actions can carry a causal link to unconnected events in the material world. It has been with us since time immemorial in the form of rain dances, virgin sacrifices, and other rituals. Its prevalence began declining as science offered better explanations for the workings of our universe, but it has never quite left us. For example, when a modern how-to guide was published in 2006, it sold 30 million copies, in 50 languages worldwide, and spawned two films (one a documentary, the other a drama).
So the Yankees are in good company. Players from many sports have been guilty of similar crimes against rationality. Some have been known to wear old clothes under their uniforms (Jordan), bounce a tennis ball a set number of times before a serve (Williams), and perform a whole host of strange rituals (Nadal). When the delay between action and outcome is as short as it is in sports, a causal link can suggest itself (e.g. wore lucky golden underpants, won match (Giambi [former Yankee]). Without first principles reasoning to question or test the relationship between a given action and a favorable outcome, magical thinking takes root and drives future decision-making.
Coders spend their days (and often nights) wrestling with logic. Perhaps due to this, they tend not to exhibit such fanciful quirks as our beloved sports stars. However, we’ll see that the sciences and indeed our own work can also fall victim to this phenomenon if we’re not careful. One famous anecdote involves a quantum physicist and Nobel laureate (Bohr) getting teased about a horseshoe hanging over his door. Did he really believe in the magical properties of equine footwear? “No,” Bohr replied, “but I am told that they bring luck even to those who do not believe in them.”
Magical thinking sneaks into our lives whenever we notice favorable outcomes and fail to sufficiently probe their underlying causes. In SQL, if a “DISTINCT” parameter can eliminate duplicates, as can a “GROUP BY”, why not use both? (True story.) While the query may return the desired result, using both of these functions together indicates that the developer understands neither. A “UNION” combines result sets from two queries with a single word, while “UNION ALL” requires three extra key strokes to produce the same result (nearly always). Will you use magical thinking to explain away why two seemingly identical operators exist, or will you rely on first principals to uncover how they differ?
Without sufficient understanding of the underlying principles, it can be easy to become convinced that the output of your code is subject to the whims of forces beyond your control. A developer once said to me “whenever I do A, then B happens, but that’s impossible!”
The fundamental property of code is that it’s linear. A computer can’t help but run through the instructions which you’ve programmed into it. When the output does not match your expectations, the cause will either stem from your lack of understanding of the fundamentals (e.g. what a “GROUP BY” is intended to do), or your lack of familiarity with the tools (e.g. that a “UNION” also eliminates duplicate records in addition to combining two sets of data).
Since it is not in our nature to think like a computer, such mis-steps will invariably occur. To err is human, after all, and the most magical thought of all is to believe otherwise. If there is one fallacious belief responsible for more errors than all the others combined, it’s that a change can be performed without testing because “X”. Variations of “X” include:
- it’s a one-line change
- I already tested this in another script
- I’m a senior developer and I know my stuff
- I’m only adding/removing a comment
- I’ve already worked it out in my head
- it will be fine
Now that you’ve seen what magical thinking is and how easily it can subvert your efforts, don’t panic — you already know how to cast the protective spell: stay on your guard, act from first principles, hang a horseshoe over your work area, and above all, test! In the next post we’ll explore a powerful testing technique which can help bullet-proof your code. Stay tuned.
This article is part of a series.