Chaos and Order - Part One: "The Monty Hall Problem"

Logic, statistics, probability. Inductive and deductive reasoning. Are you asleep yet? I’m getting paid good money to bore people into a stupor so that our robot masters can make a clean takeover. Or maybe things have changed. Maybe there are more and more of you out there who are actually kind of turned on by rational thought. Yes, logic: inductive and deductive, are rigorous disciplines, tools to ensure clarity of thought and arrive at the best possible conclusion. But logic came from the same minds that also dabbled in wild speculation. Logic has passed through the fire of almost every human discipline and imaginative daydream, and with every new arena, logic adds something more to its resume. The lens of logic can even be turned back on itself, and put to its own tests the systems of thought that we use to figure out everything else.

There is a classic problem in statistics that you might have read about (the “Monty Hall problem”). While many people use this puzzle to demonstrate the strengths of inductive reasoning, I would like to use it to examine what I feel is both the primary limitation and primary illumination of logic, an illumination that touches on the relationship between order and chaos, and on the fascinating sciences of Complexity.

So. The Monty Hall problem. The Monty Hall problem is a simple statistics scenario that has several hidden faces. Depending on how you look at it, the solution is a paradox, a magic trick, or a completely boring demonstration of simple odds; and like quantum mechanics, I’ve seen people attempt to use it to assert the powers of the human mind to alter reality. We’re going to use it to do something even more fantastic than that. We’ll use it to deconstruct logic itself.

Basically, the Monty Hall problem is as follows: You are on a game show. The host presents you with three doors and tells you: “Alright buddy, behind one of these doors is a chest of Aztec gold; behind the other two doors is nada. You choose a door, you get what is (or isn’t) behind it.”

Being a student of statistics, you realize that there is no advantage over choosing one or the other doors. You’ll just have to pick at random. Hell, go ahead and use superstition to generate your random door, it doesn’t matter. You pick door A with a sense of terse relief and cross your fingers. But instead of opening door A, the host peeks behind doors B and C. He then flings open door B revealing it to be empty.

“I’m changing the game,” Monty declares. “Door B is empty, which means the treasure is either behind A (that you chose) or C (that you didn’t). I’m going to give you a chance to change your mind. Do you want to stick with A or do you want to switch it up and go to C?”

The “problem” behind the Monty Hall problem is whether changing your mind and going with C is going to have any effect on the outcome. Is it more strategically advantageous to switch doors, or is it a wash?

Think it over for a minute. If you’ve never heard this puzzle before you might be surprised at the outcome. When this scenario is simulated on a computer in two groups: one where you stick with door A ten-thousand times, and the other where you change to door C ten-thousand times, you will find that changing to door C dramatically increases the number of times you will win. In fact, it doubles your odds.

At first blush what this seems to tell us is that by changing our minds we have altered the outcome of the game. When I initially started thinking about this puzzle, my imagination started to go wild. I was drawing connections between our three doors and quantum mechanics. Doesn’t the same thing happen there? Doesn’t changing the experiment dramatically alter the outcome? Doesn’t something as seemingly inconsequential as observation completely break down our preconceptions? And furthermore, isn’t quantum mechanics itself grounded on statistical math? Well, yes, but I was jumping the gun. I was doing what a lot of people do when they try to apply quantum rules to the macroscopic world.

Actually, there is nothing overly profound in our puzzle (though I believe that the puzzle itself reveals something profound about the way we reason). The reality of the three doors is that we really haven’t gotten much further than the most basic principles of the odds. It is the presentation of the question, set up almost like a magic trick, that misdirects us into wild speculation. Let’s break this down and look at what’s really going on.

First of all, instead of looking at the game as if we were playing the house, let’s alter our perspective a bit and say the game is you vs. your worst enemy. Whoever picks the right door wins the gold. But your worst enemy has an advantage. See, you have to pick a door, but whatever 2 doors you don’t pick go to your enemy. This gives you a 1/3 chance of winning, but your enemy has a 2/3 chance of winning. The possible scenarios for you are obvious. You either picked the door with the gold and left your enemy with nothing, or you picked nothing and left your enemy with the gold. His possible scenarios are a little more complicated, so it’s worth our time taking a closer look:

1) The enemy has one door with gold and one door with nothing
2) The enemy has one door with nothing and one door with gold
3) The enemy has two doors with nothing.

Out of those three options, your enemy has 2 scenarios that involve gold, where you have only one. This is why his odds are so much better than yours. When the host allows you to change your mind, what he is really offering is to trade your enemy’s 2/3 odds for your 1/3 odds. This is the basis of your strategic advantage. Of course you want better odds, right? So in reality there is nothing very strange going on. It is simple statistics. What makes the puzzle seem so mysterious is the fact that the host first opens a door. What makes it feel magical is the shift in perspective.

A brilliant play written by Tom Stoppard, “Rosencrantz and Guildenstern are Dead”, was made into a movie in 1990. I highly recommend devouring at least the movie, if not both. There is a scene at the beginning of the story, where R and G are walking through the forest. They’ve been paid a large sum of gold coins and they are betting on tosses. R choose “heads” every time, and he keeps winning. G is amazed that the coin could fall to heads every single time. That sort of thing just doesn’t happen, right? It is a profound experience for him. Eventually he comes to a very simple realization that logically there is no reason to be surprised by this outcome, because each individual toss has just as much of a chance as coming up heads as it does tails. If you flip a coin once and it lands on heads you are not going to be calling up your friends to tell them how mind blowing it was, so why the hell should we be amazed to see it happen over and over?

I saw this movie when I was an impressionable high school student, and this scene in particular has had an enormous effect on me ever since. When I first experienced the Monty Hall problem I went through a similar train of thought as Guildenstern. First I was amazed at the result, then a little disappointed when I truly understood the result, and finally I became profoundly moved by my own disappointment. It is not the occurrence of 92 consecutive tosses of “heads”, but rather the fact that Guildenstern should not be surprised (though he is) that is the ultimate climactic anticlimax of the human love affair with logic. It reveals a tension between inductive and deductive reasoning that borders on musical.

Here’s the thing: We’re back in the game show. Door B is open and empty. You’ve chosen door A, and now the host offers you a chance to change your answer to C. Without the setup, without the context, we see that what we’ve really got in this one moment in time, is a coin toss. And we can see it clearly when the host opens the empty door, leaving us literally with one of two options. One of two chances to win. 50/50. But this is only the case if revealing the empty door has not actually made any difference to the game.

Intuitively, I think that for one reason or another most of us want to say that it has made a difference. At the very least it has added new information, right? However, since there is only one prize, and your enemy gets two doors, at least one of those doors will always be empty, and if it is the host’s intention to offer you a chance to swap, then the host will always open that door. Because this revelation is a constant in the system, it has no mathematical bearing on the story; however, it does have a psychological impact. Interestingly, it will have a different impact depending on how you perceive the situation. If you focus on the fact that you are changing from 1 (of 3) to 2 (of 3) closed doors, it feels like you’re gaining a 2/3 chance at the gold—an obvious advantage. But if you look at the situation as the elimination of a door and a new chance to chose between the remaining two doors, this feels like you’re changing from 1 (of 3) to 1 (of 2), or going to a 50/50 shot at the gold. Which is the right way to look at it? The changing of the doors misdirects us to the 2/3 possibility, but the open door misdirects us toward the 50/50 possibility. Try this on for size: the 50/50 situation is absolutely true. And so is the 2/3.

Am I saying you are faced with two different odds simultaneously? Logicians panic! Mathematicians tremble in fear of contradiction. I call paradox; and like most paradoxes, this one is illuminating.

Here’s the dichotomy. In any given instance the odds collapse. You could have a zillion-to-one odds, but if you win, you win all the way. In an isolated moment, the world transforms from a statistical pattern to a unique, unbounded event. This dichotomy is nothing new. Xeno reveled in it. Hume used it to challenge the very notion of causality. Existentialism was mired in it. Calculus sneaks around it (Even the tamest curve cannot maintain itself at an infinitesimal without the aid of a tangent line), and though physics is trying to eliminate it (by replacing point-particle theories with string theories), it isn’t really going to solve the problem. An instance can be as large or small as you make it and still remain an instance.

What does it all mean? Statistical odds apply to patterns in multiple outcomes, but are meaningless in the moment. When you flip a coin you have 50/50 for that flip, no matter how many times you’ve flipped in the past, no matter what those flips yielded.

This doesn’t just apply when you are given only two options. Let’s use roulette as a more dramatic example. Spin the wheel, drop the ball. The ball has 38 slots it can land in. You pick one number. What are your odds? 1/38, right? Well, yes and no. If you pick 11 and it lands in 11, should you be surprised? Not really. All things being equal the ball had just as much of a chance of landing in 11 as it did in any other number on the wheel. The surprise is all in your mind, it’s all in context, and the way our minds never isolate events—we always analyze them over time, in respect to generalizations we have been constructing since before we knew we were constructing generalizations. Of course if you play multiple times, you should not expect to land on 11 every (or even every other) time. And if that happened you would be surprised. This is because odds actually do work over time, and the larger your length of time, the better they seem to work.

Let’s go back to Monty Hall. If you play the game ten thousand times, and you adopt a strategy of changing doors, experiment shows that you will win approximately 2/3 of the time, but if you don’t change you will win 1/3 of the time. As we’ve learned, the reason behind this is that by changing your mind you are (in context) opting for more options. You’re saying I want to open two doors instead of one. But this only works over time. In a single instance you are really only picking a single door, regardless of what sort of elaborate voodoo method you used to pick it. There is no orderly pattern. It’s just you vs. the universe, and your fingers crossed and a perfectly even chance of winning or not.

The immediate question that comes to my mind when we want to bring this into the real world is, why? WHY does a pattern emerge over time? Why is it amazing to flip heads 92 times in a row? Why should choosing the 2/3 option lead to more winnings? It seems obvious, but when you consider how raw and random your initial iteration is, it doesn’t really leave you with much to go on. If an instance is either/or, why over time does some index of odds emerge? The most intuitive answer seems to be that the universe has a tendency to distribute itself more or less evenly. If you’re looking for it, you’ll see this everywhere. It’s in the air you breathe. Air molecules are distributed around your room. You don’t ever start to suffocate because all the air has suddenly packed itself into one corner. From physics to chemistry to politics, distribution over time emerges as a pretty reliable principle. Energy disperses, things move toward stability, prizes will even distribute themselves between all three doors over a long enough time frame, and coins will tend to land sometimes heads and sometimes tails. Isn’t this is what 50/50 means after all—some to the left, some to the right, some dark, some light. The fundamental distribution of the moment leads to the multifaceted distribution of complex options over time.

To really make this settle in, let’s make it personal. A mathematician friend and I were talking about all of this the other day, and we were considering the scenario of 92 “heads” coin tosses in a row. The question was: what do you do about toss 93? Would you bet on it, and if so, which way and how much?

Lets say you’ve managed to steal the week’s reports on a roulette table that shows the outcome of every spin. You analyze the report and discover that number 17 came up twenty times more than any other number. Taking for granted that the table is not rigged, would you go down and bet on 17? Or would you bet on anything but 17?

In this case the number 17 (or the coin always falling “heads”) is an anomaly, a bubble of weighted distribution. Now you’re facing the same problem investors face every day, do you bet on the bubble, or do you bet against it, knowing that sooner or later it will burst? A 92 heads-in-a-row bubble might seem tempting, but the larger the bubble the more it feels as if our next toss must defy it. Which way does your personality lean? 92 heads—well the next one HAS to be tails. Or, are you, 92 heads—This is a streak! The next one is BOUND to be heads.

Here is where the black and white specter of superstition fills in the gaps where grey statistics fail us. Given a bubble, we fall further and further from our intimacy with pattern and closer to an experience of the unbounded instance. We want to trust the undistributed pattern because we are used to trusting patterns, but at the same time we distrust it simply because it is undistributed. In the moment, in the single instance where reality seems unfettered and truly free the only question is whether we believe in the forces of order more than the forces of chaos. All of this really comes down to our faith in overall even distributions in reality. That things will balance out. What goes up must come down. Take the good with the bad. Karma works.

What is the relationship between chaos and order? The Monty Hall problem, in its illumination of statistics, hints that these two concepts (which seem diametrically opposed) might in fact be intricately linked, intimately related in some overarching principle.

It seems like a paradox, that an orderly, predictable pattern could emerge from purely random states. But this is the beautiful love affair. It is not a paradox; it is inherent in the nature of randomness if you are willing to let randomness tell its full story. And if randomness were not fundamental then we should see a lot more 92-head tosses. We should see the prize hiding in door A 10,000 times in a row. To be random IS to be distributed. To begin with an open-ended universe IS to reveal this pattern. Randomness and Order are at the very least two sides of the same coin, and we will probably discover before long that they are just two ways of saying the exact same thing.

This begs a closer look at Order and Chaos. How can we get a glimpse of that unifying principle between randomness and pattern? A glimpse not of the outcomes, but of the great Coin itself?

To be Continued….

prismatic

Love is not an emotion. Love is what emerges when every emotion finds a perfect balance with one another. When this balance is broken, when love is broken, the component parts of love spill out onto the floor, revealing ridiculous severed selves of jealousy, rage, anger, passion, compassion, fear, hope, joy and sorrow. Love is the pure white submission of the lesser emotions to one transcendent goal. In love you will feel each of them tug and pull, offset by the others, just below the horizon of understanding, but when the beam cracks, you'll have your hands full of every prismatic piece dissected, vibrant and on the loose. Like subatomic particles descending through the atmosphere, like descending through the levels of hell, love decays through time into her elements, sometimes even to a state of hatred, where all emotions are isolated and vying with one another for dominance. Love was not an external glue that once held them together. Love was the epiphenomenal condition of their chance encounter with inner harmony. How fortunate to find yourself so configured, even for a day. How fortunate, at last, to find it was only an illusion.