An Example of Bayesian Updating

A simple example of Bayesian updating, leading to Bayes' Theorem for distributions

Introduction

Let’s take a look at how we can use Bayes’ Theorem to update our beliefs about a situation, given some data!

The Problem and Solution

Imagine that you are in the following situation.

You have a box in front of you. You know that in the box are two regular coins (R), with a probability of flipping Heads of $\frac{1}{2}$, and on trick coin (A), which always lands on Heads. You choose a coin at random from the box and flip it, resulting in Heads. Which coin do you think it was?

This is a classic problem for Bayes’ Theorem - we have a result, and want to reason “backwards” to the causes. Let’s apply the theorem to calculate the probability of the coin being a regular coin or a trick coin.

Bayes’ Theorem is

Let’s start by calculating the probability that the coin we flipped was regular. Then we have

Although at this point we can clearly reason out that the probability that it is the all-Heads coin is also $\frac{1}{2}$, let’s go through the work anyway.

Before we move on, let’s examine the result that $P(H) = P(R)P(H|R) + P(A)P(H|A)$. If we wanted to be slightly more explicit, we could express this as

using the conditional probability formula. Note that this is actually independent of which condition (regular coin or all heads) we are actually dealing with.

Now let’s say that we flip the coin again, and again it is heads. What does this do to our belief about the nature of the coin?

Again, let’s go through the calculations:

and

So, as expected, as we continue to flip heads our belief that the coin that we have is the all-Heads one grows and grows.

Let’s take a look at these results graphically:

Notice the relabelling of the coins.

If we didn’t know better, those would look like the graph of probability distributions on the nature of the coin! In fact, that is exactly what those are. By calculating all of our degree of belief in the nature of the coin, we are exactly creating a distribution over it!

Bayes’ Theorem for Distributions

Although it may not be clear at the moment, what we are doing when updating our beliefs in this manner is using a version of Baye’s Theorem for an entire distribution, rather than a point estimate of a probability. This version of Baye’s Theorem, for a distribution $f$ on an unknown variable $\theta$ is

In fact, we can go even further. We know that the posterior distribution will be a probability distribution, and so must sum to 1. As such, we can view the marginal ($f(\text{data})$) as a normalizing factor to ensure that this is the case. Thus, we can express this same idea as

Conclusion

We began with a simple problem which we could solve using Bayes’ Theorem. During that problem, we updated our beliefs about the probability of the nature of a coin, and used that to motivate the idea of this update as updating a distribution rather than merely a point estimate. We then extended that (with very little justification) to two formulations of Bayes’ Theorem for Distributions.