CS 70 Discrete Mathematics for CS

Fall 2004  Rao  Lecture 19

Variance

Question: At each time step, I flip a fair coin. If it comes up Heads, I walk one step to the right; if it comes up Tails, I walk one step to the left. How far do I expect to have traveled from my starting point after n steps?

Denoting a right-move by +1 and a left-move by −1, we can describe the probability space here as the set of all words of length n over the alphabet \{±1\}, each having equal probability \frac{1}{2^{n}}. Let the r.v. X denote our position (relative to our starting point 0) after n moves. Thus

X = X_{1}+ X_{2}+ ··· + X_{n},

where X_{i} = \{ \begin{matrix}+1 & if ith toss is Heads; \\ −1 & otherwise. \end{matrix}

Now obviously we have E(X) = 0. The easiest rigorous way to see this is to note that E(X_{i}) = (\frac{1}{2} ×1)+(\frac{1}{2} × (−1)) = 0, so by linearity of expectation E(X) = ∑ _{i=1}^{n} E(X_{i}) = 0. Thus after n steps, my expected position is 0! But of course this is not very informative, and is due to the fact that positive and negative deviations from 0 cancel out.

What the above question is really asking is: What is the expected value of |X|, our distance from 0? Rather than consider the r.v. |X|, which is a little awkward due to the absolute value operator, we will instead look at the r.v. X^{2}. Notice that this also has the effect of making all deviations from 0 positive, so it should also give a good measure of the distance traveled. However, because it is the squared distance, we will need to take a square root at the end.

Let’s calculate E(X^{2}):

E(X^{2})= E((X_{1}+ X_{2}+ ··· + X_{n})^{2})
= E(∑ _{i=1}^{n} X_{i}^{2}+ ∑_{i≠j} X_{i}X_{j})
= ∑ _{i=1}^{n} E(X_{i}^{2}) + ∑_{i≠j} E(X_{i}X_{j})

In the last line here, we used linearity of expectation. To proceed, we need to compute E(X_{i}^{2}) and E(X_{i}X_{j}) (for i ≠ j). Let’s consider first X_{i}^{2}. Since X_{i} can take on only values ±1, clearly X_{i}^{2} = 1 always, so E(X_{i}^{2}) = 1. What about E(X_{i}X_{j})? Since X_{i} and X_{j} are independent, it is the case that E(X_{i}X_{j}) = E(X_{i})E(X_{j}) = 0.^{1} Plugging these values into the above equation gives

E(X^{2}) = (n × 1) + 0 = n.

So we see that \underline{our expected squared distance from 0 is n}. One interpretation of this is that we might expect to be a distance of about \sqrt{n} away from 0 after n steps. However, we have to be careful here: we cannot simply argue that E(|X|) = \sqrt{E(X^{2})}= \sqrt{n}. (Why not?) We will see shortly (see Chebyshev’s Inequality below) how to make precise deductions about |X| from knowledge of E(X^{2}).

1Two random variables X and Y are independent if the events “X = a” and “Y = b” are independent for all pairs of values a,b.

If X,Y are independent, then we have E(XY) = E(X)E(Y); you’ll be asked to prove this on one of your homeworks. Note that E(XY) = E(X)E(Y) is false for general r.v.’s X,Y; as an example just look at E(X_{i}^{2}) in the present discussion.

CS 70, Fall 2004, Lecture 19 1

For the moment, however, let’s agree to view E(X^{2}) as an intuitive measure of “spread” of the r.v. X. In fact, for a more general r.v. with expectation E(X) = μ, what we are really interested in is E((X − μ)^{2}), the expected squared distance from the mean. In our random walk example, we had μ = 0, so E((X − μ)^{2}) just reduces to E(X^{2}).

Definition 19.1 (variance): For a r.v. X with expectation E(X) = μ, the \underline{variance} of X is defined to be

Var(X) = E((X − μ)^{2}).

The square root \sqrt{Var(X)}is called the \underline{standard deviation} of X.

The point of the standard deviation is merely to “undo” the squaring in the variance. Thus the standard deviation is “on the same scale as” the r.v. itself. Since the variance and standard deviation differ just by a square, it really doesn’t matter which one we choose to work with as we can always compute one from the other immediately. We shall usually use the variance. For the random walk example above, we have that Var(X) = n, and the standard deviation of X is \sqrt{n}.

The following easy observation gives us a slightly different way to compute the variance that is simpler in many cases.

Theorem 19.1: For a r.v. X with expectation E(X) = μ, we have Var(X) = E(X^{2}) − μ^{2}.

Proof: From the definition of variance, we have

Var(X) = E((X − μ)^{2}) = E(X^{2}− 2μX + μ^{2}) = E(X^{2}) − 2μE(X) + μ^{2} = E(X^{2}) − μ^{2}.

In the third step here, we used linearity of expectation. □

Let’s see some examples of variance calculations.

1. Fair die. Let X be the score on the roll of a single fair die. Recall from an earlier lecture that E(X) = \frac{7}{2}. So we just need to compute E(X^{2}), which is a routine calculcation:

E(X^{2}) = \frac{1}{6} (1^{2}+ 2^{2}+ 3^{2}+ 4^{2}+ 5^{2}+ 6^{2}) = \frac{91}{6}.

Thus from Theorem 22.1

Var(X) = E(X^{2}) − (E(X))^{2} = \frac{91}{6} − \frac{49}{4} = \frac{35}{12}.

2. Biased coin. Let X the the number of Heads in n tosses of a biased coin with Heads probability p (i.e., X has the binomial distribution with parameters n,p). We already know that E(X) = np. Writing as usual X = X_{1}+ X_{2}+ ··· + X_{n}, where

X_{i} = \{ \begin{matrix}1 & if ith toss is Head; \\ 0 & otherwise, \end{matrix}

we have

E(X^{2})= E((X_{1}+ X_{2}+ ··· + X_{n})^{2})
= ∑ _{i=1}^{n} E(X_{i}^{2}) + ∑_{i≠j} E(X_{i}X_{j})
= (n × p) + (n(n − 1) × p^{2})
= n^{2}p^{2}+ np(1 − p).

In the third line here, we have used the facts that E(X_{i}^{2}) = p, and that E(X_{i}X_{j}) = E(X_{i})E(X_{j}) = p^{2} (since X_{i},X_{j} are independent). Note that there are n(n − 1) pairs i,j with i ≠ j.

Finally, we get that Var(X) = E(X^{2}) − (E(X))^{2} = np(1 − p). As an example, for a fair coin the expected number of Heads in n tosses is \frac{n}{2}, and the standard deviation is \frac{\sqrt{n}}{2}.

Notice that in fact Var(X) = ∑_{i}Var(X_{i}), and the same was true in the random walk example. This is no coincidence, and it depends on the fact that the X_{i} are mutually independent. So in the independent case we can compute variances of sums very easily. Note that this isn’t the case, however, in example 4.

3. Poisson distribution. Let X have the Poisson distribution with parameter λ. We saw in the last lecture that E(X) = λ. And we also have

E(X^{2}) = ∑_{i=0}^{∞} i^{2}e^{−λ} \frac{λ^{i}}{i!} = λ ∑_{i=1}^{∞} ie^{−λ} \frac{λ^{i−1}}{(i − 1)!} = λ ( ∑_{i=1}^{∞} (i − 1)e^{−λ} \frac{λ^{i−1}}{(i − 1)!} + ∑_{i=1}^{∞} e^{−λ} \frac{λ^{i−1}}{(i − 1)!} ) = λ(λ +1).

[Check you follow each of these steps. In the last step, we have noted that the two sums are respectively E(X) and ∑_{i}Pr[X = i] = 1.]

Finally, we get Var(X) = E(X^{2}) − (E(X))^{2} = λ. So, for a Poisson random variable, the expectation and variance are equal.

4. Number of fixed points. Let X be the number of fixed points in a random permutation of n items (i.e., the number of students in a class of size n who receive their own homework after shuffling). We saw in an earlier lecture that E(X) = 1 (regardless of n). To compute E(X^{2}), write X = X_{1}+ X_{2}+ ··· + X_{n},

where X_{i} = \{ \begin{matrix}1 & if i is a fixed point; \\ 0 & otherwise \end{matrix}

Then as usual we have

E(X^{2}) = ∑_{i=1}^{n} E(X_{i}^{2}) + ∑_{i≠j} E(X_{i}X_{j}).  (1)

Since X_{i} is an indicator r.v., we have that E(X_{i}^{2}) = Pr[X_{i} = 1] = \frac{1}{n}. In this case, however, we have to be a bit more careful about E(X_{i}X_{j}): note that we cannot claim as before that this is equal to E(X_{i})E(X_{j}), because X_{i} and X_{j} are not independent (why not?). But since both X_{i} and X_{j} are indicators, we can compute E(X_{i}X_{j}) directly as follows:

E(X_{i}X_{j}) = Pr[X_{i} = 1 ∧ X_{j} = 1] = Pr[both i and j are fixed points] = \frac{1}{n(n − 1)} .

[Check that you understand the last step here.] Plugging this into equation (1) we get

E(X^{2}) = (n × \frac{1}{n}) + (n(n − 1) × \frac{1}{n(n−1)} ) = 1 + 1 = 2.

Thus Var(X) = E(X^{2}) − (E(X))^{2} = 2 − 1 = 1. I.e., the variance and the mean are both equal to 1. Like the mean, the variance is also independent of n. Intuitively at least, this means that it is unlikely that there will be more than a small number of fixed points even when the number of items, n, is very large.

Chebyshev’s Inequality

We have seen that, intuitively, the variance (or, more correctly, the standard deviation) is a measure of “spread”, or deviation from the mean. Our next goal is to make this intuition quantitatively precise. What we can show is the following:

Theorem 19.2: [Chebyshev’s Inequality] For a random variable X with expectation E(X) = μ, and for any α > 0,

Pr[|X − μ| ≥ α] ≤ \frac{Var(X)}{α^{2}}.

Before proving Chebyshev’s inequality, let’s pause to consider what it says. It tells us that the probability of any given deviation, α, from the mean, either above it or below it (note the absolute value sign), is at most \frac{Var(X)}{α^{2}}. As expected, this deviation probability will be small if the variance is small. An immediate corollary of Chebyshev’s inequality is the following:

Corollary 19.3: For a random variable X with expectation E(X) = μ, and standard deviation σ = \sqrt{Var(X)},

Pr[|X − μ| ≥ βσ] ≤ \frac{1}{β^{2}} .

Proof: Plug α = βσ into Chebyshev’s inequality. □

So, for example, we see that the probability of deviating from the mean by more than (say) two standard deviations on either side is at most \frac{1}{4}. In this sense, the standard deviation is a good working definition of the “width” or “spread” of a distribution.

We should now go back and prove Chebyshev’s inequality. The proof will make use of the following simpler bound, which applies only to non-negative random variables (i.e., r.v.’s which take only values ≥ 0).

Theorem 19.4: [Markov’s Inequality] For a non-negative random variable X with expectation E(X) = μ, and any α > 0,

Pr[X ≥ α] ≤ \frac{E(X)}{α}.

Proof: From the definition of expectation, we have

E(X)= ∑_{a}a × Pr[X = a]
≥ ∑_{a≥α} a × Pr[X = a]
≥ α ∑_{a≥α} Pr[X = a]
= α Pr[X ≥ α].

The crucial step here is the second line, where we have used the fact that X takes on only non-negative values. (Why is this step not valid otherwise?) □

Now we can prove Chebyshev’s inequality quite easily.

Proof of Theorem 22.2: Define the r.v. Y = (X − μ)^{2}. Note that E(Y) = E((X − μ)^{2}) = Var(X). Also, notice that the probability we are interested in, Pr[|X − μ| ≥ α], is exactly the same as Pr[Y ≥ α^{2}]. (Why?) Moreover, Y is obviously non-negative, so we can apply Markov’s inequality to it to get

Pr[Y ≥ α^{2}] ≤ \frac{E(Y)}{α^{2}}= \frac{Var(X)}{α^{2}}.

This completes the proof. □

Let’s apply Chebyshev’s inequality to answer our question about the random walk at the beginning of the lecture. Recall that X is our position after n steps, and that E(X) = 0, Var(X) = n. Corollary 22.3 says that, for any β > 0, Pr[|X| ≥ β\sqrt{n}] ≤ \frac{1}{β^{2}}. Thus for example, if we take n = 10^{6} steps, the probability that we end up more than 10000 steps away from our starting point is at most \frac{1}{100}.

Here are a few more examples of applications of Chebyshev’s inequality (you should check the algebra in them):

1. Coin tosses. Let X be the number of Heads in n tosses of a fair coin. The probability that X deviates from μ = \frac{n}{2} by more than \sqrt{n} is at most \frac{1}{4}. The probability that it deviates by more than 5\sqrt{n} is at most \frac{1}{100}.

2. Poisson distribution. Let X be a Poisson r.v. with parameter λ. The probability that X deviates from λ by more than 2\sqrt{λ} is at most \frac{1}{4}.

3. Fixed points. Let X be the number of fixed points in a random permutation of n items; recall that E(X) = Var(X) = 1. Thus the probability that more than (say) 10 students get their own homeworks after shuffling is at most \frac{1}{100}, however large n is.