CS 70 Discrete Mathematics for CS

Fall 2004  Rao  Lecture 20

I.I.D. Random Variables

Estimating the bias of a coin

Question: We want to estimate the proportion p of Democrats in the US population, by taking a small random sample. How large does our sample have to be to guarantee that our estimate will be within (say) 10% (in relative terms) of the true value with probability at least 0.95?

This is perhaps the most basic statistical estimation problem, and shows up everywhere. We will develop a simple solution that uses only Chebyshev’s inequality. More refined methods can be used to get sharper results.

Let’s denote the size of our sample by n (to be determined), and the number of Democrats in it by the random variable S_{n}. (The subscript n just reminds us that the r.v. depends on the size of the sample.) Then our estimate will be the value A_{n} = \frac{1}{n}S_{n}.

Now as has often been the case, we will find it helpful to write S_{n} = X_{1} + X_{2} + ···+ X_{n}, where X_{i} =

\{ \begin{matrix}1 & if person i in sample is a Democrat; \\ 0 & otherwise. \end{matrix}

Note that each X_{i} can be viewed as a coin toss, with Heads probability p (though of course we do not know the value of p: this is what we’re trying to estimate!). And the coin tosses are independent.^{1}

What is the expectation of our estimate?

E(A_{n}) = E(\frac{1}{n}S_{n}) = \frac{1}{n}E(X_{1}+ X_{2}+ ··· + X_{n}) = \frac{1}{n} × (np) = p.

So for any value of n, our estimate will always have the correct expectation p. [Such a r.v. is often called an unbiased estimator of p.] Now presumably, as we increase our sample size n, our estimate should get more and more accurate. This will show up in the fact that the variance decreases with n: i.e., as n increases, the probability that we are far from the mean p will get smaller.

To see this, we need to compute Var(A_{n}). And since A_{n} = \frac{1}{n}∑ _{i=1}^{n} X_{i}, we need to figure out how to compute the variance of a sum of random variables.

Theorem 20.1: For any random variable X and constant c, we have

Var(cX) = c^{2}Var(X).

And for independent random variables X,Y, we have

Var(X + Y) = Var(X) + Var(Y).

1We are assuming here that the sampling is done “with replacement”; i.e., we select each person in the sample from the entire population, including those we have already picked. So there is a small chance that we will pick the same person twice.

CS 70, Fall 2004, Lecture 20 1

Proof: From the definition of variance, we have

Var(cX) = E((cX − E(cX))^{2}) = E((cX − cE(X))^{2}) = E(c^{2}(X − E(X))^{2}) = c^{2}Var(X).

The proof of the second claim is left as an exercise. Note that the second claim does not in general hold unless X and Y are independent. □

Using Theorem 23.1, we can now compute Var(A_{n}):

Var(A_{n}) = Var(\frac{1}{n} ∑_{i=1}^{n} X_{i}) = (\frac{1}{n})^{2}Var(∑_{i=1}^{n} X_{i}) = (\frac{1}{n})^{2} ∑_{i=1}^{n} Var(X_{i}) = \frac{σ^{2}}{n} ,

where we have written σ^{2} for the variance of each of the X_{i}. So we see that \underline{the variance of A_{n} decreases} \underline{linearly with n}. This fact ensures that, as we take larger and larger sample sizes n, the probability that we deviate much from the expectation p gets smaller and smaller.

Let’s now use Chebyshev’s inequality to figure out how large n has to be to ensure a specified accuracy in our estimate of the proportion of Democrats p. A natural way to measure this is for us to specify two parameters, ϵ and δ, both in the range (0,1). The parameter ϵ controls the error we are prepared to tolerate in our estimate, and δ controls the confidence we want to have in our estimate. A more precise version of our original question is then the following:

Question: For the Democrat-estimation problem above, how large does the sample size n have to be in order to ensure that

Pr[|A_{n}− p| ≥ ϵp] ≤ δ ?

In our original question, we had ϵ = 0.1 and δ = 0.05. Notice that ϵ measures the relative error, i.e., the error as a ratio of the target value p. This seems like a more reasonable convention than the absolute error (based on Pr[|A_{n}− p| ≥ ϵ]). This is because a given absolute error (say, ±0.1) might be quite small in the context of measuring a large value like p = 0.5, but very large when measuring a small value like p = 0.05. In contrast, the relative error treats all values of p equally.

Let’s apply Chebyshev’s inequality to answer our more precise question above. Since we know Var(A_{n}), this will be quite simple. From Chebyshev’s inequality, we have

Pr[|A_{n}− p| ≥ ϵp] ≤ \frac{Var(A_{n})}{(ϵp)^{2}}= \frac{σ^{2}}{np^{2}ϵ^{2}}.

To make this less than the desired value δ, we need to set

n ≥ \frac{σ^{2}}{p^{2}} × \frac{1}{ϵ^{2}δ}.  (1)

Now recall that σ^{2} = Var(X_{i}) is the variance of a single sample X_{i}. So, since X_{i} is a 0/1-valued r.v., we have σ^{2} = p(1 − p), and inequality (1) becomes

n ≥ \frac{1 − p}{p} × \frac{1}{ϵ^{2}δ}.  (2)

Plugging in ϵ = 0.1 and δ = 0.05, we see that a sample size of n = 2000 × \frac{1−p}{p} is sufficient.

At this point you should be worried. Why? Because our formula for the sample size contains p, and this is precisely the quantity we are trying to estimate! But we can get around this as follows. We just pick a value p′that we know for sure is less than p. (For example, in the Democrats problem we could certainly take p′= \frac{1}{3}.) Then we use p′in place of p in equation (2). Since p′is less than p, this will always lead us to take at least enough samples (why?). In the Democrats example, with p′= \frac{1}{3}, this means we would take a sample size of n = 2000 × 2 = 4000.

Estimating a general expectation

What if we wanted to estimate something a little more complex than the proportion of Democrats in the population, such as the average wealth of people in the US? Then we could use exactly the same scheme as above, except that now the r.v. X_{i} is the wealth of the ith person in our sample. Clearly E(X_{i}) = μ, the average wealth (which is what we are trying to estimate). And our estimate will again be A_{n} = \frac{1}{n}∑ _{i=1}^{n} X_{i}, for a suitably chosen sample size n. Once again the X_{i} are independent, and all have the same distribution: such a collection of r.v.’s is usually called independent and identically distributed, or i.i.d. for short. As before we have E(A_{n}) = μ and Var(A_{n}) = \frac{σ^{2}}{n}, where σ^{2} = Var(X_{i}) is the variance of the X_{i}. (Recall that the only facts we used about the X_{i} was that they were independent and had the same distribution.)

From equation (1), it is enough for the sample size n to satisfy

n ≥ \frac{σ^{2}}{μ^{2}} × \frac{1}{ϵ^{2}δ}.  (3)

Here ϵ and δ are the desired error and confidence respectively, as before. Now of course we don’t know the other two quantities, μ and σ^{2}, appearing in equation (3). In practice, we would use a lower bound on μ and an upper bound on σ^{2} (just as we used a lower bound on p in the Democrats problem). Plugging these bounds into equation (3) will ensure that our sample size is large enough.

For example, in the average wealth problem we could probably safely take μ to be at least (say) $20k (probably more). However, the existence of people such as Bill Gates means that we would need to take a very high value for the variance σ^{2}. Indeed, if there is at least one individual with wealth $50 billion, then assuming a relatively small value of μ means that the variance must be at least about \frac{(50×10^{9})^{2}}{250×10^{6}} = 10^{13}. (Check this.) However, this individual’s contribution to the mean is only \frac{50×10^{9}}{250×10^{6}} = 200. There is really no way around this problem with simple uniform sampling: the uneven distribution of wealth means that the variance is inherently very large, and we will need a huge number of samples before we are likely to find anybody who is immensely wealthy. But if we don’t include such people in our sample, then our estimate will be way too low.

As a further example, suppose we are trying to estimate the average rate of emission from a radioactive source, and we are willing to assume that the emissions follow a Poisson distribution with some unknown parameter λ — of course, this λ is precisely the expectation we are trying to estimate. Now in this case we have μ = λ and also σ^{2} = λ (see the previous lecture). So \frac{σ^{2}}{μ^{2}} = \frac{1}{λ}. Thus in this case a sample size of n = \frac{1}{λϵ^{2}δ} suffices. As an example, to estimate the mean number of chocolate chips in a cookie, with relative error 0.1 and confidence 95%, and assuming that λ ≥ 4 (i.e., the mean is at least 4), it would suffice to take 500 samples.

The Law of Large Numbers

The estimation method we used in the previous two sections is based on a principle that we accept as part of everyday life: namely, the Law of Large Numbers. This asserts that, if we observe some random variable many times, and take the average of the observations, then this average will converge to a single value, which is of course the expectation of the random variable. In other words, \underline{averaging tends to smoothe out any large fluctuations}, and the more averaging we do the better the smoothing.

Theorem 20.2: [Law of Large Numbers] Let X_{1},X_{2},...,X_{n} be i.i.d. random variables with common expectation μ = E(X_{i}). Define A_{n} = \frac{1}{n}∑ _{i=1}^{n} X_{i}. Then for any α > 0, we have

Pr[|A_{n}− μ| ≥ α] → 0 as n → ∞ .

We will not prove this theorem here. Notice that it says that the probability of any deviation α from the mean, however small, tends to zero as the number of observations n in our average tends to infinity. Thus by taking n large enough, we can make the probability of any given deviation as small as we like. [Note, however, that the Law of Large Numbers does not say anything about how large n has to be to achieve a certain accuracy. For that, we need Chebyshev’s inequality or some other quantitative tool.]

Actually we can say something much stronger than the Law of Large Numbers: namely, the distribution of the sample average A_{n}, for large enough n, looks like a bell-shaped curve centered about the mean μ. The width of this curve decreases with n, so it approaches a sharp spike at μ. This fact is known as the \underline{Central Limit Theorem}.

To say this precisely, we need to define the “bell-shaped curve.” This is the so-called \underline{Normal distribution}, and it is the first (and only) non-discrete distribution we will meet in this course. For random variables that take on continuous real values, it no longer makes sense to talk about Pr[X = a]. As an example, consider a r.v. X that has the uniform distribution on the continuous interval [0,1]. Then for any single point 0 ≤ a ≤ 1, we have Pr[X = a] = 0. However, clearly it is the case that, for example, Pr[\frac{1}{4} ≤ X ≤ \frac{3}{4}] = \frac{1}{2}. So in place of point probabilities Pr[X = a], we need a different notion of “distribution” for continuous random variables.

Definition 20.1 (density function): For a real-valued r.v. X, a real-valued function f(x) is called a \underline{(probability) density function} for X if

Pr[X ≤ a] = ∫_{−∞}^{a} f(x)dx.

Thus we can think of f(x) as defining a curve, such that the area under the curve between points x = a and x = b is precisely Pr[a ≤ X ≤ b]. Note that we must always have ∫_{−∞}^{∞} f(x)dx = 1. (Why?) As an example, for the uniform distribution on [0,1] the density would be

f(x) = \{ \begin{matrix}0 & for x < 0; \\ 1 & for 0 ≤ x ≤ 1; \\ 0 & for x > 1. \end{matrix}

[Check you agree with this. What would be the density for the uniform distribution on [−1,1]?]

Expectations of continuous r.v.’s are computed in an analagous way to those for discrete r.v.’s. Thus

E(X) = ∫_{−∞}^{∞} xf(x)dx.

And also

Var(X) = E(X^{2}) − (E(X))^{2}, where E(X^{2}) = ∫_{−∞}^{∞} x^{2}f(x)dx.

[You should check that, for the uniform distribution on [0,1], the expectation is \frac{1}{2} and the variance is \frac{1}{12}.]

Now we are in a position to define the Normal distribution.

Definition 20.2 (Normal distribution): The \underline{Normal distribution with mean μ and variance σ^{2}} is the distribution with density function

f(x) = \frac{1}{σ\sqrt{2π}} e^{−(x−μ)^{2}/2σ^{2}} .

CS 70, Fall 2004, Lecture 20 4

[The reason for the constant factor \frac{1}{σ\sqrt{2π}} is that this is the value of the integral of the exponential function

e^{−(x−μ)^{2}/σ^{2}} . So we have to normalize by this value to get ∫_{−∞}^{∞} f(x)dx = 1. If you enjoy calculus, you might like to do the integrals to check that the expectation ∫ xf(x)dx is indeed μ and that the variance is indeed σ^{2}.]

If you plot the above density function f(x), you will see that it is a symmetrical bell-shaped curve centered around the mean μ. Its height at the mean is about 0.4, and its width is determined by the variance σ^{2}, as follows: 50% of the mass is contained in the interval of width 0.67σ either side of the mean, and 99.7% in the interval of width 3σ either side of the mean. (Note that, to get the correct scale, deviations are on the order of σ rather than σ^{2}.)

Now we are in a position to state the Central Limit Theorem. Because our treatment of continuous distributions has been rather sketchy, we shall be content with a rather imprecise statement. This can be made completely rigorous without too much extra effort.

Theorem 20.3: [Central Limit Theorem] Let X_{1},X_{2},...,X_{n} be i.i.d. random variables with common expectation μ = E(X_{i}) and variance σ^{2} = Var(X_{i}) (both assumed to be < ∞). Define A_{n} = \frac{1}{n}∑ _{i=1}^{n} X_{i}. Then as n → ∞, the distribution of A_{n} approaches the Normal distribution with mean μ and variance \frac{σ^{2}}{n}.

Note that the variance is \frac{σ^{2}}{n} (as we would expect) so the width of the bell-shaped curve decreases by a factor of \sqrt{n}as n increases.

The Central Limit Theorem is actually a very striking fact. What it says is the following. If we take an average of n observations of any r.v. X, then the distribution of that average will be a bell-shaped curve centered at μ = E(X). Thus all trace of the distribution of X disappears as n gets large: all distributions, no matter how complex,^{2} look like the Normal distribution when they are averaged. The only effect of the original distribution is through the variance σ^{2}, which determines the width of the curve for a given value of n, and hence the rate at which the curve shrinks to a spike.

In class, we saw how the distribution of A_{n} behaves for increasing values of n, when the X_{i} have the geometric distribution with parameter \frac{1}{6}. As an exercise, try doing the same thing for several different r.v.’s X_{i}, all of which have the same mean but very different distributions. You should observe the appearance of the bellshaped curve in all cases (with a width determined by the variance of the particular X_{i}).

2Strictly speaking, we do need to assume that the mean and variance of X are finite.

CS 70, Fall 2004, Lecture 20 5