CS 70 Discrete Mathematics for CS

Fall 2004  Rao  Lecture 16

Conditional Probability

A pharmaceutical company is marketing a new test for a certain medical condition. According to clinical trials, the test has the following properties:

1. When applied to an affected person, the test comes up positive in 90% of cases, and negative in 10% (these are called “false negatives”).

2. When applied to a healthy person, the test comes up negative in 80% of cases, and positive in 20% (these are called “false positives”).

Suppose that the incidence of the condition in the US population is 5%. When a random person is tested and the test comes up positive, what is the probability that the person actually has the condition? (Note that this is presumably not the same as the simple probability that a random person has the condition, which is just \frac{1}{20}.)

This is an example of a \underline{conditional probability}: we are interested in the probability that a person has the condition (event A) given that he/she tests positive (event B). Let’s write this as Pr[A|B].

How should we compute Pr[A|B]? Well, since event B is guaranteed to happen, we need to look not at the whole sample space Ω, but at the smaller sample space consisting only of the sample points in B. What should the probabilities of these sample points be? If they all simply inherit their probabilities from Ω, then the sum of these probabilities will be ∑_{ω∈B} Pr[ω] = Pr[B], which in general is less than 1. So we need to scale the probability of each sample point by \frac{1}{Pr[B]}. I.e., for each sample point ω ∈ B, the new probability becomes

Pr[ω|B] = \frac{Pr[ω]}{Pr[B]}.

Now it is clear how to compute Pr[A|B]: namely, we just sum up these scaled probabilities over all sample points that lie in both A and B:

Pr[A|B] = ∑_{ω∈A∩B} Pr[ω|B] = ∑_{ω∈A∩B} \frac{Pr[ω]}{Pr[B]}= \frac{Pr[A ∩ B]}{Pr[B]}.

Definition 16.1 (conditional probability): For events A,B in the same probability space, such that Pr[B] > 0, the \underline{conditional probability of A given B}is

Pr[A|B] = \frac{Pr[A ∩ B]}{Pr[B]}.

Let’s go back to our medical testing example. The sample space here consists of all people in the US — denote their number by N (so N ≈ 250 million). The population consists of four disjoint subsets:

T P: the true positives (90% of \frac{N}{20} = \frac{9N}{200} of them);

F P: the false positives (20% of \frac{19N}{20} = \frac{19N}{100} of them);

T N: the true negatives (80% of \frac{19N}{20} = \frac{76N}{100} of them);

F N: the false negatives (10% of \frac{N}{20} = \frac{N}{200} of them).

Now let A be the event that a person chosen at random is affected, and B the event that he/she tests positive. Note that B is the union of the disjoint sets T P and F P, so

|B| = |T P| + |F P| = \frac{9N}{200} + \frac{19N}{100} = \frac{47N}{200} .

Thus we have

Pr[A] = \frac{1}{20} and Pr[B] = \frac{47}{200}.

Now when we condition on the event B, we focus in on the smaller sample space consisting only of those \frac{47N}{200} individuals who test positive. To compute Pr[A|B], we need to figure out Pr[A ∩ B] (the part of A that lies in B). But A ∩ B is just the set of people who are both affected and test positive, i.e., A ∩ B = T P. So we have

Pr[A ∩ B] = \frac{|T P|}{N}= \frac{9}{200}.

Finally, we conclude from Definition 17.1 that

Pr[A|B] = \frac{Pr[A ∩ B]}{Pr[B]}= \frac{9/200}{47/200} = \frac{9}{47} ≈ 0.19.

This seems bad: if a person tests positive, there’s only about a 19% chance that he/she actually has the condition! This sounds worse than the original claims made by the pharmaceutical company, but in fact it’s just another view of the same data.

[Incidentally, note that Pr[B|A] = \frac{9/200}{1/20} = \frac{9}{10}; so Pr[A|B] and Pr[B|A] can be very different. Of course, Pr[B|A] is just the probability that a person tests positive given that he/she has the condition, which we knew from the start was 90%.]

To complete the picture, what’s the (unconditional) probability that the test gives a correct result (positive or negative) when applied to a random person? Call this event C. Then

Pr[C] = \frac{|T P|+|T N|}{N}= \frac{9}{200} + \frac{76}{100} = \frac{161}{200} ≈ 0.8.

So the test is about 80% effective overall, a more impressive statistic.

But how impressive is it? Suppose we ignore the test and just pronounce everybody to be healthy. Then we would be correct on 95% of the population (the healthy ones), and wrong on the affected 5%. I.e., this trivial test is 95% effective! So we might ask if it is worth running the test at all. What do you think?

Here are a couple more examples of conditional probabilities, based on some of our sample spaces from the previous lecture.

1. Balls and bins. Suppose we toss m = 3 balls into n = 3 bins; this is a uniform sample space with 3^{3} = 27 points. We already know that the probability the first bin is empty is (1 − \frac{1}{3})^{3} = (\frac{2}{3})^{3} = \frac{8}{27}. What is the probability of this event given that the second bin is empty? Call these events A,B respectively. To compute Pr[A|B] we need to figure out Pr[A ∩ B]. But A ∩ B is the event that both the first two bins are empty, i.e., all three balls fall in the third bin. So Pr[A ∩ B] = \frac{1}{27} (why?). Therefore,

Pr[A|B] = \frac{Pr[A ∩ B]}{Pr[B]}= \frac{1/27}{8/27} = \frac{1}{8}.

Not surprisingly, \frac{1}{8} is quite a bit less than \frac{8}{27}: knowing that bin 2 is empty makes it significantly less likely that bin 1 will be empty.

2. Dice. Roll two fair dice. Let A be the event that their sum is even, and B the event that the first die is even. By symmetry it’s easy to see that Pr[A] = \frac{1}{2} and Pr[B] = \frac{1}{2}. Moreover, a little counting gives us that Pr[A ∩ B] = \frac{1}{4}. What is Pr[A|B]? Well,

Pr[A|B] = \frac{Pr[A ∩ B]}{Pr[B]}= \frac{1/4}{1/2} = \frac{1}{2}.

In this case, Pr[A|B] = Pr[A], i.e., conditioning on B does not change the probability of A.

Independent events

Definition 16.2 (independence): Two events A,B in the same probability space are \underline{independent}if Pr[A|B] = Pr[A].

Note that independence is symmetric: i.e., if Pr[A|B] = Pr[A] then it must also be the case that Pr[B|A] = Pr[B]. To see this, use the definition of conditional probabilities:

Pr[B|A] = \frac{Pr[A ∩ B]}{Pr[A]}= \frac{Pr[A ∩ B]}{Pr[B]}× \frac{Pr[B]}{Pr[A]}= \frac{Pr[A|B]}{Pr[A]}× Pr[B] = Pr[B].

In the last step here, we used our assumption that Pr[A|B] = Pr[A]. [We are assuming here that Pr[A] and Pr[B] are both > 0. Otherwise the conditional probabilities are not defined.]

Examples: In the balls and bins example above, events A,B are not independent. In the dice example, events A,B are independent.

Knowing that events are independent is very useful, because of the following simple observation:

Theorem 16.1: If A,B are independent, then Pr[A ∩ B] = Pr[A]Pr[B].

Proof: From the definition of conditional probability we have

Pr[A ∩ B] = Pr[A|B]Pr[B] = Pr[A]Pr[B],

where in the second step we have used independence. □

Note that the condition in Theorem 17.1 actually holds if and only if A and B are independent. In fact, this condition is often given as the definition of independence, rather than the definition we are using.

All the above generalizes to any finite set of events:

Definition 16.3 (mutual independence): Events A_{1},...,A_{n} are \underline{mutually independent} if for every 1 ≤ i ≤ n and every subset I ⊆ {1,...,n} − {i},

Pr[A_{i}|⋂_{j∈I} A_{j}] = Pr[A_{i}] .

I.e., the probability of A_{i} does not depend on any combination of the other events.

Theorem 16.2: If events A_{1},...,A_{n} are mutually independent, then

Pr[A_{1}∩ ...∩ A_{n}] = Pr[A_{1}] × Pr[A_{2}] × ··· × Pr[A_{n}].

We won’t prove this theorem here because it is a special case of the more general Theorem 17.3, which we will prove below (check this!). Note that it is possible to construct three events A,B,C such that each pair is independent but the triple A,B,C is not mutually independent.

Combinations of events

In most applications of probability in Computer Science, we are interested in things like Pr[⋃ _{i=1}^{n} A_{i}] and Pr[⋂ _{i=1}^{n} A_{i}], where the A_{i} are simple events (i.e., we know, or can easily compute, the Pr[A_{i}]). The intersection ⋂_{i}A_{i} corresponds to the logical and of the events A_{i}, while the union ⋃_{i}A_{i} corresponds to their logical or. As an example, if A_{i} denotes the event that a failure of type i happens in a certain system, then ⋃_{i}A_{i} is the event that the system fails.

In general, computing the probabilities of such combinations can be very difficult. In this section, we discuss some situations where it can be done.

Intersections of events

From the definition of conditional probability, we immediately have the following \underline{product rule} (sometimes also called the \underline{chain rule}) for computing the probability of an intersection of events.

Theorem 16.3: [Product Rule] For events A,B, we have

Pr[A ∩ B] = Pr[A]Pr[B|A].

More generally, for events A_{1},...,A_{n},

Pr[⋂ _{i=1}^{n} A_{i}] = Pr[A_{1}] × Pr[A_{2}|A_{1}] × Pr[A_{3}|A_{1}∩ A_{2}] × ··· × Pr[A_{n}|⋂ _{i=1}^{n−1} A_{i}] .

Proof: The first assertion follows directly from the definition of Pr[B|A] (and is in fact a special case of the second assertion with n = 2).

To prove the second assertion, we will use simple induction on n (the number of events). The base case is n = 1, and corresponds to the statement that Pr[A] = Pr[A], which is trivially true. For the inductive step, let n > 1 and assume (the inductive hypothesis) that

Pr[⋂ _{i=1}^{n−1} A_{i}] = Pr[A_{1}] × Pr[A_{2}|A_{1}] × ··· × Pr[A_{n−1} |⋂ _{i=1}^{n−2} A_{i}].

Now we can apply the definition of conditional probability to the two events A_{n} and ⋂ _{i=1}^{n−1} A_{i} to deduce that

Pr[⋂ _{i=1}^{n} A_{i}] = Pr[A_{n}∩ (⋂ _{i=1}^{n−1} A_{i})]= Pr[A_{n}|⋂ _{i=1}^{n−1} A_{i}] × Pr[⋂ _{i=1}^{n−1} A_{i}]
= Pr[A_{n}|⋂ _{i=1}^{n−1} A_{i}] × Pr[A_{1}] × Pr[A_{2}|A_{1}] × ··· × Pr[A_{n−1} |⋂ _{i=1}^{n−2} A_{i}],

where in the last line we have used the inductive hypothesis. This completes the proof by induction. □

Note that Theorems 17.1 and 17.2 are special cases of the product rule for independent events.

Sequences of trials

Many experiments can be viewed as sequences of simpler experiments, or trials. In these cases, it is more natural to define the probability space in terms of conditional probabilities, using the product rule. As an illustration, consider the sample space Ω of n tosses of a biased coin discussed in the previous lecture. We can write Ω as the product Ω = Ω_{1}× Ω_{2}× ...× Ω_{n}, where Ω_{i} = {H,T } is the sample space of the ith coin toss.^{1}

How should we define probabilities in Ω? Each sample point in Ω is an n-tuple ω = (ω_{1},ω_{2},...,ω_{n}), where ω_{i} ∈ Ω_{i} is the outcome of the ith toss. Using the product rule, we must have

Pr[ω] = Pr[ω_{1}] × Pr[ω_{2}|ω_{1}] × ··· × Pr[ω_{n}|ω_{1},ω_{2},...,ω_{n−1} ].  (1)

So if we define all the conditional probabilities Pr[ω_{i}|ω_{1},ω_{2},...,ω_{i−1} ], we will in fact have defined the entire probability space!

Now the key fact in this example is that the coin tosses are supposed to be independent, so each conditional probability is just the same as the corresponding unconditional probability for a coin. Thus equation (1) becomes

Pr[ω] = Pr[ω_{1}] × Pr[ω_{2}] × ··· × Pr[ω_{n}].

Thus we see that the probability of any sample point ω is Pr[ω] = p^{r}(1 − p)^{n−r}, where r is the number of Heads in ω. Of course, this is exactly how we defined this sample space in the previous lecture. However, the point is that now we have a rational basis for our definition: it is an inevitable consequence of the fact that the coin tosses are independent.

Here are some more examples.

1. Coin tosses. Toss a fair coin three times. Let A be the event that all three tosses are heads. Then A = A_{1}∩ A_{2}∩ A_{3}, where A_{i} is the event that the ith toss comes up heads. We have

Pr[A] = Pr[A_{1}] × Pr[A_{2}|A_{1}] × Pr[A_{3}|A_{1}∩ A_{2}]

= Pr[A_{1}] × Pr[A_{2}] × Pr[A_{3}]
= \frac{1}{2} × \frac{1}{2} × \frac{1}{2} = \frac{1}{8}.

The second line here follows from the fact that the tosses are mutually independent. Of course, we already know that Pr[A] = \frac{1}{8} from our definition of the probability space in the previous lecture. The above is really a check that the space behaves as we expect.^{2}

If the coin is biased with heads probability p, we get, again using independence,

Pr[A] = Pr[A_{1}]Pr[A_{2}]Pr[A_{3}] = p^{3}.

And more generally, the probability of any sequence of n tosses containing r heads and n − r tails is p^{r}(1 − p)^{n−r}. This is in fact the reason we defined the probability space this way in the previous lecture: we defined the sample point probabilities so that the coin tosses would behave independently.

2. Balls and bins. The sample space here is the product Ω = Ω_{1}×Ω_{2}×··· ×Ω_{m}, where Ω_{i} = {1,2,...,n} is the set of n bins that could be chosen by the ith ball. (Recall that there are m balls and n bins.) Since we specify that the balls are thrown independently, by similar reasoning to the above we must have Pr[ω] = Pr[ω_{1}] × ··· × Pr[ω_{n}] = (\frac{1}{n})^{m} for each sample point ω. This is again in agreement with our definition of this probability space in the previous lecture.

1Recall that, for sets A,B, the Cartesian product A × B is the set consisting of all (ordered) pairs (a,b) with a ∈ A and b ∈ B.

2Strictly speaking, we should really also have checked from our original definition of the probability space that Pr[A_{1}], Pr[A_{2}|A_{1}] and Pr[A_{3}|A_{1}∩ A_{2}] are all equal to \frac{1}{2}.

CS 70, Fall 2004, Lecture 16 5

The product rule also gives us a new way to compute probabilities of events. Let A be the event that bin 1 is empty. Then we can write A = ⋂ _{i=1}^{m} A_{i}, where A_{i} is the event that ball i misses bin 1. Clearly Pr[A_{i}] = 1− \frac{1}{n} for each i. Also, by construction of our sample space the events A_{i} are mutually independent. So by the product rule

Pr[A] = Pr[A_{1}] × Pr[A_{2}|A_{1}] × ··· × Pr[A_{n}|⋂ _{i=1}^{n−1} A_{i}] = Pr[A_{1}] × Pr[A_{2}] × ··· × Pr[A_{n}] = ( 1 − \frac{1}{n}) ^{m}.

This agrees with the answer we obtained in the previous lecture by counting sample points.

So now we have two methods of computing probabilities in many of our sample spaces. It is useful to keep these different methods around, both as a check on your answers and because in some cases one of the methods is easier to use than the other.

3. Card shuffling. Each sample point ω can be viewed as a sequence of 52 choices, ω = (ω_{1},ω_{2},...,ω_{52}), where ω_{i} is the ith card of the deck. First we pick the top card, ω_{1}, from all 52 cards; then we pick the second card, ω_{2}, from the remaining 51 cards; then ω_{3} from the remaining 50 cards; and so on.^{3} The probability of choosing each ω_{1} is clearly \frac{1}{52}; given ω_{1}, the probability of choosing each possible ω_{2} is uniform, namely \frac{1}{51}; given ω_{1} and ω_{2}, the probability of each possible ω_{3} is \frac{1}{50}; and so on. These conditional probabilities imply, using the product rule:

Pr[ω] = Pr[ω_{1}] × Pr[ω_{2}|ω_{1}] × ...× Pr[ω_{52}|ω_{1},...,ω_{51}] = \frac{1}{52} × \frac{1}{51} × \frac{1}{50} × ...× \frac{1}{2} × \frac{1}{1} = \frac{1}{52!}.

Reassuringly, this is in agreement with our definition of the probability space in the previous lecture, based on counting permutations.

Let A be the event that the top card is an ace. Then we see immediately, just by considering the first choice ω_{1}, that Pr[A] = \frac{4}{52} = \frac{1}{13}. And if B is the event that the top two cards have the same suit, then we can view B as the event that the second choice produces the same suit as the first. This can be calculated as follows:

Pr[suit(ω_{2}) = suit(ω_{1})] = ∑_{ω_{1}} Pr[ω_{1}] × Pr[suit(ω_{2}) = suit(ω_{1})|ω_{1}] = ∑_{ω_{1}} Pr[ω_{1}] × \frac{12}{51} = \frac{12}{51}.

4. Poker hands. Again we can view the sample space as a sequence of choices. First we choose one of the cards (note that it is not the “first” card, since the cards in our hand have no ordering) uniformly from all 52 cards. Then we choose another card from the remaining 51, and so on. For any given poker hand, the probability of choosing it is (by the product rule):

\frac{5}{52} × \frac{4}{51} × \frac{3}{50} × \frac{2}{49} × \frac{1}{48} = \frac{1}{\binom{52}{5}},

just as before. Where do the numerators 5, 4, 3, 2, 1 come from? Well, for the given hand the first card we choose can be any of the five in the hand: i.e., five choices out of 52. The second can be any of the remaining four in the hand: four choices out of 51. And so on. This arises because the order of the cards in the hand is irrelevant.

3Note that we can’t write Ω = Ω_{1} × Ω_{2} × ...× Ω_{52} here since the sample space Ω_{i} depends on ω_{1},ω_{2},ω_{i−1} . (E.g., Ω_{2}, the sample space for the second card, consists of all cards except ω_{1}.)

CS 70, Fall 2004, Lecture 16 6

Let’s use this view to compute the probability of a flush in a different way. Clearly this is 4 × Pr[A], where A is the probability of a Hearts flush. And we can write A = ⋂ _{i=1}^{5} A_{i}, where A_{i} is the event that the ith card we pick is a Heart. So we have

Pr[A] = Pr[A_{1}] × Pr[A_{2}|A_{1}] × ··· × Pr[A_{5}|⋂ _{i=1}^{4} A_{i}] .

Clearly Pr[A_{1}] = \frac{13}{52} = \frac{1}{4}. What about Pr[A_{2}|A_{1}]? Well, since we are conditioning on A_{1} (the first card is a Heart), there are only 51 remaining possibilities for the second card, 12 of which are Hearts. So Pr[A_{2}|A_{1}] = \frac{12}{51}. Similarly, Pr[A_{3}|A_{1}∩ A_{2}] = \frac{11}{50}, and so on. So we get

4 × Pr[A] = 4 × \frac{13}{52} × \frac{12}{51} × \frac{11}{50} × \frac{10}{49} × \frac{9}{48},

which is exactly the same fraction we computed in the previous lecture.

So now we have two methods of computing probabilities in many of our sample spaces. It is useful to keep these different methods around, both as a check on your answers and because in some cases one of the methods is easier to use than the other.

5. Monty Hall. Recall that we defined the probability of a sample point by multiplying the probabilities of the sequence of choices it corresponds to; thus, e.g.,

Pr[(1,1,2)] = \frac{1}{3} × \frac{1}{3} × \frac{1}{2} = \frac{1}{18}.

The reason we defined it this way is that we knew (from our model of the problem) the probabilities for each choice conditional on the previous one. Thus, e.g., the \frac{1}{2} in the above product is the probability that Monty opens door 2 conditional on the prize door being door 1 and the contestant initially choosing door 1. Once again, we used these conditional probabilities to define the probabilities of our sample points.

Unions of events

You are in Las Vegas, and you spy a new game with the following rules. You pick a number between 1 and 6. Then three dice are thrown. You win if and only if your number comes up on at least one of the dice.

The casino claims that your odds of winning are 50%, using the following argument. Let A be the event that you win. We can write A = A_{1}∪ A_{2}∪ A_{3}, where A_{i} is the event that your number comes up on die i. Clearly Pr[A_{i}] = \frac{1}{6} for each i. Therefore,

Pr[A] = Pr[A_{1}∪ A_{2}∪ A_{3}] = Pr[A_{1}] + Pr[A_{2}] + Pr[A_{3}] = 3 × \frac{1}{6} = \frac{1}{2}.

Is this calculation correct? Well, suppose instead that the casino rolled six dice, and again you win iff your number comes up at least once. Then the analogous calculation would say that you win with probability 6 × \frac{1}{6} = 1, i.e., certainly! The situation becomes even more ridiculous when the number of dice gets bigger than 6.

The problem is that the events A_{i} are not disjoint: i.e., there are some sample points that lie in more than one of the A_{i}. (We could get really lucky and our number could come up on two of the dice, or all three.) So if we add up the Pr[A_{i}] we are counting some sample points more than once.

Fortunately, there is a formula for this, known as the \underline{Principle of Inclusion/Exclusion}:

CS 70, Fall 2004, Lecture 16 7

Theorem 16.4: [Inclusion/Exclusion] For events A_{1},...,A_{n} in some probability space, we have

Pr[⋃ _{i=1}^{n} A_{i}] = ∑_{i=1}^{n} Pr[A_{i}] − ∑_{{i,j}}Pr[A_{i}∩ A_{j}] + ∑_{{i,j,k}}Pr[A_{i}∩ A_{j} ∩ A_{k}] − ··· ± Pr[⋂ _{i=1}^{n} A_{i}].

[In the above summations, {i,j} denotes all unordered pairs with i ≠ j, {i,j,k} denotes all unordered triples of distinct elements, and so on.]

I.e., to compute Pr[⋃_{i}A_{i}], we start by summing the event probabilities Pr[A_{i}], then we subtract the probabilities of all pairwise intersections, then we add back in the probabilities of all three-way intersections, and so on.

We won’t prove this formula here; but you might like to verify it for the special case n = 3 by drawing a Venn diagram and checking that every sample point in A_{1}∪ A_{2}∪ A_{3} is counted exactly once by the formula. You might also like to prove the formula for general n by induction (in similar fashion to the proof of Theorem 17.3).

Taking the formula on faith, what is the probability we get lucky in the new game in Vegas?

Pr[A_{1}∪ A_{2}∪ A_{3}] = Pr[A_{1}] + Pr[A_{2}] + Pr[A_{3}] − Pr[A_{1}∩ A_{2}] − Pr[A_{1}∩ A_{3}] − Pr[A_{2}∩ A_{3}] + Pr[A_{1}∩ A_{2}∩ A_{3}].

Now the nice thing here is that the events A_{i} are mutually independent (the outcome of any die does not depend on that of the others), so Pr[A_{i} ∩ A_{j}] = Pr[A_{i}]Pr[A_{j}] = (\frac{1}{6})^{2} = \frac{1}{36}, and similarly Pr[A_{1} ∩ A_{2} ∩ A_{3}] = (\frac{1}{6})^{3} = \frac{1}{216}. So we get

Pr[A_{1}∪ A_{2}∪ A_{3}] = (3 × \frac{1}{6}) − (3 × \frac{1}{36}) + \frac{1}{216} = \frac{91}{216} ≈ 0.42 .

So your odds are quite a bit worse than the casino is claiming!

When n is large (i.e., we are interested in the union of many events), the Inclusion/Exclusion formula is essentially useless because it involves computing the probability of the intersection of every non-empty subset of the events: and there are 2^{n}− 1 of these! Sometimes we can just look at the first few terms of it and forget the rest: note that successive terms actually give us an overestimate and then an underestimate of the answer, and these estimates both get better as we go along.

However, in many situations we can get a long way by just looking at the \underline{first}term:

1. Disjoint events. If the events A_{i} are all disjoint (i.e., no pair of them contain a common sample point — such events are also called mutually exclusive), then

Pr[⋃ _{i=1}^{n} A_{i}] = ∑_{i=1}^{n} Pr[A_{i}].

[Note that we have already used this fact several times in our examples, e.g., in claiming that the probability of a flush is four times the probability of a Hearts flush — clearly flushes in different suits are disjoint events.]

2. Union bound. Always, it is the case that

Pr[⋃ _{i=1}^{n} A_{i}] ≤ ∑_{i=1}^{n} Pr[A_{i}].

This merely says that adding up the Pr[A_{i}] can only overestimate the probability of the union. Crude as it may seem, in the next lecture we’ll see how to use the union bound effectively in a Computer Science example.