Example 8: Interpreting Test Results
February 10, 2022The central idea of etcetera abduction is that conditional probabilities can be reified as first-order literals with equivalent prior probabilities. We assume conditional independence between these literals, and estimate the joint probability of a set of literals as the product of these priors.
That all sounds well and good, but it can be hard to grasp the nuances and implications of this approach, and how it relates to traditional probability theory. I found that there are some good intuitions to be gained about etcetera abduction by modeling simple textbook examples of probabilistic inference.
Here's an example that I found on the Wikipedia page for "Base Rate Fallacy" (link).
A test for an infectious disease is run on a population of 1000 persons, in which 40% are infected. The test has a false positive rate of 5%, and a 0% false negative rate. The expected outcome of the 1000 tests would be that 400 infected people would receive a true positive result (1000 * 0.4 * 1.0) and 30 healthy people would receive a false positive result (1000 * 0.6 * 0.05). If a person receives a positive test result, what is the probability that they are actually infected?
Bayes' Theorem provides us with the answer:
Pr( infected | positive ) = Pr( positive | infected ) * Pr( infected ) / Pr( positive )
= true_positive_rate * infected_base_rate / positive_rate
= 1.0 * 0.4 / 0.43
= 0.930232558139535
If a person receives a positive result, they can be over 93% confident that the test correctly indicates infection.
The wikipedia article further shows how a positive result from this same test, applied in a different population with a very low infection rate, would indicate infection with much less confidence, e.g. a positive result in a population with 2% infection rate would have a confidence of only 29% of correctly indicating infection.
Let's translate the first example into logic, and see how to apply Etcetera Abduction to compute these same numbers.
First, let's start with the observation: A person (P) receives a positive test result.
(positive P)
Second, we encode the conditional probability P( positive | infected ) as a definite clause. This represents the true positive rate of the test, which in this example is 100%.
(if (and (infected x)
(etc_true_positive 1.0 x))
(positive x))
Second, we encode the conditional probability for the false positive case, P( positive | healthy ), which is known to be 5%.
(if (and (healthy x)
(etc_false_positive 0.05 x))
(positive x))
Third, we need to include the base rates, encoding them as etcetera literals that are the priors for being healthy (60%) or infected (40%).
(if (etc_healthy 0.6 x)
(healthy x))
(if (etc_infected 0.4 x)
(infected x))
Finally, we can include the prior probability of getting a positive test result, irrespective of whether you are healthy or infected. In this example, that prior probability is 43% (400 + 30 / 1000).
(if (etc_positive 0.43 x)
(positive x))
Now let's ask Etcetera Abduction to interpret the observation (positive P), and see what it comes up with:
> python -m etcabductionpy -i infection1.lisp ((etc_positive 0.43 P)) ((etc_infected 0.4 P) (etc_true_positive 1.0 P)) ((etc_false_positive 0.05 P) (etc_healthy 0.6 P)) 3 solutions.
Interesting! There are actually three ways to interpret the observation. The first, most probable interpretation of a positive test result is to make no assumptions at all as to whether person P is healthy or infected. This reminds us that interpretation is not the same thing as classification. The best, most-probable explanation for why someone would receive a positive test result is that, indeed, that sort of thing happens exactly 43% of the time.
Now let's ask the question: What is the exact probability that this first interpretation is correct? For this, we again turn to Bayes' Theorem:
Pr( interpretation | observations ) = Pr( observations | interpretation ) * Pr( interpretation ) / Pr( observations )
The probability of the interpretation is the product of its etcetera literals, Pr( interpretation ) = 0.43.
The probability of the observations is also known in this example: positive results happen 43% of the time: Pr(observations) = 0.43.
The final term is Pr( observations | interpretation ), and here is the most important concept to grasp when using etcetera abduction: The probability of the observations given the interpretation is always 100%. Why? Because we are using logic as our reasoning mechanism: we accept that our axioms are always true, and the set of assumptions in any solution found by etcetera abduction always fully entails the input observations. If the interpretation is true, then the observations are true, 100% of the time.
Essentially, etcetera abduction moves all of the uncertainty that is represented by the conditional probability term Pr( observations | interpretation ) in Bayes' Theorem over to the Pr( interpretation ) term. To reiterate the central idea, etcetera abduction reifies conditional probabilities as first-order literals with equivalent prior probabilities. The math is the same, but where we factor in the uncertainty is slightly different.
With the conditional probability term always equal to 1 in etcetera abduction, we can write a new version of Bayes' Theorem that applies to etcetera abduction, as well as any other reasoning framework where an interpretation logically entails the observations:
If (interpretaiton => observations), Pr( interpretation | observations ) = 1 * Pr( interpretation ) / Pr( observations )
When we plug in the numbers for the first interpretation, we get 0.43 / 0.43 = 1.0. What does this mean? Basically, it is saying that if we make no assumptions as to whether person P is infected or healthy, then we are guaranteed to be correct. The safest bet is not to bet at all.
Now let's look at the second most probable interpretation:
((etc_infected 0.4 P) (etc_true_positive 1.0 P))
As with all solutions, forward-chaining on these literals fully entails the observation, given our knowledge-base axioms. Importantly, this interpretation entails something else of interest as well: (infected P).
What is the probability of this interpretation? Pr( interpretation ) is the joint probability of these two etcetera literals. In etcetera abduction, we always make the assumption that etcetera literals are conditionally independent, and compute joint probability naively as the product of their priors, so Pr( interpretation ) = 0.4 * 1.0 = 0.4
Plugging in these numbers, we compute the likelihood of this interpretation as 0.4 / 0.43 = 0.930232558139535. This is exactly the result obtained in the Wikipedia example. A positive test result in this population indicates infection over 93% of the time.
What about the third interpretation? This solution describes the other interpretation of a positive test result, namely that it is a false positive. Forward chaining on these two etcetera literals also entails the observation (positive P), as well as another important literal, (healthy P). What is the likelihood that this interpretation is correct?
Pr( interpretation | observations ) = 1 * Pr( interpretation ) / Pr( observations )
= (0.05 * 0.6) / 0.43
= 0.06976744186046512
Great! We could interpret a positive result as a false-positive, but our confidence in this interpretation is under 7%, exactly the result obtained by traditional probabilistic reasoning methods.
Just for completeness, let's change the base rate of infection in our population, and see if we can replicate the second example in the Wikipedia article.
(if (etc_healthy 0.98 x)
(healthy x))
(if (etc_infected 0.02 x)
(infected x))
(if (etc_positive 0.069 x)
(positive x))
> python -m etcabductionpy -i infection2.lisp ((etc_positive 0.069 P)) ((etc_false_positive 0.05 P) (etc_healthy 0.98 P)) ((etc_infected 0.02 P) (etc_true_positive 1.0 P)) 3 solutions.
Here we see that a false positive is much more likely than a true positive if we apply this test in a population with low incidence of infection. It is still the safest bet to assume nothing (interpretation #1), but if you had to make a choice, the likelihood of interpretation #2 is around 71% (0.05 * 0.98 / 0.069), while interpretation #3 is around 29% (0.02 * 1.0 / 0.069).
It is reassuring that etcetera abduction gives the correct probabilities in these two simple textbook examples. Things get a lot more interesting when there are multiple observations, and when the true joint probability of these multiple observations isn't known. For now, though, a couple of important facts about etcetera abduction can be underscored:
- If there is only one observation, the best interpretation should ALWAYS be its prior probability.
- The probability of the best interpretation should NEVER be greater than the true probability of the observations.
When presented with a positive test result in these textbook examples, the safest bet is to make no assumptions whatsoever. The best explanation for positive results is that they do occur some percent of the time, as that is the way the universe works. If somehow we were able to discover some set of antecedents that logically entailed the single observation that were collectively MORE PROBABLE than the prior probability, then we would have to admit that our estimate of this prior probability was too low all along.
In these examples, the probability of best interpretation (assuming nothing) is equal to the probability of the observation. If it were any bigger, than Bayes' Theorem would give us a Pr( interpretation | observations ) that was greater than 1.0. We would have to admit that our estimate of the probability of the observation was too low all along.
This second fact becomes super-important when we have multiple observations, and we don't know their true joint probability - as is routinely the case in real-world probabilistic reasoning problems. We could, of course, naively assume that the joint probability of the observations is the product of their priors, but we would run the risk of under-estimating their true joint probability. This is precisely where etcetera abduction can help: given a knowledge base, we can search for some interpretation(s) of multiple observations that can used to compute a better estimate of their true joint probability, compared to the naive estimate. More on that next.