God and Machine Learning

What does the most important formula in artificial intelligence have to do with God?

Machine Learning, a prominent application of AI, empowers machines to analyze data, extract insights, and make predictions.

Large Language Models like ChatGPT are Machine Learning models that are trained on massive amounts of text data to learn patterns and generate text based upon predictable outcomes.

Your conversations with ChatGPT run (mostly) smoothly because it is constantly predicting the most likely next word in the sentence.

And at the heart of these predictions in Machine Learning and Data Science is Bayes’ Theorem.

But Data Scientists performing Bayesian calculus might be unaware of its origins in discussions about the existence of God and the evidence for miracles.

Ready to update your priors?

It helped Alan Turing break Germany’s Enigma cipher. It was used to determine the authorship of the Federalist Papers. It’s what sorts your spam e-mail into the junk folder. It even informed COVID testing.

Bayes’ Theorem describes how to adjust your beliefs based on new information.

It allows statisticians to update conclusions when presented with new evidence. It’s a way to calculate posterior probability—the revised probability of an event after inputting additional data.

It’s not enough to consider only the probability of something given our background knowledge (called the “prior” probability). Bayes’ Theorem states that the posterior probability can be calculated by multiplying the prior probability by the likelihood of the evidence given an event’s occurrence, and then dividing it by the probability of the evidence itself.

If that sounds confusing, don’t worry, the formula looks even more confusing:

Here P is probability, A is an event, and B is another event or information or evidence. The probability of A given B (posterior probability) is equal to the prior probability of A multiplied by the probability of B given A, divided by the probability of B.

Okay. Maybe an example would help.

Let’s take it into the realm of the miraculous—like weather prediction.

It’s a cloudy day. Is it going to rain? What’s the likelihood of rain, given the cloudiness?

The probability of rain, when there are clouds, is represented as P(R|C). That’s what we want to find out.

Maybe we know that 80% of the time rainfall is caused by cloudy weather. So the probability of clouds given rain is 80%, or P(C|R) = 0.8.

But how often does it rain in general? This is the prior probability of rain, or P(R). This month, it rains about a quarter of the days. P(R) = 0.25.

And 50% of the days this month are cloudy. So P(C) = 0.5.

We see the clouds. Will it rain today?

We could guess that there’s a 25% chance of rain, considering rain’s prior probability. But what happens when we add our knowledge that it’s a cloudy day?

Bayes’s Theorem tells us to multiply the prior probability of rain times the probability of clouds given rain, and then divide it by the probability of clouds.

0.25 x 0.8 = 0.2
0.2 / 0.5 = 0.4

So, there’s a 40% chance of rain. Notice that the posterior probability that accounts for the clouds is higher than the prior probability. We’ve concluded that there’s a better chance of rain, given the evidence.

Bayesian inference is used in Data Science to revise estimates as data is gathered. Machine Learning deploys a Naive Bayes algorithm to classify information and act upon it. It’s the basis for all predictive modeling today.

But long before LLMs applied Bayesian methods, Thomas Bayes defined this principle in 1763.

Bayes lived in the era of the Enlightenment. It was a time that witnessed profound developments in science and mathematics. But it was also a setting for religious skepticism.

The Reverand Thomas Bayes was not only a statistician but a Presbyterian minister.

And Bayes, along with his friend and posthumous editor Richard Price, developed this theorem in part in response to the British skeptic David Hume.

David Hume claimed that, even if a miracle happened, you could never be justified in believing that a miracle happened.

Because a miracle is extremely improbable. And it isn’t reasonable to believe that the least probable thing is what took place.

“That no testimony is sufficient to establish a miracle, unless the testimony be of such a kind, that its falsehood would be more miraculous, than the fact, which it endeavors to establish.” (Hume, An Enquiry Concerning Human Understanding, X, “Of Miracles”)

Hume is saying that even if someone were healed or raised from the dead, it would never be rational to believe it. Because the sheer unlikeliness of it drowns out any evidence presented for it. No amount of evidence could ever overcome the intrinsic improbability of miracles.

Now, the idea that a miracle is intrinsically improbable is itself question-begging. It assumes a naturalistic understanding of reality. But that’s precisely the issue at hand: do miracles happen? To say, “We know that miracles don’t happen because we know that miracles don’t happen” is obvious circular reasoning.

But Bayes objects. This is fallacious.

For example: let's say I tell you that my driver's license number is 005749632. Out of all the possible 9-digit number combinations, that’s an extremely rare number!

On Hume’s analysis, you should never believe that such an improbable number is mine.

But what if I took out my driver’s license and showed you?

You’d also have to calculate the probability I’d have a state-issued ID with that precise number on it if it weren't my driver’s license number. (Sure, I could have created a fake one, but I have three kids and don’t have time for that!). Given the evidence, the most reasonable conclusion is that this is my number.

Or let’s say that the news reports a Powerball jackpot winner. The odds are astronomically low that any one person would win the jackpot. Like, 1 in 292 million! Name any person, and you could be almost certain that person isn’t the winner. So can we never believe the news? Well, some would wonder for other reasons…

But what’s the likelihood that the news would report this individual as the winner if that weren’t the case?

You need to weigh the probability of the evidence as well. That’s the denominator in Bayes’ Theorem.

Here’s how it works in a simplified “odds form” of Bayes’:

Notice that the denominator on the far right considers the likelihood that the evidence would be as it is if a miracle did not happen.

Even if the probability of a miracle based on our background knowledge of the world is extremely low, when multiplied by the probability that the evidence is best explained by the miracle hypothesis, the probability of the miracle hypothesis could be relatively high.

The point is: you still need to evaluate the evidence!

You can't just dismiss the claim out of hand like Hume did because miracles in general are (as you consider them) improbable.

And if the probability of naturalistic explanations for an event is low—if alternative hypotheses are ad hoc or lack explanatory power and scope—then the miraculous explanation could have a high posterior probability.

Hume continues to haunt discussions about God, miracles, and history.

There’s Bart Ehrman’s contention that “history can only establish what probably happened, and by definition, a miracle probably didn’t.”

There’s Carl Sagan’s refrain (popularized by Neil deGrasse Tyson and others) that “extraordinary claims require extraordinary evidence.”

But this idea that you need a tremendous amount of evidence, even a “miraculous” amount, in order to counterbalance the low prior probability of an extraordinary event is demonstrably false.

We have Bayes to thank for that.

Perhaps, when we evaluate the evidence, we have God to thank for that.