"Everything Is Predictable
By Tom Chivers
Atria, 384 pages, $29.99
First articulated in the 18th century by a hobbyist-mathematician seeking to reason backward from effects to cause, Bayes' theorem [1] spent the better part of two centuries struggling for recognition and respect. Yet today, argues Tom Chivers in "Everything Is Predictable," it can be seen as "perhaps the most important single equation in history." It drives the logic of spam filters, artificial intelligence and possibly our own brains. It may soon help us work through tricky social problems like vaccine hesitancy. Once you start to look for it, Mr. Chivers says, you start to see Bayes' theorem everywhere.
At its core, the theorem provides a quantitative method for getting incrementally wiser by continuously updating what you think you know -- your prior beliefs, which initially might be subjective -- with new information. Your refined belief becomes the new prior, and the process repeats.
The essential concept was developed by Thomas Bayes, a Presbyterian minister in Britain, and rescued from oblivion by his friend Richard Price, who published it in 1763. The work "sunk almost without trace," Mr. Chivers says, until it was independently discovered and refined in 1774 by the French polymath Pierre-Simon Laplace.
Before the theorem and its broader pattern of reasoning could gain much traction, Mr. Chivers explains, it was overtaken by approaches that felt less "soft and squishy." In the early 19th century, for example, Belgian prodigy Adolphe Quetelet began measuring anything that could be objectively quantified, like the chest size of Scottish soldiers. His interest was "social physics" -- using the rigors of mathematics to improve society.
In the early 20th century, the charge to treat data hermetically was led by two British scholars, Karl Pearson and R.A. Fisher, who despised subjectivity, Bayesian reasoning and each other. The analytical methods they developed -- particularly the concept of "statistical significance" -- are ubiquitous in scientific publications today, representing a core principle of what is now known as "frequentist statistics." In this framework, the analysis relies strictly on observed data, independent of prior beliefs.
Bayesian statistics, by contrast, dynamically update beliefs about a hypothesis as more data become available.
Despite the "furious rejection of Bayesianism" by Fisher and his colleagues, Mr. Chivers says, "people kept rediscovering or reinventing it, because it kept working." Harold Jeffreys, a Cambridge geologist, used it to assess Earth's composition; insurance underwriters used it to set premiums for workplace liabilities; artillery commanders used it to aim projectiles. Even as the techniques of Pearson and Fisher became enshrined in statistics departments, a rebellious community of Bayesians persisted on the fringes, devouring books that were "passed down almost like samizdat," as Mr. Chivers puts it. In 1979, the dispersed Bayesian community convened in Valencia, Spain, for a conference that would become a quadrennial tradition, famous for late-night singalongs featuring alcohol-infused statisticians warbling "There's no theorem like Bayes' theorem" and "Then I saw Tom Bayes, now I'm a believer."
At times Mr. Chivers, a London-based science journalist who now writes for Semafor, seems overwhelmed by an admittedly complex subject, and his presentation lacks the clarity of Sharon Bertsch McGrayne's "The Theory That Would Not Die" (2011).
Yet he is onto something, since Bayes' moment has clearly arrived. He notes that Bayesian reasoning is popular among "people who come from the new schools of data science -- machine learning, Silicon Valley tech folks." The mathematician Aubrey Clayton tells him that, in the cutting-edge realms of software engineering, "Bayesian methods are what you'd use."
Bayes' theorem is considered essential for decision-making under conditions of uncertainty. It's used when a radiology AI identifies a cancer and when ChatGPT composes a story. If there is a problem, Mr. Chivers suggests, it is that we don't use Bayesian reasoning often enough. It could improve the quality of scientific literature by giving researchers a "vehicle for skepticism." It might also help us make better predictions. The "superforecasters" identified by the political psychologist Philip Tetlock use a Bayesian approach, establishing priors, tracking their results and revising their assumptions as they attempt to predict events such as currency moves and election results.
It's notoriously difficult for most people to grasp problems in a structured Bayesian fashion. Suppose there is a test for a rare disease that is 99% accurate. You'd think that, if you tested positive, you'd probably have the disease. But when you figure in the prior -- the fact that, for the average person (without specific risk factors), the chance of having a rare disease is incredibly low -- then even a positive test means you're still unlikely to have it.
When quizzed by researchers, doctors consistently fail to consider prevalence -- the relevant prior -- in their interpretation of test results.
Even so, Mr. Chivers insists, "our instinctive decision-making, from a Bayesian perspective, isn't that bad." And indeed, in practice, doctors quickly learn to favor common diagnoses over exotic possibilities.
Bayesian reasoning also reminds us that when doctors encounter patient preferences that seem irrational (refusing childhood vaccines comes to mind), they should focus on identifying the underlying beliefs, since it's likely the priors that are dubious, not the reasoning that follows. Relatedly, Mr. Chivers notes, depression may result from "inappropriately strong priors on some negative belief." The promise of psychedelic medicines like psilocybin could lie in their ability to "flatten" these priors and help patients access a balanced worldview.
Our brains work by making models of the world, Mr. Chivers reminds us, assessing how our expectations match what we earn from our senses, and then updating our perceptions accordingly. Deep down, it seems, we are all Bayesians.
---
Dr. Shaywitz is a physician-scientist and board adviser, a lecturer at Harvard Medical School and an adjunct fellow at the American Enterprise Institute." [2]
1. "Bayes theorem:
Explanation in sentences:
Definition:
2. Thinking Prior to Thought. Shaywitz, David A. Wall Street Journal, Eastern edition; New York, N.Y.. 15 May 2024: A.15.
Komentarų nėra:
Rašyti komentarą