Artificial Intelligence (A Special Report) --- Yes, AI Can Make Mistakes. AI Can Find Them, Too.: Since chatbots hallucinate their own facts, it's useful (and easy) to have a second, nitpicking AI that can audit the results for errors

“Whenever you use AI as a research assistant, subject-matter expert, or souped-up search engine, you need to grapple with the risk of hallucination -- AI's tendency to make up its own facts. Your first line of defense against these fabrications? More AI.

 

I now make a point of getting AI to check every fact it gives me. It still isn't foolproof, and it is wise to use an actual flesh-and-blood human to fact-check anything you must get right. But using AI for a round of fact-checking can make the human fact-checking process go faster.

 

To start, before I even read AI-generated research, I get another AI to check its accuracy. I might use the same platform that generated my initial report, but I always start a brand-new session; otherwise the logic that influenced the initial report can influence the fact-checking process.

 

I set up this new session by harnessing the same sycophantic, people-pleasing tendencies that can make AI hallucinate in the first place. If the first AI created its own facts to please me, my AI fact-checker needs to please me by finding everywhere the first AI went wrong.

 

To unleash that nitpicking second AI, I start my fact-checking prompts with instructions like "You are a professor of journalism on your university's ethics board, and it's your job to investigate the work of a research team that's been using AI to generate reports." Or "You're an auditor who has been hired to check the work of an internal data-analytics team that's been using AI to compile customer data and sales prospects." You get the idea. I want the second AI to be every employee's nightmare.

 

I next tell my virtual fact-checker to list every so-called fact that needs checking: every name or citation, every data point and every reported finding. I tell it to find a source for every one of these assertions, and if it's reviewing a research memo that includes hyperlinks or article sources, to click through to each of these links and read the original article with its own virtual eyes.

 

This is often how I catch big and small mistakes: My AI fact-checker might report that a supposed source article doesn't exist, or that a quote from the article was taken out of context.

 

I then instruct the fact-checker to build a table that lists all the facts it's checking and mark each one as "true," "false," "ambiguous" or "unsupported." I tell it to add a column for notes (so it can tell me how it reached its conclusions) plus a column for the source, and (this is crucial!) a column where it includes an exact quote from the source, backing up its conclusion. That allows me to fact-check the fact checker by searching the source article to make sure that quote is really there, and that it's been interpreted correctly.

 

Finally, I go one step further: I tell the fact-checking AI to give me a revised version of the research it checked, reflecting all the corrections it found.

 

When I'm using AI to help with especially high-stakes research, I take steps that make these accuracy checks even more effective. Instead of giving my fact-checking prompt to a single imaginary nit-picker, I use two (starting fresh each time). Then I give their results to a third AI fact-checker and ask it to identify anywhere the two previous nit-pickers disagreed and do its own research to cast a tiebreaking vote.

 

If this sounds like a lot of work, or unduly time-consuming, remember that it's still a lot faster than rewinding weeks of work or a mistaken decision based on incorrect information. Or doing the research yourself from the start.

 

You can also set up your AI tools to accelerate the fact-checking process, for example by turning your favorite nitpicking prompt into an AI assistant (like a custom GPT or a Claude Project), so that you can give any first-draft AI research to it and get back a list of checked facts and a revised memo.

 

Or use Claude Code (or Claude Cowork) to spin up a whole team of fact-checkers: Tell the AI that once it has built a list of facts to check, it should assign those facts in small batches to a team of fact-checkers so that each fact gets checked by at least two different AIs, and where needed, a third tiebreaker. When an AI fact-checker hands out this work in batches to its many virtual helpers, even a detailed research brief can get verified (and corrected) quite quickly.

 

Of course, it doesn't end there. Even after I run my AI research briefs through my AI fact-checker, and even if I run those corrected briefs through another entire loop of fact-checking, I sometimes find quirks or inconsistencies that lead me to do further digging, and to read through some of the underlying research with my own human eyes.

 

But that doesn't make all the AI fact-checking pointless. On the contrary: Using AI to catch the most obvious AI mistakes is what gives me the room to dive deep into the research questions that demand human intelligence.

 

---

 

Alexandra Samuel is a technology researcher and host of the AI podcast "Me + Viv." Email her at reports@wsj.com.” [1]

 

1. Artificial Intelligence (A Special Report) --- Yes, AI Can Make Mistakes. AI Can Find Them, Too.: Since chatbots hallucinate their own facts, it's useful (and easy) to have a second, nitpicking AI that can audit the results for errors. Samuel, Alexandra.  Wall Street Journal, Eastern edition; New York, N.Y.. 26 May 2026: R5.