“Verify the output” is the standard advice for using AI responsibly. It quietly assumes you’d recognize a wrong answer, and outside your own field you usually can’t. Below is the full script of this episode as an article.
Today I want to take apart one piece of advice. You’ve heard it, and I’ve given it. Verify the output.
Every responsible talk about AI ends there. Use the tools, but check what they give you. And I think that advice has a hole in the middle of it, exactly where people get hurt.
So here’s my claim, and you’re welcome to disagree. The most dangerous AI mistakes happen in fields you can’t check, and “verify the output” fails right there, because verifying takes the expertise you don’t have.
I’m going to make that case with three witnesses, starting with a novelist and ending with a man who used to be a lawyer. Then I’ll give you something to do about it.
The novelist
The novelist first.
In two thousand two, Michael Crichton gave a speech called “Why Speculate?” In it he named something he called the Murray Gell-Mann Amnesia effect, after the physicist, who he’d once discussed it with.
Here’s how Crichton described it. You open the newspaper to an article on a subject you know well. In Gell-Mann’s case, physics. In Crichton’s, show business. And you see that the journalist has, in his words, “absolutely no understanding of either the facts or the issues.” Sometimes the story is so wrong it’s backward. He called those “wet streets cause rain” stories.
Then you turn the page to national or international affairs, and you read with renewed interest, as if the rest of the paper were somehow more accurate. Crichton’s line for it was, “You turn the page, and forget what you know.”
He also pointed out that we don’t behave this way anywhere else. If a person keeps exaggerating or lying to you, you soon discount everything they say. Courts even have a Latin phrase for it, falsus in uno, falsus in omnibus. False in one thing, false in all.
Hold onto that, because twenty-four years later, the newspaper talks back.
The engineer
The second witness is Gergely Orosz. He writes The Pragmatic Engineer, a newsletter about software engineering, and he knows that field cold.
On October nineteenth, he posted on X about a pattern he keeps running into. When he uses AI in an area where he’s an expert, like coding on a project he knows well, he spots the problems right away. The model makes errors, or in his word, it’s “BS’ing” him, and he catches it. No big deal. He corrects it or starts over.
But when he’s not an expert, he wrote, he finds that he trusts it, and only later learns it did something stupid. His example was a legal case. He asked three different AI models for feedback on it. He’d also paid a lawyer, but the lawyer was slow. All three models confidently told him one thing. The lawyer told him another. And the lawyer was right.
His opening line was, “This messes with my mind ALL THE TIME.”
What strikes me is who this happened to. By his own account, he catches the model’s mistakes immediately in his own field. The skill didn’t travel. Out of his field, the confident answer looked exactly as solid as the true one, and three models agreeing made it look more solid still.
That’s Crichton’s amnesia, with one change. The newspaper never answered your follow-up question. AI does, fluently, in the same calm tone whether it’s right or wrong.
The lawyer, the fixer, and the chatbot
The third witness shows what happens when “verify the output” gets passed down a chain of people.
Michael Cohen, Donald Trump’s former fixer, is a disbarred attorney. In late twenty twenty-three, he was trying to end his court supervision early. His lawyer, David Schwartz, was drafting the motion. Cohen used Google Bard to find supporting cases, and on November twenty-fifth he emailed three of them to Schwartz, with summaries.
The cases did not exist.
In a sworn declaration, Cohen said he hadn’t realized Bard was a generative text service that, quote, “could show citations and descriptions that looked real but actually were not.” He thought it was, his words, “a super-charged search engine.” He also said he didn’t have access to the standard tools lawyers use to confirm a case, and he trusted Schwartz and his team to vet them.
Schwartz, for his part, said he did not independently review the cases, because of the reputation of another lawyer involved in the matter. The motion was filed. The government’s lawyers answered it and didn’t flag the fake cases either. It was Cohen’s new attorney, E. Danya Perry, who told the court she was unable to verify the citations.
In March of twenty twenty-four, Judge Jesse Furman declined to sanction anyone, and called the episode “embarrassing.”
Look at how the checking was supposed to work. The person who couldn’t verify handed the work to a person who could. And that person assumed somebody else already had. Everyone was told to verify the output. Each of them believed the verifying lived one step away.
The argument
So here’s the argument, plainly.
“Verify the output” quietly assumes you know what a wrong answer looks like. Inside your field, you do. You’ve seen a thousand right answers, so a wrong one snags. Outside your field, you’ve got nothing to snag on, and AI is very good at producing answers with nothing to snag on. Clean grammar and a calm tone, with a citation in the right format.
That’s why I’d argue the danger is uneven. The mistakes in your own field are the cheap ones, because you catch them. The expensive ones land in the fields where you most need help, which are the same fields where you can’t tell help from harm.
And to be fair to the other side, AI outside your expertise is still useful. It can teach you a field’s vocabulary and help you ask a better question. I use it that way all the time. The risk comes in when you take the answer and act on it.
I’m trained in organizational psychology and family therapy. Being trained in two fields means I’ve also spent a lot of time near fields I’m not trained in. And the healthiest professionals I know share a habit. They know where the edge of their competence is, and they say so out loud. “That’s outside my lane, let me get you to someone.” AI doesn’t say that unprompted. So you have to say it for it.
What you can do
Here’s what you can do.
Start by drawing your map. Take a sheet of paper and list the areas where you’ve used AI in the last month. Next to each one, write a single honest answer. Could I catch a wrong answer here without looking anything up? Yes or no. Every “no” is a field where “verify the output” is not available to you.
For those “no” fields, borrow the expertise before you rely on the answer. Gergely Orosz described his own rule in a follow-up post. When he’s not an expert, he assumes the AI will BS him and he won’t know. So he either puts in the time to become more of an expert, which he admits usually isn’t an option, or if it matters, he goes to an expert. He mentioned his lawyer, and his accountant, who isn’t going anywhere.
You can make that cheaper. Use the AI to prepare for the expert, not to replace them. Ask it to list the questions a specialist would ask about your situation, and the assumptions its own answer depends on. Then take that list to a real person. A thirty-minute call goes a long way when you walk in knowing what to ask.
When it’s your name going on something, set one rule. If you can’t check it and you didn’t have someone qualified check it, it doesn’t go out. That’s the lesson of the Cohen case. Every person in that chain thought the checking lived somewhere else.
And be suspicious when several models agree. Orosz asked three, and all three were confidently wrong in the same direction. Agreement between models can feel like a second opinion. A second opinion has to come from somewhere that can actually be wrong differently.
Turn the page
Crichton said we forget what we know when we turn the page. My version for this year is shorter. Notice the moment you turn the page. When you leave your own field, the answer on the screen sounds exactly the same, and that’s the moment to go find someone whose field it is.
So this week, ask yourself one question. Where am I trusting AI most, and could I tell if it were wrong there?
—
This show extends the book AI Empowered. Read the first chapter free at https://www.aiempoweredbook.com, or get the book on Amazon.
Aaron Douglas runs Auspicious, a fractional marketing practice.
Blog: https://auspicious.llc/insights/
X: https://x.com/aaron_douglas
LinkedIn: https://www.linkedin.com/in/aaronddouglas/
