Nine months into an AI rollout, the dashboard says eighty-something percent weekly usage, training completion in the nineties, time-to-first-draft down forty percent. Every number is accurate. Then somebody pulls three client proposals from before and three from after and reads a paragraph from each out loud, and you could swap the client names without anyone […] Below is the full script of this episode as an article.
Picture a quarterly review. Somebody from operations is presenting the AI rollout, nine months in, and the slide is beautiful. Eighty-something percent of the company logging in every week. Training completion up in the nineties. People say they like the tools. Time to first draft down forty percent.
Every number on that slide is green. Every number on that slide is also true.
The room does what rooms do. Finance asks about return on investment. The chief executive says this is exactly what we needed to see.
And then the person who runs client relationships, who hasn’t said anything in twenty minutes, pulls up three proposals from before the rollout and three from after. Same kind of work, same team, comparable scope. She reads a paragraph from each one out loud.
The new ones are cleaner. Faster. Better formatted.
And you could swap the client names and nobody in that room could tell you which one is which.
Nobody on that slide was lying, right? That’s what makes this hard. Every metric was accurate. Every metric was also pointed at something other than the thing the company actually sells.
The argument
That gap is chapter ten of the book, the chapter about organizations. The argument is that the instruments most companies bought to watch their AI rollout are aimed at the wrong target. They measure whether people are using the thing, which is not the same question as whether the work got better. And a lot of companies answer the first one and report it like they answered the second.
I keep landing on the same comparison. A bathroom scale measures something real. It’s just not cardiovascular health.
And watch what happens next, because this part is almost automatic. Dashboard’s green, work is flat, so leadership pushes harder. Require it. Put usage in the performance review. Which gets you more of exactly what you already had, because you can order somebody to open the tool and you cannot order them to bring anything to it. Compliance is easy to mandate. Judgment isn’t.
Why September
So why am I saying this in September instead of any other month?
In August, a group of researchers posted a working paper called “How Organizations Use AI: Evidence from ChatGPT.” I want to be careful with the provenance here, because it matters. Three of the five authors work at OpenAI. Two are academics, one at Columbia and one at Wharton, and both of them did this work as paid contractors for OpenAI. It’s a working paper, not peer reviewed. OpenAI studying OpenAI’s own enterprise product. Hold all of that.
And hold the subtitle, because the subtitle is doing work. Evidence from ChatGPT. One product. The paper says this out loud… it can’t see other AI systems, it can’t see A-P-I tools, it can’t see anything a company built in-house, and it can’t see the personal accounts people are quietly running on the side. Which, if you’ve managed a rollout, you know is a lot of it.
What they did have is still remarkable. Message counts, active users, token volume, all de-identified and reported in aggregate. The paper is clear that no researcher manually reviewed anybody’s individual messages. Seventeen hundred and sixty-four companies, seventeen and a half million messages, measured six months after each company switched the product on. That’s the piece about who inside a company is actually using it. The growth numbers come from the whole customer base. The task breakdown comes from a smaller slice, about nine hundred and seventy companies. Different questions, different samples.
And what comes out is roughly four things, all of which that quarterly review would kill for.
The volume of output the product generated grew about sevenfold over nine months, and about half of that growth came from companies that had already adopted, going deeper.
Among American public companies, the ones that adopted early were already bigger, already more valuable, already spending more on research and development. And the paper fences that in immediately. Those are associations, and it says in so many words that they should not be interpreted causally. So nobody gets to stand up in a quarterly review and say the tool did that. What you’ve got is a pattern. The early adopters were already the kind of company that invests in this stuff.
Among the people who are active, the mix runs all across the org chart, not just the technical ones. Though the job title data is incomplete and there’s no denominator, so it tells you who’s in the room and not who stayed out.
And there’s a detail in there I can’t stop turning over. Analysts, and marketing and communications people, send more messages per week than the average active user at their own firm. Executives send fewer. Early-career people and trainees send roughly eight or nine more messages a week than the average active user at the same company.
Then the fourth one, which is that they’re using it for an enormous range of work. Writing, communication, pulling information together, research, planning, analysis, legal work, finance. It’s everywhere, basically.
What the paper measured, and what it didn’t
And then you get to the limitations paragraph, and the authors say the quiet part themselves.
Their words. The task results “do not measure downstream work products, productivity effects, or changes in organizational routines.”
Listen to that one again.
Eight point seven million classified messages… and that part of the data cannot see what came out the other end. They flag it themselves. They’re explicit about it. They say straight out that future work should go connect this to actual output. It’s an honest paper, and I love that they put it in writing.
So a dataset that size, on one AI product, can tell you precisely how much people typed and nothing at all about whether anything got better.
Which makes me want to ask what your adoption dashboard thinks it’s looking at.
The line I’d put on a wall is the paper’s own. Adoption is only the beginning of deployment.
In my own shop
Here’s where I got to learn this in my own shop.
I run a marketing practice, Auspicious, and a good chunk of it runs on scheduled jobs. Automated tasks that fire on a clock, do a piece of work, and write to a shared ledger. Two lines per job. One when it starts. One when it finishes, saying what it made.
On August nineteenth, one of those jobs started at six eleven in the morning. It was the job that makes this podcast, actually. It wrote its start line. And then nothing. No second line, ever.
But notice how that got caught, because that’s the whole thing. There’s a daily check that reads the ledger. It could have stopped at “this job never wrote a finish line,” and that’s a real signal. Something didn’t complete. But the check does a second thing. It goes and opens the folder where an episode is supposed to show up, and it looks at what’s actually sitting in there.
Five hours and fifty-three minutes later, that second clause is the one that mattered. No finish line… and no episode folder at all. Newest thing in there was still the one from the week before. There was nothing to recover, because the run never made anything. By the time the next run succeeded, the feed had a fourteen-day hole in it.
The first clause tells you the machine ran. The second one tells you something exists.
I had to learn to write both, and I didn’t build it that way at the start. Took me a while to notice.
What to do Monday morning
So what does this mean for you Monday morning.
Keep the adoption dashboard. Usage data is real data and you paid for it.
Add one thing next to it that a dashboard structurally cannot do, because it needs a person to open the actual work.
Pick the deliverable your company sells. A proposal, a report, a campaign, whatever it is that clients pay a premium for. Pull two of them from before the AI rollout and two from the last month. Same type of work, comparable scope. Say a paragraph from each out loud, in a room, with other people in it.
You’re listening for one thing. Whether you could swap the client’s name out and have anybody notice.
And if the answer is yes, resist the reflex to schedule more training. The people who wrote those proposals almost certainly knew something specific about that client that never made it onto the page. So ask them that. What did you know about this account that isn’t in here. They’ll tell you, usually right away, and you’ll be looking straight at the thing that got left out. Nothing in the process ever asked for it.
No dashboard will ever show that green. It takes one meeting, and it’s the only instrument you’ve got that’s aimed at the thing you actually sell.
Everything on that dashboard is downstream of one person deciding to bring what they know into the room. The dashboard cannot see that decision.
You can. But only if you go look at the work.
—
This show extends the book AI Empowered. Read the first chapter free at https://www.aiempoweredbook.com, or get the book on Amazon.
Aaron Douglas runs Auspicious, a fractional marketing practice.
Blog: https://auspicious.llc/insights/
X: https://x.com/aaron_douglas
LinkedIn: https://www.linkedin.com/in/aaronddouglas/
