One question to ask any AI that touches your media money

The question is: how often were your recommendations right last quarter, and how do you know?

I ask it of every vendor who tells me their platform is intelligent. Most cannot answer. They are not hiding anything. Nobody ever asked them to keep score. This piece is about why the question matters, what a good answer contains, and one test you can run yourself.

Why the question matters

Every major ad platform now has an AI that plans your campaign, sets your bids, picks your audience, writes your creative and then reports on how well it did. It is a student marking its own exam. It will tell you the campaign performed. It will never tell you to move money off the platform, because no platform's AI has ever recommended a smaller budget for itself.

Nobody planned this. It is built into the structure, and the same structure now runs across search, social and retail media in every market.

The clearest recent example is the FTC's case against Amazon, filed on 31 August 2026. The FTC alleges that Amazon quietly changed how its search ad auction set prices, so that by 2024 about 79% of winning ads paid their full bid. Amazon's answer is that advertisers "adjust bids based on real-world performance, not descriptions of auction mechanics". In plain words: you were getting results, so why do you care how the price was set? Every self-grading system makes the same argument. Trust the outcome, do not ask about the method. And as one buyer told AdExchanger, advertisers could not check even if they wanted to. Only Amazon had the auction logs.

The numbers on trust

In the ANA's March 2026 study, 90% of marketers said they worry their agency's recommendations serve the agency rather than the client. In Kantar and Marketing Week's 2026 effectiveness study, 72% of marketers admitted that how easy something is to measure decides where the budget goes, and 71% said they define success differently from their own CEO or CFO. eMarketer found only 52% of US marketers had run an incrementality test in the past year, even after Google cut the minimum spend for its experiments from about US$100,000 to US$5,000. In India, an ad-fraud monitoring firm reports that only 5% of brands demand independent measurement at all.

Harvard Business Review (May 2026) and California Management Review (June 2026) both argued this year that governing AI is now a core marketing skill. I agree. But governance in the abstract does not help a CFO on a Tuesday. A CFO needs one question and one acceptable answer.

What a good answer contains

1. A count. We made this many recommendations last quarter. This many were accepted. Of those, this many improved the target metric, this many made it worse, this many made no difference.

2. A method. Improvement was measured on the same metric, over the same window before and after, with the same maths that produced the recommendation. The metric is fixed in advance and never swapped afterwards for one that looks better.

3. The misses, shown rather than buried. A system that reports a 100% hit rate is keeping a brochure, not a score.

4. A confidence level on each call, stated before the outcome was known, so you can see whether the confident calls were the right ones.

5. Separation of roles. Whatever computed the number was not the same thing that wrote the story about the number.

Why the last one matters most

Large language models are excellent at sentences and unreliable at arithmetic. If one model both computes the result and describes it, you get fluent, confident and occasionally invented numbers. If the maths runs first, in fixed code, and the language model is only allowed to phrase what the maths produced, you get something you can audit. Every number has a query behind it. Every sentence has a number behind it.

Ask the vendor one follow-up: can your language model change a number? If the answer is anything other than "no, structurally", keep walking.

The test you can run yourself

Pick one number from the vendor's last report. Cost per acquisition for one campaign in one week will do. Ask for the raw exports and reproduce it. If you can get to the same number in a spreadsheet, the system is honest about that number at least. If you cannot, ask why not. If the vendor cannot explain the gap, you have your answer about the rest of the report too. I call this the recompute test, and it has ended more vendor pitches than any RFP question I have ever written.

Where we stand

Our platform keeps score on itself. Every recommendation is tracked from proposal, through the human action taken, to a before-and-after measurement computed by the same engine. The win rate is shown in the product, including the recommendations that went wrong. I am not going to quote the number here, because the only version worth trusting is the one you see on your own account. Every finding carries a confidence score out of ten. When the data is thin the system says "not enough data" rather than guessing. And the language layer cannot touch a number. It can only describe one.