Opinion

The A/A test is back.

Published on · 2 min read

Only, nobody calls it that any more.

An old e-commerce reflex

Anyone who ran A/B tests in e-commerce knows it. We tested a button, a block, a feature: version A for half the visitors, version B for the other half. And from time to time, we showed the same version to both groups. If the tool announced a winner, it was the tool that had a problem.

At the time, we did it once, to check the tool. With AI, the answer itself changes: ask the same question twice and you will not get the same answer both times. Technical teams call that the noise floor.

Reducing the noise

We do not remove it. We reduce it, then we monitor it.

Take a collection of reports that you want to be able to query. The AI does not answer from memory: it looks things up in the reports, and the figures come from them, not from the model. The less it invents, the less it varies.

But that only works with good data. The same format from one report to the next. The same analysis filed under the same label, everywhere. It is the least visible work, and the one that decides everything.

Check, then monitor

Next, a double check: a second, more powerful model re-reads and flags inconsistencies. A name that changes, a figure that does not match its source.

And monitoring: a fixed set of questions, asked again every week at first, then every month. Same questions, same data. If the answers move, something has changed, and we know before the user does.

The same logic is used to measure what AI assistants say about a brand: repeat, check, monitor. We explain it in our measurement method.

It is an A/A test, turned into a maintenance routine. The tool is intelligent, provided the data is clean.

Portrait of Yvan Forestier

Yvan ForestierCo-founder of Reflets

Back to top

Let’s talk about what’s taking up your time.

A 30-minute video call, booked in our calendar.

Contact us