Case study

Measuring what AI assistants say about a brand: how we built our method

Published on · 6 min read

When a customer asks ChatGPT for advice, does the AI think of your brand? To answer that properly, a screenshot is not enough. Here is how we built our measurement method, from real campaigns for brands and for the release of a film.

Where we started: two real questions from clients

The first came from a brand. It was launching a media and press-relations campaign, and wanted to know whether ChatGPT, Gemini or Perplexity would talk about it differently afterwards.

The second came from a distributor, for the cinema release of a French film. We measured three dates, a week apart: before the release, the week of the release, then the week after.

In both cases, the question was simple. The answer called for a method. These two campaigns helped us build and refine it; to show its choices and their effects, we use the film as our example here.

An AI almost never answers the same way twice

Ask ChatGPT the same question twice and you will not get the same list. A single answer says yes or no, 0 or 100%. That is not a measurement, it is a roll of the dice.

So we ask each question ten times, in two rounds about 24 hours apart: that is what makes the measurement of each question reliable. But ten answers to the same question resemble each other: they are not worth ten independent opinions. What makes the overall figure precise is mainly the number of different questions. One measurement date means about fifty questions, or roughly 2,500 answers. And we read each AI assistant separately, with its own range (a 95% confidence interval).

Share of answers that mention the brand, with its range

Why many questionsOne answer gives only 0 or 100%. Ten answers to one question give 50%, with a range of 24 to 76%, too wide. 800 answers to 16 questions give 34%, with a range of 25 to 44%: usable.1 answer0 or 100%yes or no: 0 or 100%10 answers to 1 question50%too wide · range 24 to 76%800 answers to 16 questions34%usable · range 25 to 44%0%25%50%75%100%
What narrows the range is mainly different questions; repeating makes each question reliable. Fictional example.

The second round measures variability from one session to the next. It gives a benchmark for telling a real movement apart from the usual fluctuations in answers. We also redo some of the questions by hand, directly in the AI assistants’ own interface, to check that the automatic measurement sees the same thing as a user.

Asking questions like a customer

Asking “What do you think of this film?” tells you nothing: the AI is bound to talk about it. The real question is whether it thinks of it on its own.

So most of our questions never contain the name. They are written the way people talk: “Which films are out right now?”, “Which film should I go and see this weekend?”. Only a small share then asks what is said about the film. The questions are agreed with the client, then frozen: exactly the same at every date.

These questions are not drawn at random from what people actually ask AI assistants: nobody has access to that data. It is a purposive sample, like quota sampling in an opinion poll: built to a written method, as close as possible to real wording. The range therefore tells you how precise the measurement is on those questions, not a margin of error across all conversations. That is also why a before-and-after comparison is the most solid reading: the same questions at every date, so the bias in how they were chosen is the same on both sides.

Before launching, we also check the name itself. A film title is often a common word. We need to be sure the AI is talking about that film and not about something with the same name.

Being mentioned is not enough: it also has to be correct

An AI can mention your film and get everything else wrong. So we separate three things: being mentioned, being recommended, and being described correctly.

For the facts, we start from a fact sheet approved by the client: release date, director, cast, genre, where to watch it. Every answer is compared with that sheet. A piece of information is only declared incorrect if two judges agree: two AI models from different families, the second re-reading every piece of information the first has marked incorrect. And for the reading of the answers themselves (is the film mentioned, and in what tone), a person reviews a randomly drawn sample, to check that the machine reads correctly.

Why not review every answer? Because it is not necessary. A person reads 30 randomly drawn answers of each type: those that say whether the film is mentioned, those that describe its image and, for a brand, those where the AI chooses between two brands. A reading error often affects only one of these types, so we look for it type by type. If the machine gets one answer in ten wrong for a type, this review has a 96% chance of spotting at least one error. It is not a guarantee on every answer: it is a safety net that catches frequent errors. And every error found corrects the instruction given to the AI that does the reading; once the correction is checked, that date’s results are recalculated and the rule applies to all the answers concerned.

Chance of spotting at least one error when 30 answers of one type are reviewed

  • 96% chance of spotting an error that affects 1 answer in 10
  • < 10% error rate if none is found (95% bound)
  • 1 rule corrected for each error found, then that date’s results are recalculated
Human reviewChance of spotting at least one error when 30 answers of one type are reviewed: 79% if the error affects 5% of answers, 96% if it affects 10%.0%25%50%75%100%0%5%10%15%20%5% → 79%10% → 96%share of answers affectedby a reading error
Human review: the chance of spotting a reading error, depending on how frequent it is. Fictional example.

On the film, this check paid off. Before the release, more than one answer in three got wrong where to watch the film. After the release, almost none did. The error corrected itself. Another persisted: one AI presented the film as selected in competition at a major festival, which was incorrect.

Share of answers with an incorrect piece of information

Incorrect information, before and after the releaseShare of answers with an incorrect piece of information, before then after the release. Where to watch the film goes from 37% to 1%, the release date from 11% to 1%; awards and festivals rises from 4% to 8%. By AI assistant, all fall: Gemini from 34% to 10%, Vibe (formerly Le Chat) from 28% to 13%.before the releaseafter the releaseBy topicThe right film (not a namesake)2% → 1%Release date11% → 1%Where to watch it37% → 1%Storyline2% → 3%Awards and festivals4% → 8%Director1% → 0%Lead actors3% → 1%Genre1% → 2%0%50%100%By AI assistantClaude15% → 5%Gemini34% → 10%Vibe (formerly Le Chat)28% → 13%ChatGPT6% → 4%Perplexity10% → 6%0%50%100%
Incorrect information, topic by topic and AI by AI, before and after the release. Real campaign, names and values changed.

Several figures, not a single score

A single “visibility score” is reassuring, but it hides what matters. We report six figures, drawn from the same answers: visibility, the breakdown by AI assistant, recommendation, image, facts and competitors. Each with its range.

  • Visibility

    34%

    272/800 answers that mention the brand

    range 25–44%

  • By AI assistant

    • ChatGPT38%
    • Gemini24%
    • Vibe34%
    • Claude28%
    • Perplexity36%

    calculated question by question: the ranges overlap, no clear gap

  • Recommendation

    28%

    224/800 answers that recommend it

    range 19–38%

  • Image

    • positive 58%
    • mixed 31%
    • negative 11%

    recurring criticism: a bit expensive

  • Facts

    • correct 86%
    • imprecise 6%
    • incorrect 5%
    • needs checking 3%

    incorrect = both judges agree

  • Competitors

    • Competitor A31%
    • Your brand24%
    • Competitor B18%
    • Competitor C11%
The six figures from one measurement date, each with its range. Fictional example.

The breakdown by AI assistant is enough to show why. For the same film, in the same week, some AI assistants mentioned it in about eight answers out of ten, and another in fewer than two out of ten. An average of those figures describes no real AI.

Before, during, after: real change or just noise?

A rise or a fall only counts if it exceeds what the measurement can tell apart. We compare dates question by question, on the same questions asked each time.

On the film, the curve tells three things. In the week of the release, its visibility rises sharply: a real peak. The following week, it falls back almost to its starting level. Between the first and the last date, there is no real difference. The share of answers that recommend going to see it, however, has genuinely risen.

To read these movements, we keep a control: the other films mentioned in the same answers. If they all move together, the movement comes from the moment (the release season, the state of the web, an update to the AI), not from your campaign. If they stay stable, the movement is specific to the film, without proving that the campaign caused it.

Share of answers that mention the film, at each date

Before, during, after the releaseMentioned when asked which films are out: 49%, then 61% in the week of the release, then 52%: a peak, with no real gap between before and after. Mentioned when asked which film to go and see: 28%, 44%, then 39%: a real gap. Other films stay around 12%.Cinema releaseBeforeReleaseAfterreal gap49%61%52%which films are out28%44%39%which film to see12%13%12%other films (benchmark)
  • Mentioned when asked which films are out
  • Mentioned when asked which film to go and see
  • Other films, on average (benchmark)
The film’s visibility around its release, with the other films as a benchmark. Real campaign, names and values changed.

What the measurement changed

It first corrected a hunch. The buzz around the film, very present in the celebrity press, appeared in almost none of the AI assistants’ answers. What counted were the reference sites: film listings, guides to new releases, online encyclopaedia entries. And the campaign’s articles weighed less and less there as the weeks went by.

It then pointed to specific actions: which listings to correct, which angle to take to which media, which AI to monitor. And a lesson for the next release: measure before the press tour, to steer the interviews rather than observe afterwards.

That is what we expect from a measurement: not a percentage to show off, but a basis for making decisions.

Want to know what AI assistants say about your brand, your product or your film? We offer a free initial assessment. Book a call

Notes

The charts come from real campaigns run by Reflets. Names and values have been changed to protect our clients’ confidentiality.

Ranges: 95% confidence intervals, calculated question by question (repeated answers to the same question count together), and never narrower than if every answer were independent. They apply to the questions asked, not to all conversations with AI assistants.

Human review: if no error is found in 30 randomly drawn answers of the same type, the reading error rate for that type is at most 9.5%, with 95% confidence (exact upper bound of the binomial distribution, known as one-sided Clopper-Pearson). Hence the “less than 10%” in the chart.

Our method draws in particular on Schulte et al. (2026), Don’t Measure Once: Measuring Visibility in AI Search, arXiv:2604.07585. The studies cited are preprints: support, not proof.

Portrait of Yvan Forestier

Yvan ForestierCo-founder of Reflets

Back to top

Let’s talk about what’s taking up your time.

A 30-minute video call, booked in our calendar.

Contact us