How we work

We never pass off noise as a result.

On Monday, we ask ChatGPT the same question 10 times: the brand is mentioned 4 times. On Tuesday: 6 times. Nothing has changed. In a report, that is written up as +50%.

How we work

No promise without a measurement.

  1. Measure

    A baseline in numbers, before anything else.

  2. Prepare

    We get your data in order with you. Ready to grow.

  3. Propose

    Build, buy or make better use of what you have.

  4. Prove

    Measured again afterwards. Not a gut feeling.

Our method applied to ourselves

Our protocol

  1. 1. We repeat, we vary

    Each question, 10 times, to each AI assistant: it never answers the same way twice. The more different questions we ask, the more precise the figure.

  2. 2. We check

    Every measurement is repeated on another day. If the two disagree, we draw no conclusion. And a person reviews answers drawn at random, to check that the machine reads them correctly.

  3. 3. We prove

    Every figure comes with its range (a 95% confidence interval), which shows how precise it is on the questions asked, and links back to the answers we kept.

Result: between two dates, a real gap of 10 percentage points is detected more than 9 times out of 10 (in simulation, with the full protocol).

Example measurement

What two AI assistants answer about a film at its release: share of answers that mention it, against its competitors

Mentioned by ChatGPT

  1. The film 40%
  2. Competitor 1 5%
  3. Competitor 2 0%
  4. Competitor 3 0%

Mentioned by Gemini

  1. The film 95%
  2. Competitor 1 35%
  3. Competitor 2 15%
  4. Competitor 3 5%

Demonstration measurement on a recent film, not client work, and shorter than a campaign: rough figures, with no range. Film and competitor names anonymised.

Our measurement method in detail

Our one limit

The “before” measurement has to be taken before: it cannot be rebuilt afterwards.

Let’s talk about what’s taking up your time.

A 30-minute video call, booked in our calendar.

Contact us