AI Visibility Tracking in Python: Build a Prompt-Panel Tracker for Brand Mentions and Citations

A fixed question set, two APIs with web search turned on, a regex, and a confidence interval. Here is the whole pipeline, and the traps that make most homemade trackers lie.

AI visibility tracking means asking AI assistants the questions your buyers ask, on a schedule, and recording whether your brand was named, how it was described and which sources were cited. You can build a working version in about 150 lines of Python: a CSV of prompts, API calls with web search enabled, a mention parser, a citation parser and a score with an interval.

This post is the engineering companion to our article What Is AI Visibility? Why Ranking #1 on Google No Longer Guarantees Customers. That piece explains why it matters. If you want the manual, non-technical version first, our guide AI Visibility Audit: How to Check What AI Assistants Say About Your Business, Layer by Layer walks through the full audit that this tracker automates. This one shows how to measure it yourself in code, and where the measurement breaks.

Last updated: 13 November 2026

What AI visibility tracking actually measures

Before writing code, decide what a row in your dataset means. In our AI visibility tracking setup one row is one answer: one prompt, sent to one engine, on one day, in one run. If you have wondered how to track AI search visibility without a dashboard vendor, this table is the whole data model.

For each row we store:

Two numbers come out of this. Mentions: was your brand named in the answer text? Citations: which URLs did the answer show as sources? They are different signals, and mixing them is the most common reason a tracker reports nonsense. Keeping AI citations vs mentions in separate columns is the most important design decision in this tracker. The short version for engineers: mentions are the target, citations are the debug log.

Step 1: a fixed prompt panel

The prompt panel is the test suite of your AI visibility tracking. Treat it like one: version it, never edit it mid-series, and keep it in one language.

prompt_id,prompt
p01,What is the best way for a B2B software company to show up in ChatGPT answers?
p02,How do I check if AI assistants recommend my brand?
p03,Which companies help businesses get cited in Google AI Overviews?
p04,How can a small team automate customer support with AI agents?

Write 20 to 40 prompts in the words buyers use; you cannot track brand mentions in AI search on questions nobody asks. Pull them from sales call notes, support tickets and your search queries. Mix "best X for Y", "how do I", and "X or Y" questions.

Why freeze the wording? Because wording and language move results more than the brand does. In a study of 12,933 AI responses about 20 brands across eight query languages, three providers and 15 prompts, a mixed model assigned 26.49% of the variance in brand-description scores to the query language and only 1.46% to the brand itself (Zatuchin, arXiv 2607.13304, 2026). Change the language halfway through a series and you have measured your instrument, not your visibility.

Step 2: call the engines with web search on

For AI visibility tracking you want answers that look like what a buyer sees, which means retrieval turned on. Both of the big APIs support it.

OpenAI (Responses API with the web search tool). Cited sources come back as url_citation annotations on the output text (OpenAI docs, Web search).

import os
from openai import OpenAI

oa = OpenAI()  # reads OPENAI_API_KEY

def ask_openai(prompt: str) -> tuple[str, list[str]]:
    resp = oa.responses.create(
        model=os.environ["OPENAI_MODEL"],
        tools=[{"type": "web_search"}],
        input=prompt,
    )
    urls = []
    for item in resp.output:
        if item.type != "message":
            continue
        for part in item.content:
            for ann in getattr(part, "annotations", None) or []:
                if ann.type == "url_citation":
                    urls.append(ann.url)
    return resp.output_text, urls

Gemini (grounding with Google Search). Sources come back in grounding_metadata.grounding_chunks (Gemini API docs, Grounding with Google Search).

from google import genai
from google.genai import types

gm = genai.Client()  # reads GEMINI_API_KEY

def ask_gemini(prompt: str) -> tuple[str, list[str]]:
    r = gm.models.generate_content(
        model=os.environ["GEMINI_MODEL"],
        contents=prompt,
        config=types.GenerateContentConfig(
            tools=[types.Tool(google_search=types.GoogleSearch())]
        ),
    )
    md = r.candidates[0].grounding_metadata
    chunks = (md.grounding_chunks or []) if md else []
    # chunk.web.uri is a redirect link; chunk.web.title holds the source domain
    return r.text or "", [c.web.title for c in chunks if c.web]

Keep model names in environment variables and log them with every row. Good AI visibility tracking records the instrument as carefully as the result. When a provider updates a model, your numbers can shift overnight, and you need to see that in the data.

One honest caveat. API answers are a proxy, not a copy of the consumer app. The apps add memory, personalisation and their own system prompts. Use the API for the trend line, and once a month run a handful of prompts by hand in the apps to check the two still agree.

Step 3: parse mentions

A mention parser is a regex over your brand's aliases, with word boundaries so you do not match inside other words.

import re

ALIASES = ["Your Brand", "YourBrand", "yourbrand.com", "Your Product"]
PATTERN = re.compile(
    r"\b(" + "|".join(re.escape(a) for a in ALIASES) + r")\b", re.IGNORECASE
)

def find_mention(text: str) -> tuple[bool, int | None]:
    m = PATTERN.search(text)
    if not m:
        return False, None
    # position = how many other brand-like list items come before the first hit
    before = text[: m.start()]
    position = len(re.findall(r"^\s*(?:\d+\.|[-*])\s", before, re.MULTILINE)) + 1
    return True, position

Position here is rough: it counts list items before the first hit. That is good enough to separate "named first" from "named seventh", which is what you actually care about.

Do not stop at presence. Save the sentence around each hit and review a sample by hand every month for accuracy. An answer that names you but says you do something you do not is a mention you want to fix, not celebrate.

Step 4: parse citations

Normalise URLs to registrable domains, then flag your own.

from urllib.parse import urlparse

OWN_DOMAINS = {"yourbrand.com"}

def to_domain(u: str) -> str:
    host = urlparse(u).netloc if "://" in u else u
    return host.lower().removeprefix("www.")

def citation_fields(urls: list[str]) -> tuple[list[str], bool]:
    domains = sorted({to_domain(u) for u in urls if u})
    own = any(d == o or d.endswith("." + o) for d in domains for o in OWN_DOMAINS)
    return domains, own

Remember what a citation is and is not. When researchers audited four generative search engines, only 74.5% of citations actually supported the sentence they were attached to, and only 51.5% of generated sentences were fully supported by their citations (Liu, Zhang and Liang, arXiv 2304.09848, 2023). A cited URL tells you what the engine had in hand, not exactly where a claim about you came from. That is why, in AI citations vs mentions, citations are the diagnostic layer.

Step 5: repeat runs and a score with an interval

LLM output is not deterministic, even when you think you pinned it. Repeated-run experiments on two public models across five prompt categories found that non-determinism persists even at temperature 0.0 (Nicholson, arXiv 2601.19934, January 2026). One run per prompt is an anecdote.

Run each prompt on each engine on at least three separate days per month. Then compute share of answer, the share of all answers that name you, with a Wilson interval so small samples do not look more precise than they are:

from math import sqrt

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float, float]:
    if n == 0:
        return 0.0, 0.0, 0.0
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return p, max(0.0, c - h), min(1.0, c + h)

# 30 prompts x 2 engines x 3 runs = 180 answers, named in 54
print(wilson(54, 180))  # (0.30, ~0.24, ~0.37)

Report it as "30% (24-37%) of 180 answers". That one line is your AI visibility score, and it is honest about its own noise. A month that moves from 30% to 33% on this sample is inside the interval: do not call it a win.

Note that the precision comes mainly from the number of distinct prompts, not from repeating the same few prompts many times. Repeat runs of one prompt are correlated with each other. If your interval is too wide, add prompts before you add runs.

Step 6: store, schedule, review

This is the part of AI visibility tracking most teams skip. Append every row to a CSV or a SQLite table, keyed by prompt_id, engine, run_date and run_no. Keep the full answer_text: you will change your parser, and you want to re-score history without new API calls.

import csv, datetime, json

def run_panel(prompts: list[dict], run_no: int, out="runs.csv"):
    today = datetime.date.today().isoformat()
    engines = {"openai-web": ask_openai, "gemini-search": ask_gemini}
    with open(out, "a", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        for p in prompts:
            for name, ask in engines.items():
                text, urls = ask(p["prompt"])
                hit, pos = find_mention(text)
                domains, own = citation_fields(urls)
                w.writerow([p["prompt_id"], name, today, run_no,
                            hit, pos, ";".join(domains), own, json.dumps(text)])

Schedule it with cron or Windows Task Scheduler three times a month on different weekdays. Add retries with backoff, and log failures as failures, not as "not mentioned". A timeout that is counted as a miss quietly drags your score down.

Step 7: add Google's first-party number

For Google's AI features you do not have to simulate anything. Search Console's generative AI performance report shows how many times links to your site were shown in AI Overviews and AI Mode. It reports impressions, not clicks (Google Search Console Help, Generative AI performance report). Export it monthly and put it next to your share of answer. It counts links, not brand names, so it sits with citations in your report.

What to do with the output

Once you can track brand mentions in AI search reliably, the dataset answers practical questions:

• Which prompts never name you? Those are content gaps. Write the page that answers that exact question.
• Which domains get cited most in your category? Rank cited_domains by count across all rows. The top ten is where to publish, pitch or get listed.
• Where are you cited but not named? Rows with own_cited = True and mentioned = False mean the engine is using your explanation without crediting your brand. Put your brand name, product and the key claim together on those pages.
• Which engine is weakest? Split share of answer by engine. Engines differ a lot, and a single blended number hides it.
• Did a fix work? Compare the AI visibility score for the affected prompts before and after the change, with intervals, over at least two monthly cycles.

For the strategy behind these fixes, our Medium post AI Visibility vs SEO: Why Ranking on Google Is No Longer the Whole Story explains why a page-one ranking and an AI mention are separate goals.

Common bugs in homemade AI visibility tracking

FAQ

What is AI visibility tracking?

AI visibility tracking is the routine of asking AI assistants a fixed set of buyer questions on a schedule and recording whether your brand is named, how it is described and which sources are cited. The output is a share-of-answer score plus a list of trusted sources.

Can I track brand mentions in AI search without a paid tool?

Yes. A CSV of prompts, the OpenAI and Gemini APIs with web search on, a regex for your brand names and a spreadsheet are enough to start. A tool saves time on scale and reporting, but the method is the same.

What is the difference in AI citations vs mentions for a tracker?

A mention is your brand named in the answer text; a citation is a source URL shown with the answer. Store them in separate columns; the AI citations vs mentions split is what makes the dataset actionable. Set targets on mentions and use citations to find which sources to work on.

How accurate is an AI visibility score from the API?

It is a reliable trend line if the prompt panel, language, engines and schedule stay fixed, and you report the interval. It is not an exact copy of what each person sees in the consumer apps, so spot-check a few prompts by hand each month.

How often should AI visibility tracking run?

Three runs a month on separate days suits most teams. Run more often only after a major content or product change, and only on the same prompt panel.

How to track AI search visibility in Google specifically?

Use Search Console's generative AI performance report for impressions in AI Overviews and AI Mode, and run your prompt panel through the Gemini API with Google Search grounding for brand mentions.

Wrapping up

AI visibility tracking is a small data pipeline with strict test discipline: a frozen prompt panel, repeat runs, separate mention and citation columns, and a score that carries its interval. Build that, and you can track brand mentions in AI search and know how to track AI search visibility month after month. You will know whether AI assistants name you and why, which no analytics dashboard can tell you.

We build and run this kind of answer monitoring at Sigzen AI, across the major assistants, for teams that would rather read the report than maintain the scripts.