I know this is going to be slightly controversial because everyone's excited about using LLMs to automate audits right now. But after running visibility work for local clients for a while, I had to make a hard decision: I don't trust AI to evaluate whether a business is visible or not, and I think a lot of people in this space are making a quiet mistake by doing so.

Here's the core problem.

AI assistants hallucinate. Not sometimes — regularly. They produce confident, well-formatted, citation-flavored answers that are factually wrong. They even ship with their own disclaimers telling you this. And the same prompt on Tuesday returns a different answer on Thursday. That variability isn't a quirk. It's a structural property of how these models work.

Now apply that to client work. You run an audit. You tell a client "you're missing these signals" or "you're doing well on these factors." Next week, the model disagrees with itself. What did you actually measure? Nothing defensible.

For anything that's going to inform a client recommendation — something they're going to act on, spend money on, or report upward — you need a deterministic result. True or false. Same answer every run. Not "the model feels like this is probably present."

The specific things I'm talking about auditing are structured data, NAP consistency, schema implementation, page load behavior, entity clarity, citation accuracy — the kind of factors that actually influence whether an AI assistant recommends a business or surfaces it in a response. These are binary. Either the structured data is there or it isn't. Either the name matches across sources or it doesn't. Either the page responds cleanly or it doesn't.

AI is genuinely useful for a lot of tasks in this workflow. Writing, summarizing, brainstorming content angles, drafting outreach. Fine. But using it as the measurement instrument itself? That's where I think people are going to get burned, and the clients are going to be the ones paying for it when the audit doesn't hold up three weeks later.

Curious if others have hit this wall. How are you separating the "AI as tool" from "AI as judge" in your audits?