I Asked ChatGPT the Same 241 Local Questions Before and After the August Swap. Reddit Went to Zero. The BBB Took Its Seat.

ChatGPT changed where it gets its answers. Overnight. Without telling anyone.

The public evidence is loud. Otterly measured Reddit citations falling 73.4 percent across 16 brand reports the week after August 8, while Google, AI Mode, Gemini, and Perplexity barely moved. Qwairy put Reddit's share of ChatGPT search citations down 95 percent in a week, with forums down 70 percent, business directories down 93 percent, and institutional sources up 73 percent.

Those are brand prompts. Tesla, Amazon, Udemy. Nobody asks ChatGPT whether Tesla is legit. People do ask it for a roofer in the next town over and a DUI lawyer at 2 a.m., and that is the half of AI search my clients live in.

So I didn't run a new test. I opened the one we already run. Every month the Local Howl app asks ChatGPT, Perplexity, Gemini, and Google AI Mode the same fixed set of local service prompts for every client and stores every answer and every cited source. Two runs landed before the swap. One landed after. That is a before and after nobody planned, which is the best kind.

241 identical prompts, 62 local businesses, three runs

The ChatGPT runs went out July 11, August 1, and October 1. The September 1 run failed: 253 of 256 prompts came back with an HTTP 500 from the data provider. I'm leaving that hole in on purpose. It is the measurement problem in one sentence.

The sourcing flipped. Reddit was ChatGPT's favorite local source before the swap: 27 citations in July, 35 across 25 prompts in August. In October: zero. Not fewer. Zero. The BBB went the other way, from 4 citations to 10 to 42, now showing up in 35 of the 241 answers. Angi went from 2 to 18. Birdeye went from nothing to 8. Directories were 5.7 percent of ChatGPT's local sources in August and 18 percent in October. Forums and social went from 9.7 percent to 2.4.

Answers got sources. In July, 112 of the 241 answers cited nothing at all. In August, 73. In October, 11. The average answer went from 1.3 cited domains to 2.6.

The businesses' own sites got more credit. Before the swap ChatGPT cited a client's website on 4 of the 241 prompts. After, 13. Small numbers, but a tripling, and in the direction the national data says: first-party pages up, forums down.

And here is the part that should scare anyone sending a monthly AI visibility report. The client was named in 80 answers in August and 89 in October. Net plus nine. Great slide. Underneath it, 29 prompts gained the name and 20 lost it. Forty-nine of 241 prompts, one in five, flipped. A single before and after snapshot would have reported a win and missed that a fifth of the ground moved.

The churn was always there. The swap changed the mix.

Before the swap the source lists were already unstable. Between July 11 and August 1, with no model change, only 16 of 91 prompts that cited sources both times cited the exact same list. After the swap it was 3 of 163. And 91 of those 163 shared not one source with their August answer. Same question, same business, completely different witnesses.

Perplexity, which had no swap, flipped on whether it named the business for about 23 percent of these same prompts between July and August. So the honest read is not that ChatGPT broke. It is that every engine is noisy month to month, and ChatGPT added a structural shift on top of the noise. If you report one screenshot, you are reporting the noise.

The local twist is the directories. The national studies say ChatGPT dumped business directories along with Reddit. In local service answers it did the opposite: the BBB, Angi, Birdeye, and HomeAdvisor are now carrying the answer. That matches what I saw from the other direction in the ask-what versus ask-whether test, where trust questions were answered almost entirely by third parties. After August, ChatGPT asks those third parties more often, for every kind of local question.

Measure like a scientist. Fix like a local.

One. Freeze the prompt set. Same prompts, same engines, same cadence, every month, and write the methodology down so a model swap shows up as a step change instead of a mystery. This is the boring half of the AI search KPI check, and it is the only half that survives the next swap.

Two. Report flips, not just totals. Named in 89 answers tells the owner nothing. Gained 29, lost 20, here are the 20 tells them where to work. Annotate failed runs instead of quietly averaging around them.

Three. Fix the new witnesses first. If ChatGPT now reads the BBB on 35 of 241 local prompts, the BBB profile is a homepage. Accreditation, matching name and address, current categories, a real response to every complaint. Same for Angi and whatever review syndication page a vendor set up three years ago.

Four. Make the site the documentation. ChatGPT went from citing client sites on 4 prompts to 13, and the national studies say first-party and documentation pages are what replaced the forums. Put the plain facts on the page: what you do, where, for whom, licensed by whom. Write service pages like a spec sheet, not a brochure. The model is shopping for verifiable statements.

Five. Do not rebuild the strategy around Reddit going to zero. It went to zero in one model update, and one model update can bring it back. Diversify the witnesses so no single source swing takes the answer with it.

Stop screenshotting ChatGPT. Track it.

What I did above only works because the same prompts were saved and rerun on a schedule. If you don't have an app doing that, the AI Visibility features in Semrush are the closest thing to buying the methodology: brand mentions, citations, sentiment, and share of voice across ChatGPT, Perplexity, Gemini, Google AI Mode, and Claude, refreshed weekly on the same basis, with the cited sources listed under each mention. The source list is where the BBB and Angi story above shows up for your business.

Pair it with Traffic Analytics so a swing in AI referral traffic sits next to the rest of the traffic picture instead of surprising you on a client call. A citation drop that coincides with a flat traffic line is noise. One that coincides with a traffic cliff is a meeting.

One snapshot is a rumor. Three runs is a story.

ChatGPT's August swap hit local search too, just not the way the national headlines describe. Reddit left. The BBB, Angi, and the businesses' own sites moved in. Answers stopped coming back empty. And a fifth of the prompts flipped on whether the business got named, which no single screenshot will ever show you.

Pick your prompts. Run them every month. Write down what changed. That is the whole trick, and it is the only one that still works after the next model swap, which, given the last two months, is probably already scheduled.