AI Crawlers Spoof Googlebot. robots.txt Can't Stop Them

If you feel like the system is rigged against your bottom line, you're not imagining it — and for the first time, the law is starting to agree.

In April 2025, a US federal court found that Google had illegally monopolised the markets for publisher ad servers and ad exchanges — owning the buyer, the seller and the auction house at once. Publishers aren't just complaining any more; they're filing. Penske Media — owner of Rolling Stone, Billboard and Variety — sued Google in 2025 over AI Overviews, arguing the summaries cannibalise the clicks its journalism earns. In February 2026 the European Publishers Council took a formal antitrust complaint over AI Overviews and AI Mode to the European Commission — which had itself opened a formal investigation, weeks earlier, into Google's use of publisher content for AI. And in the UK, the Competition and Markets Authority has gone furthest of all. Having designated Google with 'strategic market status,' in June 2026 it imposed a world-first requirement: publishers can now refuse to have their work fed into Google's AI answers without being quietly demoted in ordinary search — and Google has nine months to comply.

Read that last part twice. The right to say "use my work in your AI and I'll walk" just became real. But a right you can't enforce is a slogan — and every one of these fights runs on the same fuel: evidence of who took what, and when.

That's exactly where the bots have stopped playing fair.

The spoof is primitive, and it works

When a crawler hits your server it introduces itself with a User-Agent string — a single line of text it writes about itself. A genuine OpenAI crawler says GPTBot or OAI-SearchBot. Google's says Googlebot. Your firewall and your robots.txt make their decisions off that string.

Here's the hole. Nobody dares block Googlebot, so almost every server on the web is configured to wave it straight through. Scrapers, rogue model-trainers and data brokers know this. So they don't break your security — they pin on a name tag that says Googlebot and walk in through the front door you're holding open.

Your content leaves to train a commercial model that will never send you a visitor. And because the legacy SEO stack watches Google Analytics — which only fires when a human browser loads a page — nobody in the building ever sees it happen. The drain is real, it's measured in bandwidth and lost leverage, and it's invisible on the dashboard everyone's actually looking at.

Trusting a User-Agent is trusting a man because his t-shirt says "Security." It's self-reported. It proves nothing.

You can't ask a bot who it is. You verify it.

The good news: identity at the network level is far harder to fake than a line of text. Two checks do the heavy lifting, and neither is magic — each just rests on something a spoofer can't control.

Forward-confirmed reverse DNS (FCrDNS). Take the IP the crawler connected from, look up the name that IP points back to, then resolve that name forward again and confirm it lands on the same IP. An attacker can rename their User-Agent for free — but they can't forge the reverse-DNS record for an IP they don't own, and they can't hold a real connection open from a source IP they can't route. That mismatch is what catches them. (It's a consistency check, not cryptography — anyone selling it as an "un-spoofable cryptographic handshake" is overselling. It doesn't need the hype to work.)

Published IP-range verification. Several major operators — OpenAI and Google among them — publish the exact IP ranges their crawlers run from. A bot announcing itself as GPTBot from a discount cloud block in the wrong region isn't GPTBot; the range gives it away. Where an operator doesn't publish ranges, the reverse-DNS check above does the same job.

Run both and your logs stop being a guess. "Googlebot hit us 4,000 times this week" resolves into "1,200 verified Google, 2,800 impostors wearing the name." That second number is the one almost nobody can show you — because most tools never get past reading robots.txt.

Why this is now a legal asset, not just hygiene

Whether you plan to block the scrapers, exercise a new opt-out the moment it lands, or join a publisher action, you'll be asked the same question: prove it was taken from you.

A robots.txt directive proves what you asked. It says nothing about who ignored you. The proof lives in verified server logs — forensic telemetry that ties a specific extraction to a specific, identity-checked actor on a specific date. That's the gap between "we think AI bots are scraping us" and an evidence chain you could take to a solicitor.

This is the entire reason Unsourced verifies identity instead of trusting the honour system, and the reason we wrote up how to bait, keep or block each crawler rather than telling you to blanket-block. The day you decide to act, you want the receipts already in hand — not a month of "we should have been logging this."

The web stopped being a polite distribution network and became a contested data supply chain. You don't get a vote on the macro shift. You do get to decide who extracts from your servers — and, when it counts, whether you can prove it.

Stop trusting the t-shirt. Demand proof of identity.

— Rene Roach, Unsourced