Spuncin

Kuvaus

Spuncin turns away the crawlers that harvest your content to train AI models, and leaves everything that brings you readers alone.

It works on two levels at once, because one is not enough. Some crawlers announce themselves and respect robots.txt, so a polite opt-out is all you need. Others ignore it entirely, and only a real refusal stops them. Spuncin writes the opt-out and returns a 403, so both cases are covered.

You choose how strict

Three levels, each one including the previous. The settings page explains what each costs you, so you can decide with the facts in front of you.

1. Training only (recommended). Blocks GPTBot, ClaudeBot, Common Crawl, ByteDance and the rest of the crawlers that feed model training. Your Google ranking is untouched and you lose no visitors.
2. Training and AI search. Also blocks PerplexityBot and the ChatGPT Search crawler. Your site stops being cited in their answers, and those citations carry clickable links today, so this costs you something.
3. Everything except search engines. Also blocks the requests an assistant makes when a person asks it to open one of your pages. On the other side of those is a real reader.

Below the levels there is a full list with a checkbox per crawler, for when the level is not quite what you want.

What is never blocked

Google, Bing and the other search engines are always allowed, verified by reverse DNS so nobody gets through by copying a user agent. The same goes for the bots that build the preview when someone shares your link on Facebook, WhatsApp or elsewhere. Blocking those does damage that takes months to undo, so Spuncin does not offer it as an option.

The part nobody else tells you

Two of the most useful opt-outs are not crawlers at all. Google-Extended and Applebot-Extended have no user agent of their own: Google states that Google-Extended ”doesn’t have a separate HTTP request user agent string”, and Apple says Applebot-Extended ”does not crawl webpages itself”. They exist only as a name to write in robots.txt. A plugin that only inspects user agents can never act on them, whatever it claims. Spuncin writes them into robots.txt, which is the only thing that works.

Honest about the limits

Spuncin blocks crawlers that identify themselves. A scraper pretending to be Chrome is a different problem, and solving it needs tools that sit in front of PHP entirely. We would rather say so than promise something the plugin cannot do.

The list holds 154 known crawlers. On a normal blog you will see a dozen of them; the rest are in there because it costs nothing to keep them.

Privacy

Data collected. When a crawler is turned away, Spuncin adds one to a daily counter for that crawler’s name. Nothing else is recorded. No IP address, no user agent string, no requested URL, no per-request rows.

External services. None. Spuncin does not connect to any third-party service and sends nothing anywhere.

Cookies. None.

Network traffic. When a request claims to be a search engine, Spuncin performs a reverse DNS lookup on the connecting IP to confirm it. That is a DNS query from your own server, sent to your server’s resolver, and it happens only for requests that already match a blocked crawler.

Credits

The crawler names come from the ai.robots.txt project, used under the MIT licence, reproduced in CREDITS.txt. How each crawler is categorised, and therefore what gets blocked at which level, is Spuncin’s own work: the upstream list is a catalogue of AI-related crawlers, not a list of things that should be blocked.