On this page
A hosting rule is often the first one a site adds to a VPN check: blockCondition: { isHosting: true } stops the scraper on a rented cloud server. It also stops Googlebot. On November 2, 2026, we matched every address in the 13 range files that Google, Bing, OpenAI, Perplexity, Anthropic and Apple publish for their crawlers against our data. Every Bing, OpenAI, Perplexity and Anthropic address, and every Google one but a /21 of its user-triggered agents, is high-confidence hosting, and none is a VPN, proxy, Tor exit, relay or CDN. The fix isn't a different classification. It's an allowlist of those published ranges that runs before the check. Every command and output below ran on November 18 against the live API, with vpndetection-express 2.0.31 on Express 5.3.0 and Node.js 22.
What a hosting rule does to a crawler
This is the rule, in Express, as before.js, run on a paid plan's key, since isHosting comes with one (pricing):
import express from 'express';
import { vpndetection } from 'vpndetection-express';
const app = express();
app.set('trust proxy', 'loopback');
app.use(vpndetection({
apiKey: process.env.VPNDETECTION_API_KEY,
blockCondition: { isHosting: true },
}));
app.get('/', (req, res) => res.send('allowed\n'));
app.listen(3000);Our tests call the app from the same machine, with the visitor's address in X-Forwarded-For as a proxy in front would send it, and trust proxy believes it from loopback. Googlebot's request, from an address in Google's own common-crawlers.json (once googlebot.json), gets the same answer a scraper does:
curl -s -w ' %{http_code}\n' \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
-H 'X-Forwarded-For: 66.249.66.1' localhost:3000/{"error":"access denied"} 403A visitor on a home broadband line got allowed. The API answers 66.249.66.1 with is_hosting true, hosting.provider google at high confidence, and is_vpn false. That's the right answer: it's a Google datacenter address, not a home line.
Nothing breaks at once. Pages drop out of the index as Google recrawls them, and Search Console's page indexing report lists them under "Blocked due to access forbidden (403)", which Google's help page says means "your server is returning this error incorrectly". Your own access log shows it sooner. In nginx's combined format, this counts the refused requests that claim to be Googlebot, by address:
awk '$9 == 403 && /Googlebot/ { print $1 }' /var/log/nginx/access.log \
| sort | uniq -cNot every one of those addresses is Google's: a user agent proves nothing, as the section on user agents below shows.
Every published crawler range is hosting
We took every prefix in the 13 files on November 2 and matched it, address by address, against our hosting, VPN, Tor, relay and CDN data and our 30-day residential, datacenter, mobile and public proxy data.
| Published file | IPv4 prefixes | Addresses | High-confidence hosting | VPN, proxy, Tor, relay or CDN |
|---|---|---|---|---|
| Googlebot | 170 | 5,088 | 100% | none |
| Google special crawlers | 136 | 4,352 | 100% | none |
| Google user-triggered fetchers | 534 | 17,088 | 100% | none |
| Google user-triggered fetchers (Google-owned) | 252 | 8,064 | 100% | none |
| Google user-triggered agents | 12 | 70,016 | 97.1% | none |
| Bingbot | 28 | 4,736 | 100% | none |
| OpenAI GPTBot | 18 | 2,704 | 100% | none |
| OpenAI OAI-SearchBot | 39 | 2,560 | 100% | none |
| OpenAI ChatGPT-User | 235 | 69,280 | 100% | none |
| PerplexityBot | 13 | 13 | 100% | none |
| Perplexity-User | 10 | 10 | 100% | none |
| Anthropic | 14 | 1,037 | 100% | none |
| Applebot | 24 | 4,752 | none | none |
Google's five files also list 1,077 IPv6 prefixes, and every one of them overlaps high-confidence hosting space too. The other files list no IPv6.
Three things follow:
- Only the hosting rule needs an allowlist. No crawler address is flagged VPN or proxy, so a rule on
isVpnalone leaves every crawler alone. Keep the two rules apart: refuse VPNs where you need to, and where you can, challenge hosting rather than refusing it. The code below keeps the plain refusal. - Applebot gets through anyway. None of its 4,752 addresses is hosting, since Apple's network isn't a cloud for rent. A site that tests its rule with Applebot sees a crawler pass, and misses that every other one is refused.
- The files change. Google moved its five files to
/static/crawling/ipranges/in late October, all five dated October 30, and the old address ofuser-triggered-agents.jsonstill serves a copy from March. Perplexity's two are dated October 28 and OpenAI's ChatGPT-User file October 27, while Bing's dates from January 2024. Fetch them on a schedule rather than copying prefixes into your code.
Let verified crawlers past the check
crawlers.js fetches the files at startup and keeps their prefixes in Node's built-in net.BlockList, which matches IPv4 and IPv6 subnets, so there's nothing to install. A file that fails to fetch keeps its last good copy. Each fetch gives up after 10 seconds.
import { BlockList, isIPv6 } from 'node:net';
const GOOGLE = 'https://developers.google.com/static/crawling/ipranges';
// Each operator's published ranges. Leave out any agent you don't want in.
const RANGE_FILES = [
`${GOOGLE}/common-crawlers.json`,
`${GOOGLE}/special-crawlers.json`,
`${GOOGLE}/user-triggered-fetchers.json`,
`${GOOGLE}/user-triggered-fetchers-google.json`,
`${GOOGLE}/user-triggered-agents.json`,
'https://www.bing.com/toolbox/bingbot.json',
'https://openai.com/gptbot.json',
'https://openai.com/searchbot.json',
'https://openai.com/chatgpt-user.json',
'https://www.perplexity.com/perplexitybot.json',
'https://www.perplexity.com/perplexity-user.json',
'https://claude.com/crawling/bots.json',
'https://search.developer.apple.com/applebot.json',
];
const lastGood = new Map(); // file -> its prefixes, kept when a fetch fails
let crawlers = new BlockList();
export async function refreshCrawlers() {
await Promise.all(RANGE_FILES.map(async (url) => {
try {
const res = await fetch(url, { signal: AbortSignal.timeout(10_000) });
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const { prefixes } = await res.json();
lastGood.set(url, prefixes.map((p) => p.ipv4Prefix ?? p.ipv6Prefix));
} catch (err) {
console.warn(`kept the last copy of ${url}: ${err.message}`);
}
}));
const next = new BlockList();
for (const prefixes of lastGood.values()) {
for (const prefix of prefixes) {
const [net, bits] = prefix.split('/');
next.addSubnet(net, Number(bits), isIPv6(net) ? 'ipv6' : 'ipv4');
}
}
crawlers = next;
}
export function isVerifiedCrawler(ip) {
if (typeof ip !== 'string') return false;
return crawlers.check(ip, isIPv6(ip) ? 'ipv6' : 'ipv4');
}app.js refreshes the list daily and hands the check to the middleware's skip, which runs before the lookup:
import express from 'express';
import { vpndetection } from 'vpndetection-express';
import { isVerifiedCrawler, refreshCrawlers } from './crawlers.js';
await refreshCrawlers();
const DAY = 24 * 60 * 60 * 1000;
setInterval(() => refreshCrawlers().catch(console.warn), DAY).unref();
const app = express();
app.set('trust proxy', 'loopback');
app.use(vpndetection({
apiKey: process.env.VPNDETECTION_API_KEY,
blockCondition: { isHosting: true },
skip: (req) => isVerifiedCrawler(req.ip),
}));
app.get('/', (req, res) => {
res.send(req.vpndetection ? 'checked, allowed\n' : 'verified crawler\n');
});
app.listen(3000);req.ip has to be the visitor's address, not your proxy's, before any of this means anything. The Express tutorial covers trust proxy.
VPNDETECTION_API_KEY=... node app.js, then the same Googlebot request, and the same user agent from one of Google's IPv6 addresses:
curl -s \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
-H 'X-Forwarded-For: 66.249.66.1' localhost:3000/
curl -s \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
-H 'X-Forwarded-For: 2001:4860:4801:10::1' localhost:3000/verified crawler
verified crawlerA skipped request makes no lookup. The home broadband line still got the full check, and the answer checked, allowed.
A Googlebot user agent proves nothing
Anyone can send Googlebot's user agent. The allowlist goes by address, so the same header from an AWS server outside every file gets the full check, and the hosting rule refuses it:
curl -s -w ' %{http_code}\n' \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
-H 'X-Forwarded-For: 34.192.18.44' localhost:3000/{"error":"access denied"} 403Never allowlist by user agent: it lets in every scraper that copies one. Google's verification page gives a second method, a reverse DNS lookup on the address followed by a forward lookup on the name it returns. That costs two DNS queries per new address, and the range files answer the same question from memory.
The same skip in the other packages
The range files and the check port to any language, and most of our packages take it the same way. Give it the client address the package looks up, not the connection's:
Spring Boot's skip-paths and Symfony's skip_paths match paths, not addresses, and Laravel's package documents no skip. With those three, make the crawler check in your own filter or middleware, ahead of ours.
Which agents to let in
That's your call, and it's one line each in RANGE_FILES. The files cover two kinds of agent, and Anthropic's one file covers both:
- Crawlers (Googlebot, Bingbot, GPTBot, OAI-SearchBot, PerplexityBot, Applebot) fetch pages on their own schedule. Whether one may use your pages, for search or for training, is robots.txt's job, not the VPN check's. Anthropic's help center says an IP block "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file".
- User-triggered fetchers (ChatGPT-User, Perplexity-User and Google's) fetch a page because a person asked for it. Google's page on them says its own fetchers of this kind "ignore robots.txt rules", because a user asked for the fetch. Leave one out of the list and its requests get the full check, so the hosting rule refuses them.
The API answers these lookups for any language. To match addresses on your own servers instead, the Hosting IP database holds the same ranges. For the other side of this rule, what a hosting block does and doesn't catch among VPN users, see Does blocking datacenter ASNs stop VPN users?. The docs have each package's options.


