Rate-limiting residential proxy scrapers: IP, /24, ASN all miss

In July, under the heading "Residential proxies", LWN described the scrapers hammering sites like its own: "It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times" (LWN). A residential proxy network rents out home connections as exits, so each of those requests comes from what looks like a household.

The usual advice is to rate-limit by something wider than the address. In September, a site owner being scraped on Hacker News was told to set "strict rate limits per IP / per /24 / per ASN" (Hacker News). We measured each of those keys against our residential proxy data, then built a limiter keyed on what kind of address a request comes from.

Why each key misses

This is the residential proxy IPv4 in our data on October 20, 2026, seen in the last day and in the last 30 days. Networks are origin ASes from public BGP routing data:

A per-IP limit never fills. A scraper renting from a pool of five million addresses can send every request from a new one: in our test below, 40 requests from 40 residential proxy exits all got through a limit of 60 a minute.

A per-/24 counter barely moves either. Over half of one day's /24s held a single proxy address, and 81.9% held three or fewer:

Over 30 days the /24s fill up, to a median of 3 addresses, but a rate limit counts minutes, not months. A /24 with one exit sending two or three requests looks like one quiet household.

A per-network limit lands on the wrong people. One day's pool is spread over 29,134 networks, 4,251 of them with a single address. Those networks are home and mobile ISPs, since that's what makes a proxy residential. A limit tight enough to slow the scraper on one of them slows every customer of that ISP, and the scraper has 29,133 networks left.

Key the limit on the kind of address

What every one of those addresses shares is what kind of address it is. So look each visitor up, put everyone on a residential proxy, a VPN, Tor or a server into one bucket per route, and give that bucket a limit sized for the whole group. Everyone else keeps a bucket of their own.

Here it is in Express, with our middleware, vpndetection-express, and express-rate-limit:

npm install express vpndetection-express express-rate-limit

import express from 'express';
import { ipKeyGenerator, rateLimit } from 'express-rate-limit';
import { vpndetection } from 'vpndetection-express';

const app = express();
app.set('trust proxy', 'loopback'); // nginx on this machine

const lookup = vpndetection({ apiKey: process.env.VPNDETECTION_API_KEY });

// An address no household has to itself: a proxy exit, a VPN, Tor, a server.
function isShared(req) {
    const r = req.vpndetection?.result;
    return Boolean(r?.isResproxy || r?.isMobproxy || r?.isDcproxy
        || r?.isPubproxy || r?.isVpn || r?.isTor || r?.isHosting);
}

// Shared addresses fill one bucket together; everyone else gets their own.
const searchLimit = rateLimit({
    windowMs: 60_000,
    limit: (req) => (isShared(req) ? 30 : 60),
    keyGenerator: (req) => (isShared(req) ? 'shared' : ipKeyGenerator(req.ip, 64)),
    handler: (req, res) => {
        if (isShared(req)) {
            res.status(429).json({ challenge: true }); // show a CAPTCHA here
        } else {
            res.status(429).json({ error: 'slow down' });
        }
    },
});

app.get('/search', lookup, searchLimit, (req, res) => {
    res.json({ results: [] });
});

app.listen(3000);

ipKeyGenerator(req.ip, 64) keys an IPv6 visitor by their /64, the block a home network or a phone normally gets, so stepping to the next address in it doesn't buy a fresh bucket.

We ran it on October 20, 2026, with Node.js 22.22, Express 5.3.0, express-rate-limit 8.7.1 and vpndetection-express 2.0.15, on a Max plan key. Each request came through X-Forwarded-For as nginx would send it, from the same 40 residential proxy exits as before, which we don't print. For comparison, the usual limiter is rateLimit({ windowMs: 60_000, limit: 60 }) with no lookup:

Next, a Comcast home line got its results from a bucket of its own. Then a visitor on the Mullvad server 45.134.140.130, from Mullvad's own server list, got {"challenge":true} on its first request: it shares the bucket the scraper had filled.

That's the price of the shared bucket, and why it ends in a challenge rather than a block. Plenty of people on VPNs are just people, and a residential proxy exit is someone's home connection. Size the shared limit from your own traffic, a few times what flagged visitors send that route in a normal minute, and let anyone who passes the challenge skip it: rateLimit takes a skip function, which can read whatever your challenge sets in the session.

Get the client address right first

Behind nginx or a load balancer, Express sees the proxy's address unless trust proxy names it. We took the trust proxy line out and sent the same requests through nginx's header again. Every visitor became ::1, which the middleware answers itself as a private address, so nobody was flagged and everyone shared one per-IP bucket. 60 proxy exits got through, then the home line got {"error":"slow down"} on its first request. The one sign of it is a line in the middleware's log: "resolved the client address as ::1, which is not a public address".

Don't fix it with true, which believes whatever the visitor writes in X-Forwarded-For. Name your proxies: 'loopback' for nginx on the same machine, or your load balancer's subnet. How to detect VPN users in Express.js tests each setting.

What it costs, and what it misses

Each new address costs one lookup. The middleware caches each answer for an hour, so a returning visitor costs nothing, but a scraper on a new exit for every request costs a lookup for every request. That's why the lookup sits on /search alone, next to the limit, rather than in app.use. An hour is also about right for a verdict: an address can leave a pool within a day (How long do residential proxy IPs last?).

isVpn and isTor come back with a key or without one, and every paid plan adds isHosting and the proxy flags (pricing). A flag your plan doesn't include is undefined, which isShared reads as not set: with no key, all 40 exits got their results, and only the Mullvad visitor went to the shared bucket. If a lookup fails, isShared returns false, so the visitor falls back to a bucket of their own.

The class bucket can't see an exit before our data does. Those addresses still meet the per-IP limit, which is why it stays. And a scraper that signs in needs a limit per account, which no address check replaces. Every flag, and the API itself, is in the docs.