1,370,574 requests were refused in the last 24 hours

By site

Refused counts are exact. Request totals include allowed traffic scaled from a 10% sample, marked with a tilde.

By name

Crawlers that announce themselves, grouped by what they are for, with the policy our firewall states for each. The share refused shows whether the policy is holding. Per address-hour is requests per source address per hour: a fleet on rented addresses sits near 1.

By network

Where the requests come from, by autonomous system. A residential ISP with real customers shows several requests per address per hour and is mostly allowed. A cloud or hosting network sending browser-shaped requests at one per address per hour is a fleet, and gets challenged.

Incidents

What actually happened, dated, newest first. The ones marked as our mistake are here because a firewall that blocks people is a worse problem than any scraper.

1. 2026-09-01·Abusive traffic·time.io

   Tencent cloud addresses rate-limited on time.io

   A burst from Tencent's cloud network (AS132203) hit time.io's pages at a rate no person browses at. The warden's proposal was applied through its normal pipeline: 60 requests per minute per address on that network, six-hour expiry.

   Verified afterwards with a real browser: time.io still answered 200 to a person. That check is the whole point of the pipeline.

2. 2026-08-31·Policy·aboutus.com, dns.pizza, time.io, the.com, par.is

   Sixty-one percent of everything is denied, and it still bills

   Reading the August invoice: about 1.5 million requests a day across the constellation, 61% of them bots we refuse. A denied request still counts as an edge request, an observability event and, when it is rate-limited, a rate-limit charge.

   Web Analytics had grown 244% in a month. On aboutus.com, 98,000 "visitors" viewing 5.5 pages each were crawlers that execute JavaScript, so they show up as people in the analytics.

   Blocking is not free. The cheapest bot is the one that never gets a response, which is why the observatory tracks cost, not just counts.

3. 2026-08-26·Our mistake·the.com, aboutus.com

   We blocked 3,864 real people in a day and nobody knew

   the.com had a 100-per-minute rate limit keyed on the TLS fingerprint (JA4). Every user of one browser build shares one fingerprint, so desktop Chrome worldwide was a single bucket. A second rule capped page requests at 30 per minute per address, and one page load is about eight requests.

   Measured from the log drain: 3,864 people received a 429 in 24 hours, from 229 networks over the week. The same night aboutus.com was returning 403 to Chrome 151 users on a consumer-VPN provider's exits, including someone on the pricing page.

   Both rules were loosened the moment it was seen. An hourly check now pages if a deny or rate-limit ever lands on a fingerprint that real browsers use.

   Never key a rate limit on a TLS fingerprint below thousands per minute. Never go under 120 per minute per address on paths a browser loads. Firewalls that stop people cost more than any scraper.

4. 2026-08-19·Blind spot·time.io

   The alarm that fired 37 times for the rule working

   A new check paged 37 times in two hours, at high priority, because Meta's AI crawler (meta-externalagent) was being denied on time.io. That deny was the rule doing precisely its job.

   The fix was to publish intent: the warden now reads the live firewall rules and tells the watcher which crawlers are blocked on purpose. A deny that a live rule explains is not a page. Identity is now the network and the user-agent together, because one cloud (AS8075) is Microsoft's Bingbot and also every Azure tenant running GPTBot.

   An alarm that fires on intended behaviour is how the alarm for unintended behaviour gets ignored.

5. 2026-08-19·Blind spot·aboutus.com, dns.pizza, time.io

   Moving off the metrics API to a log drain

   Vercel caps observability metrics queries at roughly 500 a day per team. The traffic watch hit it on 2026-08-15 and went blind. Support confirmed the cap is fixed on the Pro plan.

   Since 2026-08-19 every request to every site is delivered as a log drain to a small box that rolls it up hourly. The firewall stream is delivered in full; the rest is sampled at 10%. That box is what this page reads.

6. 2026-08-15·Abusive traffic·aboutus.com

   The fleets that pretend to be desktop Chrome

   The largest thing in the logs is not a named crawler. It is millions of requests a week wearing a stock desktop-Chrome user-agent, arriving from cloud and hosting networks (ByteDance's BytePlus and ZEN-NET each send about three million a week), with roughly one request per address per hour.

   A person makes several requests from one address in an hour. A rented fleet makes one from each of thousands. The per-address rate is the tell, and it separates a fleet from a residential ISP whose customers are real.

   These get a challenge rather than a deny: a real browser solves it, and the fleet does not.

7. 2026-08-02·Abusive traffic·aboutus.com, dns.pizza

   A crawler load turned into 66,000 partner-API errors

   aboutus.com renders six dns.pizza partner-API panels on every profile page. Under crawler load that fan-out became a sustained 3 to 5% error rate on the partner API, and 66,000 502s before anyone saw it. The API's own response-code distribution was not being watched.

   Fixed on both sides: a circuit breaker in aboutus.com so a wobble cannot become a stampede, and hourly response-code buckets on the partner API with a watchdog.

8. 2026-07-20·Abusive traffic·aboutus.com, dns.pizza

   Crawler page views exhausted a 250,000-a-day quota by breakfast

   With 1.7 million profile pages and six panel calls per view, crawlers walking aboutus.com used the entire daily partner quota by 09:40 UTC, and the panels went dark for everyone else for the rest of the day.

   The quota was raised to a backstop that only abuse could reach. Quotas between our own properties are there to catch a runaway, not to ration a product.

What bots cost

A refused request is not free: it is billed as an edge request and an observability event before the firewall decides.

How this is measured

The public feed behind this page is at n1.snowc.one/public/bots.json. Partners get the deeper tier (TLS fingerprints, rule names, hourly series) through /api/partner/v1/bots. See also Newborn domains, the daily zone census.