Understanding Dropsolid Shield: Protecting Websites from Overloaded Traffic¶
Info
Dropsolid Shield is currently in beta and is being rolled out to select clients.
Overview¶
This page explains how rate limiting works on our platforms and how it helps protect websites from being overloaded by too much traffic. It also describes how you can request changes, such as allowlisting or blocklisting IP addresses.
What is rate limiting?¶
Rate limiting is a mechanism that restricts how many requests a visitor can make to a website within a certain time period.
When a rate limit is exceeded, the server starts rejecting additional requests with an HTTP 429 Too Many Requests status code. This protects overall performance and ensures the website remains available and responsive for legitimate users.
How rate limiting works¶
Our system automatically classifies every visitor into a traffic category by looking at:
- Who they are – for example: a regular visitor, a search engine bot, or a clearly abusive actor.
- Where they are from – for example: countries from which we expect normal traffic.
- What they are doing – for example: normal browsing versus bulk downloading or aggressive crawling.
Each category has its own rate limits. Some categories are more restrictive than others, depending on how risky or resource‑intensive the traffic is.
Traffic categories¶
We use the following main traffic categories:
- Whitelist (IP allowlist)
Trusted visitors, such as internal team members or approved services and partners. These are not rate‑limited. - Verified Bots
Well-known, legitimate bots such as search engine crawlers (for example Google) or link preview bots. They have a higher limit because they are expected to crawl or fetch more pages. - General
Normal visitors browsing the website. They have a balanced rate limit that protects the platform while keeping the user experience smooth. - Restricted
Visitors whose behavior looks unusual or slightly suspicious, but not clearly malicious. They are subject to lower limits. - Abusive (IP blocklist)
Traffic that is known to be harmful or clearly abusive. These visitors get a very low limit and are effectively blocked very quickly.
Managing rate limits¶
You cannot change rate limiting settings directly yourself, but you can request changes through our support channel.
Requesting an IP allowlist (whitelist)¶
If a trusted partner, integration, or external service needs to send a higher volume of traffic than the default limits allow, you can request to have their IP address added to an IP allowlist (whitelist).
What this does
- Moves the traffic into the Whitelist (IP allowlist) category.
- Removes or significantly relaxes rate limits for that IP, depending on the configuration.
Information to provide
When requesting an IP allowlist, include at least:
- The IP address or IP range.
- The purpose of the traffic (for example: monitoring, integration, import, SEO tool).
- Expected traffic pattern (for example: peak request rate, expected schedule).
Requesting an IP blocklist (blacklist)¶
If you notice that specific IP addresses are causing problems (for example, a flood of unwanted requests, scraping, or attempted abuse), you can request that they be added to an IP blocklist (blacklist).
What this does
- Moves the traffic into the Abusive (IP blocklist) category.
- Applies the strictest limits and effectively blocks most of the traffic from that IP.
Information to provide
When requesting an IP blocklist, include:
- The IP address or IP range.
- Evidence or description of the unwanted behavior (for example: log excerpts, timestamps, URLs being targeted).
Requesting changes to rate limits¶
In some cases you may want to temporarily or permanently adjust rate limits for your website. Typical scenarios include:
- A planned marketing campaign or special event that will drive a traffic spike.
- A bulk data import or content migration.
- Integration with third‑party services that need higher‑than‑normal request rates.
Information to provide
When requesting changes, include:
- The website or environment affected.
- The time frame (for example: temporary increase during an event, or a permanent change).
- Expected peak traffic (for example: estimated requests per second or per minute).
- Any known user agents or IP ranges involved.
User agent classifications¶
Some rate limiting behavior is based on the User-Agent header. We maintain lists of user agents that are whitelisted, verified, or restricted.
Whitelisted user agents¶
These user agents are considered trusted and are not rate‑limited.
| User Agent | Vendor | Purpose |
|---|---|---|
| Googlebot | Google Search indexing | |
| Bingbot | Microsoft | Bing Search indexing |
| DuckDuckBot | DuckDuckGo | DuckDuckGo search indexing |
| facebookexternalhit | Meta | Facebook link previews |
| facebookcatalog | Meta | Facebook link previews |
| Facebot | Meta | Facebook secondary crawler |
| Meta-ExternalFetcher | Meta | User-initiated fetches of individual links |
| Twitterbot | X | X / Twitter link previews |
| Slackbot | Slack | Slack link unfurling |
| Discordbot | Discord | Discord embeds |
| Meta | WhatsApp link previews | |
| TelegramBot | Telegram | Telegram link previews |
| LinkedInBot | LinkedIn link previews | |
| IndeedJobBot | Indeed | Job gathering |
| Feedfetcher-Google | RSS fetching | |
| AppleNewsBot | Apple | Apple News ingestion |
| OAI-SearchBot | OpenAI | ChatGPT search and citations |
| ChatGPT-User | OpenAI | User-triggered page fetches |
| ChatGPT-User/2.0 | OpenAI | Updated user-triggered fetches |
Verified user agents¶
These user agents are considered “good bots” and have higher rate limits than normal traffic, but are still subject to some protection.
| User Agent | Vendor | Purpose |
|---|---|---|
| Slurp | Yahoo | Yahoo search indexing |
| Baiduspider | Baidu | Baidu search indexing |
| YandexBot | Yandex | Yandex search indexing |
| SeznamBot | Seznam | Seznam.cz search indexing |
| NaverBot | Naver | Naver search indexing |
| Qwantify | Qwant | Qwant search indexing |
| PetalBot | Huawei | Huawei Petal Search indexing |
| SkypeUriPreview | Microsoft | Skype previews |
| Viber | Rakuten | Viber previews |
| SignalBot | Signal | Signal previews |
| Snapchat | Snap | Snapchat previews |
| Pinterestbot | Pinterest previews and indexing | |
| AhrefsBot | Ahrefs | SEO crawling |
| MJ12bot | Majestic | Backlink analysis |
| DotBot | Moz | SEO crawling |
| SerpstatBot | Serpstat | SEO crawling |
| SeobilityBot | Seobility | SEO crawling |
| BLEXBot | WebMeUp | SEO crawling |
| MegaIndex | MegaIndex | SEO crawling |
| ia_archiver | Internet Archive | Wayback snapshots |
| archive.org_bot | Internet Archive | Archive crawling |
| Wayback | Internet Archive | Wayback Machine fetches |
| LibraryOfCongress | Library of Congress | Archival research |
| FlipboardProxy | Feed aggregation | |
| Inoreader | Inoreader | Feed aggregation |
| NewsBlur | NewsBlur | Feed aggregation |
| Feedly | Feedly | Feed aggregation |
| Substack | Substack | Feed and post fetching |
| ClaudeBot | Anthropic | Claude live browsing |
| PerplexityBot | Perplexity | AI search indexing |
| Perplexity-User | Perplexity | User-triggered citation fetches |
| DuckAssistBot | DuckDuckGo | AI-generated answers |
| YouBot | You.com | AI assistant fetching |
| MistralAI-User | Mistral | Le Chat browsing |
| GoogleAgent-Mariner | Agentic browser tasks |
Restricted user agents¶
These user agents are allowed to access the site, but are limited more aggressively because of their potential impact or purpose.
| User Agent | Vendor | Purpose |
|---|---|---|
| Meta-ExternalAgent | Meta | Meta fallback and AI-related fetcher |
| SemrushBot | SEMrush | SEO crawling |
| Screaming Frog | Screaming Frog | Site audits |
| SiteAuditBot | Various | Automated site audits |
| GPTBot | OpenAI | Model training |
| anthropic-ai | Anthropic | Claude model training |
| claude-web | Anthropic | Web data collection |
| Google-Extended | Gemini / AI training | |
| Applebot-Extended | Apple | Apple AI training |
| cohere-ai | Cohere | Model training |
| Timpibot | Timpi | Decentralized training datasets |
Whitelisted (allowlisted) countries¶
By default, traffic from the following countries is whitelisted (allowlisted) and treated as normal, expected traffic:
- Belgium (BE)
- France (FR)
- Netherlands (NL)
- Germany (DE)