Overview¶

This guide uses Cloudflare's WAF, bot controls, and rate limiting to block abusive crawlers (for example, AI crawlers like Perplexity) before they reach a site hosted on DigitalOcean App Platform. Neither Namecheap nor DigitalOcean offers a managed WAF that covers this case, so Cloudflare sits in front.

Examples use andrewmercer.net. Substitute your own domain.

Cloudflare's dashboard layout changes from time to time. If a menu name below doesn't match exactly, look for the closest equivalent.

Prerequisites¶

• The domain's DNS is hosted on Cloudflare.
• The site's web records are proxied (orange cloud), with SSL/TLS set to Full (strict).

If you haven't done that yet, follow Moving DNS from Namecheap to Cloudflare first. WAF rules only apply to proxied traffic. A grey-clouded record bypasses all of it.

Quick check:

curl -sI https://www.andrewmercer.net | grep -i -E 'server|cf-ray'
# server: cloudflare and a cf-ray header mean the proxy is on

Free-plan limits¶

Everything below works on the Free plan. Free includes a small number of custom rules (5 at the time of writing) and one rate-limiting rule, so the rules below combine conditions where possible.

Part 1: Bot controls¶

Step 1: Block AI crawlers in one click¶

Go to Security → Bots (in newer dashboards this may be under AI Crawl Control) and turn on Block AI bots, set to block on all pages.

This uses Cloudflare's managed list of verified AI crawlers (PerplexityBot, GPTBot, ClaudeBot, and others), including detection of crawlers that don't identify themselves honestly.

Step 2: Turn on Bot Fight Mode¶

Under Security → Bots, enable Bot Fight Mode. It challenges traffic matching known bad-bot patterns.

Bot Fight Mode can interfere with legitimate automation hitting your site (uptime monitors, your own scripts, webhooks). If something breaks, check Security → Events first.

Part 2: Custom WAF rules¶

Step 3: Block by user agent¶

Go to Security → WAF → Custom rules → Create rule.

• Rule name: Block abusive crawlers
• Expression (click Edit expression):

(lower(http.request.headers["user-agent"][0]) contains "perplexity")
or (http.user_agent contains "Bytespider")
or (http.user_agent contains "Amazonbot")
or (http.user_agent contains "meta-externalagent")

On the Free plan, if lower() or header-map syntax isn't accepted, use the simpler form:

(http.user_agent contains "Perplexity")
or (http.user_agent contains "perplexity")
or (http.user_agent contains "Bytespider")

• Action: Block
• Deploy

Add or remove user agents depending on what shows up in your logs.

Step 4: Block Perplexity by IP range (catches spoofed user agents)¶

Perplexity publishes its crawler IP ranges:

• https://www.perplexity.com/perplexitybot.json
• https://www.perplexity.com/perplexity-user.json

Fetch them and extract the CIDRs:

curl -s https://www.perplexity.com/perplexitybot.json \
  | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'

Then either:

• Option A: Create an IP list (Manage Account → Configurations → Lists, if available on your plan), then use ip.src in $perplexity_ips in a custom rule.
• Option B: Paste the CIDRs straight into a custom rule:

(ip.src in {192.0.2.0/24 198.51.100.0/24})

Action: Block. These ranges change over time, so recheck them periodically, or script the update with the Cloudflare API.

Step 5: Allow-list anything you trust (optional)¶

If you have monitoring or CI that hits the site, create a custom rule placed first in order:

• Expression: (ip.src eq 203.0.113.10) or a user-agent match for your monitor
• Action: Skip → select the bot and rate-limit features to bypass

Part 3: Rate limiting¶

Step 6: Add a global rate-limiting rule¶

This protects the app from any crawler, named or not, that hammers it.

Go to Security → WAF → Rate limiting rules → Create rule.

• Rule name: Global rate limit
• Match: all incoming requests, or narrow it with not starts_with(http.request.uri.path, "/static/") to exclude static assets
• Characteristics: IP
• Threshold: for example, 100 requests per 10 seconds (Free-plan period options are limited, so pick what's offered)
• Action: Block for 10 seconds
• Deploy

Tune the threshold after watching real traffic for a day or two.

Part 4: robots.txt as a courtesy layer¶

Well-behaved crawlers stop at robots.txt, which saves a WAF hit. Serve this at /robots.txt:

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

Note that Perplexity-User (the fetcher Perplexity uses when a person asks about a page) generally ignores robots.txt. The WAF rules above are what actually enforce the block.

Part 5: Verify¶

Step 7: Test the user-agent block¶

# Should return 403
curl -s -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' \
  https://www.andrewmercer.net/

# Should return 200
curl -s -o /dev/null -w '%{http_code}\n' https://www.andrewmercer.net/

Step 8: Watch the events log¶

Security → Events shows every blocked or challenged request with the rule that matched. Filter by rule name to confirm the crawler traffic is being stopped.

Step 9: Confirm the load dropped¶

In DigitalOcean, go to Apps → your app → Insights and watch CPU and request count. CPU should fall back to normal shortly after the rules are live.

If CPU stays high while Cloudflare shows lots of blocks, check whether traffic is reaching the app directly through the *.ondigitalocean.app hostname, which bypasses Cloudflare. If it is, add app-level checks, for example rejecting requests whose Host header isn't your custom domain.

If the site itself stops loading after enabling the proxy, the problem is usually SSL mode or certificates rather than the WAF. See Part 4 of Moving DNS from Namecheap to Cloudflare.

Undoing a rule¶

If a rule blocks legitimate traffic:

• Custom or rate-limiting rule: toggle it off under Security → WAF. You don't need to delete it.
• Block AI bots / Bot Fight Mode: toggle off under Security → Bots.
• Bypass Cloudflare entirely: set the web records to DNS only (grey cloud) in DNS → Records.

Quick reference¶