#crawler traps

1 posts

October 27, 2025 at 05:22

maurycyz.com

lol... babble.c... wouldn't my wife have gotten a copy of that piece of code? 😅 Jokes aside — fun little experiment with bot-crawling here, worth a read. 📚👍🏻

Fed crawlers with nonsense — now they make up 99% of the traffic

I set up an "infinite nonsense" crawler, and it's now responsible for 99% of my server traffic; AI scrapers ignore robots.txt, rotate IPs, and chew through bandwidth and disk space. Blocking, rate-limits, or paywalls fail — the cheapest and easiest thing is to feed the bots junk.

Visit link Permalink

TL;DR: I set up an "infinite nonsense" crawler, and it's now responsible for 99% of my server traffic; AI scrapers ignore robots.txt, rotate IPs, and chew through bandwidth and disk space. Blocking, rate-limits, or paywalls fail — the cheapest and easiest thing is to feed the bots junk.

#crawler traps #llm training #server performance #web scraping