Everything you need for web crawling
A complete toolkit for crawling, scraping, and analyzing websites at scale.
Multi-Threaded Crawling
Concurrent architecture with configurable worker pools, per-host rate limiting, and CPU-aware auto-scaling for optimal performance.
Robust & Configurable
Robots.txt compliance, retry logic with exponential backoff, proxy support, TLS options, and depth control out of the box.
Rich Output Formats
JSON or text reports with detailed summaries, status code distribution, skip reason breakdowns, and retry distribution analytics.
Sitemap & Resume Mode
Auto-discover sitemaps, resume interrupted crawls without recrawling visited URLs, and handle 1000+ page sites gracefully.
Rate-Limit Handling
Smart Retry-After header parsing, automatic backoff for 429 responses, and configurable politeness delays between requests.
Content-Type Filtering
Filter downloads by MIME type, enforce page size limits, and focus on specific content types like HTML or PDFs.