Introduction
A powerful, production-ready web crawler — designed for performance, reliability, and developer happiness.
What is DeepScanBot?
DeepScanBot is a high-performance, concurrent web crawler that recursively crawls websites, respects robots.txt, handles rate-limiting, and produces comprehensive JSON or text reports.
What Does DeepScanBot Do?
DeepScanBot crawls websites starting from a given URL, follows links up to a configurable depth, and generates detailed reports about every discovered page. It handles the complexities of web crawling automatically:
- Discovers all pages on a website by following internal links
- Respects crawl rules defined in
robots.txt - Handles rate limits with automatic retry and backoff
- Filters content by MIME type for targeted crawling
- Exports results in structured JSON or readable text formats
Who Is It For?
| Role | How DeepScanBot Helps |
|---|---|
| Security Researchers | Audit websites for exposed endpoints, analyze attack surface |
| SEO Specialists | Discover all pages, find broken links, analyze site structure |
| QA Engineers | Verify page availability, test website coverage |
| Data Analysts | Collect structured data from websites for analysis |
| DevOps Engineers | Monitor website health, validate deployments |
| Content Managers | Audit content across large websites |
Key Use Cases
- Website Auditing: Crawl your entire site to verify all pages are accessible
- SEO Analysis: Discover all indexed pages and find crawl issues
- Security Assessment: Map out website structure and identify exposed endpoints
- Data Collection: Gather URLs and metadata for analysis
- Change Monitoring: Track changes across thousands of pages
- Deployment Validation: Verify all routes work after deployment
Key Features
Multi-threaded architecture with configurable worker pools, per-host rate limiting, and CPU-aware auto-scaling.
Automatically respects robots.txt rules with an option to bypass when needed.
Automatic retry with exponential backoff, Smart Retry-After header parsing, and configurable politeness delays.
JSON or text reports with detailed summaries, status code distribution, skip reason breakdowns, and retry analytics.
Auto-discover and crawl URLs from sitemap.xml, including nested sitemap indexes.
Filter downloads by MIME type, enforce page size limits, and focus on specific content types.
Why DeepScanBot?
| Feature | DeepScanBot | Other Crawlers |
|---|---|---|
| Platform | Single binary, no runtime deps | Often requires Python/Node runtime |
| Concurrency | Built-in, configurable | Often limited or complex setup |
| Rate Limiting | Per-host + global semaphores | Manual or absent |
| Robots.txt | Automatic + bypass option | Varies |
| Retry Logic | Exponential backoff + Retry-After | Often basic or missing |
| Output | JSON / Text with full analytics | Usually basic |
| Resume Mode | Built-in | Rare |
| Sitemap | Auto-discovery + nested | Rare |
Quick Start
The fastest way to get started:
# Install via npm (one-time setup)
npm install -g @mindfiredigital/deepscanbot
# Crawl a website with default settings
deepscanbot scan https://example.com
# Crawl with custom options
deepscanbot scan https://example.com depth=3 json=true concurrency=10
# Show version
deepscanbot version
# Verify installation
deepscanbot doctor
No additional runtime dependencies required. The npm package automatically installs the correct binary for your platform.