Features

DeepScanBot comes with a comprehensive set of features designed for production web crawling at scale.

---

Core Features

⚡ Concurrent Crawling

Multi-threaded architecture with configurable concurrency limits for optimal performance.

# Use 10 concurrent workers
deepscanbot scan https://example.com concurrency=10

# Control per-host limits
deepscanbot scan https://example.com concurrency=20 host-concurrency=2

# Auto-scale to CPU cores (default)
deepscanbot scan https://example.com

The crawler automatically uses all available CPU cores by default. You can override this with the concurrency option for fine-grained control.

🎯 Configurable Crawl Depth

Control how deep the crawler explores linked pages.

# Single page only (no link following)
deepscanbot scan https://example.com depth=0

# Shallow crawl (2 levels deep - default)
deepscanbot scan https://example.com

# Deep crawl (5 levels deep)
deepscanbot scan https://example.com depth=5

🛡️ Robots.txt Compliance

DeepScanBot respects robots.txt rules by default, with an option to bypass when needed.

# Default: respects robots.txt
deepscanbot scan https://example.com

# Bypass robots.txt restrictions
deepscanbot scan https://example.com ignore-robots=true

🔄 Retry Logic

Automatic retry with exponential backoff for transient failures such as timeouts, rate limits (429), and server errors (5xx).

# Retry up to 3 times with 2 second backoff
deepscanbot scan https://example.com retries=3 retry-backoff=2s

# Aggressive retry for unstable sites
deepscanbot scan https://example.com retries=5 retry-backoff=5s timeout=30

The crawler also respects Retry-After headers from servers, adding extra backoff for 429 responses.

⏱️ Politeness & Rate Limiting

Avoid overwhelming servers with configurable delays and per-host concurrency limits.

# Wait 1 second between requests to the same host
deepscanbot scan https://example.com delay=1s host-concurrency=1

# Polite crawl with low concurrency
deepscanbot scan https://example.com delay=500ms concurrency=2 host-concurrency=1

Advanced Features

📁 Content Filtering

Filter crawled pages by MIME type and enforce page size limits.

# Only crawl HTML pages
deepscanbot scan https://example.com content-types="text/html"

# Crawl HTML and PDF files
deepscanbot scan https://example.com content-types="text/html application/pdf"

# Limit page size to 1MB
deepscanbot scan https://example.com size=1024

# Allow all content types
deepscanbot scan https://example.com content-types="*/*"

🌐 Proxy Support

Route traffic through HTTP/HTTPS proxy servers.

# Use HTTP proxy
deepscanbot scan https://example.com proxy=http://127.0.0.1:8080

# Use HTTPS proxy
deepscanbot scan https://example.com proxy=https://proxy.example.com:3128

# Disable TLS verification for proxy
deepscanbot scan https://example.com proxy=http://127.0.0.1:8080 insecure=true

🗺️ Sitemap Discovery

Automatically discover and crawl URLs from the site's sitemap.xml, including nested sitemap indexes.

# Discover and crawl sitemap URLs
deepscanbot scan https://example.com sitemap=true

# Full crawl with sitemap discovery
deepscanbot scan https://example.com depth=5 sitemap=true concurrency=10

▶️ Resume Mode

Resume interrupted crawls without recrawling already visited URLs.

# Initial crawl (may be interrupted)
deepscanbot scan https://example.com depth=3 output=my-crawl

# Resume from where you left off
deepscanbot scan https://example.com depth=3 resume=true output=my-crawl

🌍 Cross-Domain Crawling

Optionally follow links to external domains beyond the starting URL.

# Follow links to other domains
deepscanbot scan https://example.com cross-domain=true

# Cross-domain with sitemap
deepscanbot scan https://example.com cross-domain=true sitemap=true

📊 Multiple Output Formats

Export results in JSON or plain text format with detailed analytics.

# Text output (default)
deepscanbot scan https://example.com output=results

# JSON output with full summary
deepscanbot scan https://example.com json=true output=results

# JSON output with show source
deepscanbot scan https://example.com json=true show-source=true

🔒 TLS & Security Options

Control TLS verification and redirect behavior.

# Disable TLS verification (for self-signed certs)
deepscanbot scan https://example.com insecure=true

# Disable following redirects
deepscanbot scan https://example.com disable-redirects=true

# Ensure unique URLs only
deepscanbot scan https://example.com unique=true

CLI Reference

Commands

deepscanbot <command> [options]

Command	Description
scan	Crawl and analyze a website
version	Show installed version
doctor	Verify installation and environment
config	Manage CLI configuration
completion	Generate shell completion script
help	Show help for any command

Scan Options

deepscanbot scan <url> [key=value ...]

Option	Default	Description
depth	2	Maximum crawl depth
timeout	2	Request timeout in seconds
concurrency	0	Max concurrent workers (0 = CPU)
host-concurrency	0	Max concurrent requests per host
content-types	"text/html"	Allowed MIME types
output	"crawler_results"	Output filename
json	false	JSON output
size	-1	Page size limit in KB
proxy	""	Proxy URL
unique	false	Unique URLs only
show-source	false	Show URL source
disable-redirects	false	Disable redirects
insecure	false	Disable TLS verification
retries	0	Retry attempts
retry-backoff	"1s"	Retry backoff duration
delay	"0s"	Politeness delay
sitemap	false	Discover sitemap.xml
resume	false	Resume mode
ignore-robots	false	Ignore robots.txt
cross-domain	false	Follow external links