Features

DeepScanBot comes with a set of features designed for production web crawling.

Core Features

Concurrent Crawling

Multi-threaded crawling with configurable concurrency limits.

# Use 10 concurrent workers
deepscanbot scan https://example.com concurrency=10

# Control per-host limits
deepscanbot scan https://example.com concurrency=20 host-concurrency=2

# Use the default of 8 concurrent workers
deepscanbot scan https://example.com

The crawler uses 8 concurrent workers by default. You can override this with the concurrency option, or pass concurrency=0 to auto-scale to the number of available CPU cores.

Configurable Crawl Depth

Control how deep the crawler explores linked pages.

# Single page only (no link following)
deepscanbot scan https://example.com depth=0

# Shallow crawl (2 levels deep - default)
deepscanbot scan https://example.com

# Deep crawl (5 levels deep)
deepscanbot scan https://example.com depth=5

Robots.txt Compliance

DeepScanBot respects robots.txt rules by default, with an option to bypass when needed.

# Default: respects robots.txt
deepscanbot scan https://example.com

# Bypass robots.txt restrictions
deepscanbot scan https://example.com ignore-robots=true

Retry Logic

Automatic retry with exponential backoff for transient failures such as timeouts, rate limits (429), and server errors (5xx).

# Retry up to 3 times with 2 second backoff
deepscanbot scan https://example.com retries=3 retry-backoff=2s

# Aggressive retry for unstable sites
deepscanbot scan https://example.com retries=5 retry-backoff=5s timeout=30

The crawler also respects Retry-After headers from servers.

Politeness and Rate Limiting

Avoid overwhelming servers with configurable delays and per-host concurrency limits.

# Wait 1 second between requests to the same host
deepscanbot scan https://example.com delay=1s host-concurrency=1

# Polite crawl with low concurrency
deepscanbot scan https://example.com delay=500ms concurrency=2 host-concurrency=1

Advanced Features

Content Filtering

Filter crawled pages by MIME type and enforce page size limits.

# Only crawl HTML pages
deepscanbot scan https://example.com content-types="text/html"

# Crawl HTML and PDF files
deepscanbot scan https://example.com content-types="text/html application/pdf"

# Limit page size to 1MB
deepscanbot scan https://example.com size=1024

# Allow all content types
deepscanbot scan https://example.com content-types="*/*"

Proxy Support

Route traffic through HTTP or HTTPS proxy servers.

# Use HTTP proxy
deepscanbot scan https://example.com proxy=http://127.0.0.1:8080

# Use HTTPS proxy
deepscanbot scan https://example.com proxy=https://proxy.example.com:3128

# Disable TLS verification for proxy
deepscanbot scan https://example.com proxy=http://127.0.0.1:8080 insecure=true

Sitemap Discovery

Automatically discover and crawl URLs from the site's sitemap.xml, including nested sitemap indexes.

# Discover and crawl sitemap URLs
deepscanbot scan https://example.com sitemap=true

# Full crawl with sitemap discovery
deepscanbot scan https://example.com depth=5 sitemap=true concurrency=10

Resume Mode

Resume interrupted crawls without recrawling already visited URLs.

# Initial crawl (may be interrupted)
deepscanbot scan https://example.com depth=3 output=my-crawl

# Resume from where you left off
deepscanbot scan https://example.com depth=3 resume=true output=my-crawl

Cross-Domain Crawling

Optionally follow links to external domains beyond the starting URL.

# Follow links to other domains
deepscanbot scan https://example.com cross-domain=true

# Cross-domain with sitemap
deepscanbot scan https://example.com cross-domain=true sitemap=true

Multiple Output Formats

Export results in JSON or plain text format with detailed analytics.

# Text output (default)
deepscanbot scan https://example.com output=results

# JSON output with full summary
deepscanbot scan https://example.com --json output=results

# JSON output with show source
deepscanbot scan https://example.com --json show-source=true

TLS and Security Options

Control TLS verification and redirect behavior.

# Disable TLS verification (for self-signed certs)
deepscanbot scan https://example.com insecure=true

# Disable following redirects
deepscanbot scan https://example.com disable-redirects=true

# Ensure unique URLs only
deepscanbot scan https://example.com unique=true

CLI Reference

Commands

deepscanbot <command> [options]

Command	Description
scan	Crawl and analyze a website
version	Show installed version
doctor	Verify installation and environment
config	Manage CLI configuration
completion	Generate shell completion script
help	Show help for any command

Scan Options

deepscanbot scan <url> [key=value ...]

Option	Default	Description
depth	2	Maximum crawl depth
timeout	2	Request timeout in seconds
concurrency	8	Max concurrent workers
host-concurrency	2	Max concurrent requests per host (0 = effective concurrency)
content-types	"text/html"	Allowed MIME types
output	"crawler_results"	Output filename base (automatically gets .txt or .json extension)
json	false	JSON output
size	-1	Page size limit in KB (-1 = unlimited)
proxy	""	Proxy URL
unique	false	Unique URLs only
show-source	false	Show URL source
disable-redirects	false	Disable redirects
insecure	false	Disable TLS verification
retries	0	Retry attempts
retry-backoff	"1s"	Retry backoff duration
delay	"0s"	Politeness delay
sitemap	false	Discover sitemap.xml
resume	false	Resume mode
ignore-robots	false	Ignore robots.txt
cross-domain	false	Follow external links

Note on Timeout: The default timeout is 2 seconds, which balances speed and reliability for most websites. Increase this value (for example, `timeout=10` or `timeout=30`) when crawling slow websites or large pages that take longer to respond. For very slow sites, consider `timeout=60` or higher.