Skip to main content

Usage Guide

Learn how to use DeepScanBot effectively for production web crawling.

Basic Usage​

After installing DeepScanBot via npm, use the deepscanbot command:

deepscanbot scan https://example.com

This will:

  • Crawl the start page
  • Follow links up to 2 levels deep
  • Output results to crawler_results.txt

Version Information​

Check the installed version of DeepScanBot:

# Using the --version flag
deepscanbot --version

# Or using the version subcommand
deepscanbot version

Common Use Cases​

Single Page Crawl​

Crawl only the specified URL without following any links:

deepscanbot scan https://example.com depth=0

Full Site Crawl​

Crawl an entire website with greater depth and higher concurrency:

deepscanbot scan https://example.com depth=5 concurrency=20

JSON Output​

Export results in JSON format for programmatic processing and integration:

deepscanbot scan https://example.com json=true output=results

Crawl with Proxy​

Route all traffic through a proxy server:

deepscanbot scan https://example.com proxy=http://127.0.0.1:8080

Content-Type Filtering​

Filter which file types to download. Multiple types must be quoted as a single argument.

# Only HTML pages
deepscanbot scan https://example.com content-types="text/html"

# Allow all content types
deepscanbot scan https://example.com content-types="*/*"

Unique URLs Only​

Avoid processing duplicate URLs:

deepscanbot scan https://example.com unique=true

Resume a Crawl​

Stop and resume a crawl without recrawling already visited URLs:

# Initial crawl
deepscanbot scan https://example.com depth=3 output=my-crawl

# Resume (skips already crawled URLs)
deepscanbot scan https://example.com depth=3 resume=true output=my-crawl

Polite Crawling​

Add delays to avoid overwhelming servers:

deepscanbot scan https://example.com delay=500ms host-concurrency=2

Crawl with Retry and Error Handling​

deepscanbot scan https://docs.example.com depth=2 retries=3 retry-backoff=2s delay=1s host-concurrency=1

Retries failed requests up to 3 times with exponential backoff, waits 1 second between requests to the same host, and allows only 1 concurrent request per host.

Cross-Domain Crawl with Sitemap​

deepscanbot scan https://example.com depth=3 cross-domain=true sitemap=true concurrency=10 host-concurrency=2 json=true

Discovers URLs from sitemap.xml, follows links to any domain, with 10 total workers and 2 per host.

Output Formats​

Text Output (Default)​

https://example.com [status=200] [result=passed]
https://example.com/about [status=200] [result=passed]
https://example.com/contact [result=discovered]
https://example.com/admin [result=skipped] [skipped=disallowed by robots.txt]

JSON Output​

{
  "start_url": "https://example.com",
  "started_at": "2024-01-01T00:00:00Z",
  "finished_at": "2024-01-01T00:00:05Z",
  "duration_ms": 5234,
  "summary": {
    "total": 45,
    "passed": 30,
    "failed": 2,
    "skipped": 8,
    "discovered": 5,
    "skipped_by_robots": 3,
    "skipped_by_domain": 2,
    "skipped_by_duplicate": 2,
    "skipped_by_content_type": 1,
    "skipped_by_depth": 0,
    "skipped_by_other": 0,
    "retried_requests": 2,
    "max_depth": 3,
    "urls_by_status_code": {
      "200": 30,
      "404": 1,
      "500": 1
    },
    "skipped_by_reason": {
      "disallowed by robots.txt": 3,
      "outside domain scope": 2,
      "duplicate": 2,
      "content type not allowed": 1
    },
    "retry_distribution": {
      "2": 1,
      "3": 1
    }
  },
  "urls": [
    {
      "url": "https://example.com",
      "source": "href",
      "depth": 0,
      "status_code": 200,
      "content_type": "text/html; charset=utf-8",
      "result": "passed",
      "attempts": 1
    }
  ],
  "skipped": [
    {
      "url": "https://external.com/page",
      "source": "href",
      "depth": 1,
      "result": "skipped",
      "skipped_reason": "outside domain scope"
    }
  ]
}

Exit Codes​

CodeMeaning
0Success
1Error (invalid arguments, crawl failure)