Usage Guide

DeepScanBot uses a simple command structure. After installation, you can start crawling websites right away.

Basic Usage

deepscanbot scan https://example.com

This crawls the start page, follows links up to 2 levels deep, and writes results to crawler_results.txt.

Note on Output Files: The output filename is specified without an extension. DeepScanBot automatically appends the appropriate extension based on the output format:

• Text format (default): crawler_results.txt
• JSON format (--json): crawler_results.json

Version Information

deepscanbot --version

Common Use Cases

Single Page Crawl

Crawl only the specified URL without following any links:

deepscanbot scan https://example.com depth=0

Full Site Crawl

Crawl with greater depth and higher concurrency:

deepscanbot scan https://example.com depth=5 concurrency=20

JSON Output

deepscanbot scan https://example.com --json output=results

Note: The --json global flag enables JSON-formatted output for commands that produce structured output (scan, version, doctor, and help).

Crawl with Proxy

deepscanbot scan https://example.com proxy=http://127.0.0.1:8080

Content-Type Filtering

# Only HTML pages
deepscanbot scan https://example.com content-types="text/html"

# Allow all content types
deepscanbot scan https://example.com content-types="*/*"

Resume a Crawl

Stop and resume later without recrawling already visited URLs:

deepscanbot scan https://example.com depth=3 output=my-crawl
deepscanbot scan https://example.com depth=3 resume=true output=my-crawl

Polite Crawling

Add delays to avoid overwhelming servers:

deepscanbot scan https://example.com delay=500ms host-concurrency=2

Crawl with Retry

deepscanbot scan https://docs.example.com retries=3 retry-backoff=2s delay=1s

Retries failed requests up to 3 times with exponential backoff.

Output Formats

Text Output (Default)

https://example.com [status=200] [result=passed]
https://example.com/about [status=200] [result=passed]
https://example.com/contact [result=discovered]
https://example.com/admin [result=skipped] [skipped=disallowed by robots.txt]

JSON Output

Use the --json flag to get structured output:

deepscanbot scan https://example.com --json
deepscanbot --version --json
deepscanbot doctor --json

Sample JSON output:

$ deepscanbot scan https://example.com depth=0 --json
{
  "status": "success",
  "data": {
    "start_url": "https://example.com",
    "output_file": "crawler_results.json",
    "started_at": "2024-01-15T10:30:00Z",
    "finished_at": "2024-01-15T10:30:05Z",
    "duration_ms": 5000,
    "summary": {
      "total": 1,
      "passed": 1,
      "failed": 0,
      "skipped": 0,
      "discovered": 0,
      "max_depth": 0
    },
    "urls": [
      {
        "url": "https://example.com",
        "depth": 0,
        "status_code": 200,
        "content_type": "text/html",
        "result": "passed"
      }
    ],
    "skipped": null
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:05Z",
    "command": "scan",
    "duration_ms": 5000
  }
}

Piping with jq

deepscanbot scan https://example.com --json | jq '.data.summary'
deepscanbot --version --json | jq -r '.data.version'

Exit Codes

Code	Meaning
0	Success
1	Invalid input (bad URL, unknown flag)
2	Validation error
3	Authentication failure
10	Authorization failure (authenticated but lacks permission)
20	Resource not found
30	Network failure
31	Timeout
70	Internal error