# Crawl site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, explicit scope patterns, controlled concurrency, stateless continuation, source-completeness reconciliation, and optional non-destructive AI relevance ranking.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.crawl`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer $UPSCRAPE_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "concurrency": 2,
    "detail": "standard",
    "max_depth": 2,
    "max_pages": 5,
    "url": "https://example.com/"
  },
  "capability": "web.site.crawl"
}'
```

## Input


| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `accept_statuses` | `array<integer>` | No |  |
| `ai` | `string` | No | Optional non-destructive relevance ranking for discovered documents. Every deterministic candidate remains in the result; auto leaves candidates unranked if AI is unavailable, while always requires ranking to succeed. Allowed values: `never`, `auto`, `always`. Default: `never`. |
| `allow_browser_fallback` | `boolean` | No | Default: `false`. |
| `allow_truncated` | `boolean` | No | Default: `false`. |
| `concurrency` | `integer` | No | From 1 to 16. Default: `4`. |
| `continuation_token` | `string` | No | Opaque stateless frontier returned by a preceding crawl slice with the same options. A transiently failed URL is retained for at most one bounded retry; permanent failures are not amplified. |
| `delay_ms` | `integer` | No | From 0 to 5000. Default: `0`. |
| `detail` | `string` | No | Allowed values: `summary`, `standard`, `full`. Default: `standard`. |
| `document` | `object` | No |  |
| `document.chunk_chars` | `integer` | No | From 500 to 20000. |
| `document.continuation_token` | `string` | No |  |
| `document.ocr` | `string` | No | Allowed values: `auto`, `never`, `always`. Default: `auto`. |
| `document.outputs` | `array<string>` | No | Requested document outputs. source returns the originally captured bytes: inline through 5.5 MB, otherwise through a hash-verified durable store, and fails if neither delivery path is available. When source is omitted, remote_source is provenance only and contains no bytes. Allowed values: `structured`, `text`, `markdown`, `source`. |
| `document.page_end` | `integer` | No | From 1 to 1000000. |
| `document.page_start` | `integer` | No | From 1 to 1000000. |
| `exclude_patterns` | `array<string>` | No |  |
| `include_patterns` | `array<string>` | No |  |
| `include_subdomains` | `boolean` | No | When false, only the seed host is allowed. When true, hosts are still limited to the seed's registrable domain; unrelated hosts are never added. Default: `false`. |
| `instructions` | `string` | No | Required when ai is auto or always. Describe which documents/files are relevant. Ranking is non-destructive: every deterministic candidate remains in the result. |
| `max_body_bytes` | `integer` | No | From 1 to 104857600. |
| `max_depth` | `integer` | No | From 1 to 10. Default: `2`. |
| `max_pages` | `integer` | No | Maximum number of result pages to fetch. From 1 to 100. Default: `10`. |
| `max_total_bytes` | `integer` | No | From 1 to 104857600. |
| `respect_robots` | `boolean` | No | Default: `true`. |
| `url` | `string` | Yes | Public http(s) site URL to crawl. |

### Example input

```json
{
  "concurrency": 2,
  "detail": "standard",
  "max_depth": 2,
  "max_pages": 5,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`.

Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This example shows one response. Values and optional fields vary with the data available.

```json
{
  "files": [],
  "final_url": "https://example.com/",
  "pages": [
    {
      "capture_backend": "http",
      "content": {
        "bytes_read": 559,
        "category": "page",
        "format": "html",
        "media_type": "text/html",
        "sha256": "[redacted:token]",
        "truncated": false,
        "type_source": "mime"
      },
      "content_type": "text/html",
      "depth": 0,
      "elapsed_ms": 0,
      "files": [],
      "final_url": "https://example.com/",
      "language": "en",
      "links": [
        {
          "text": "Learn more",
          "url": "https://iana.org/domains/example",
          "xpath": "/html[1]/body[1]/div[1]/p[2]/a[1]"
        }
      ],
      "rendered": false,
      "status_code": 200,
      "text": "Example Domain Example Domain This domain is for use in documentation examples without needing permission. Avoid use in operations. Learn more",
      "title": "Example Domain",
      "url": "https://example.com/",
      "warnings": []
    }
  ],
  "stats": {
    "ai_attempts": 0,
    "ai_completion_tokens": 0,
    "ai_mode": "never",
    "ai_prompt_tokens": 0,
    "ai_ranking_mode": "non_destructive",
    "ai_rejected_files": 0,
    "ai_relevant_files": 0,
    "ai_uncertain_files": 0,
    "ai_unselected_files": 0,
    "ai_used": false,
    "concurrency": 2,
    "crawled_pages": 1,
    "effective_delay_ms": 0,
    "elapsed_ms": 2203,
    "estimated_result_bytes": 1151,
    "failed_pages": 0,
    "files_before_compaction": 0,
    "files_found": 0,
    "files_omitted_by_compaction": 0,
    "files_returned": 0,
    "frontier_dropped": 0,
    "max_depth": 2,
    "remaining_frontier": 0,
    "response_compacted": false,
    "returned_pages": 1,
    "robots_applied": true
  },
  "url": "https://example.com/",
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `files` | `array` | 0 items |
| `final_url` | `string` | https://example.com/ |
| `pages` | `array` | 1 items |
| `pages` | `array` | 1 items |
| `stats` | `object` | 26 fields |
| `stats.ai_attempts` | `integer` | 0 |
| `stats.ai_completion_tokens` | `integer` | 0 |
| `stats.ai_mode` | `string` | never |
| `stats.ai_prompt_tokens` | `integer` | 0 |
| `stats.ai_ranking_mode` | `string` | non_destructive |
| `stats.ai_rejected_files` | `integer` | 0 |
| `stats.ai_relevant_files` | `integer` | 0 |
| `stats.ai_uncertain_files` | `integer` | 0 |
| `stats.ai_unselected_files` | `integer` | 0 |
| `stats.ai_used` | `boolean` | false |
| `stats.concurrency` | `integer` | 2 |
| `stats.crawled_pages` | `integer` | 1 |
| `stats.effective_delay_ms` | `integer` | 0 |
| `stats.elapsed_ms` | `integer` | 2203 |
| `stats.estimated_result_bytes` | `integer` | 1151 |
| `stats.failed_pages` | `integer` | 0 |
| `stats.files_before_compaction` | `integer` | 0 |
| `stats.files_found` | `integer` | 0 |
| `stats.files_omitted_by_compaction` | `integer` | 0 |
| `stats.files_returned` | `integer` | 0 |
| `stats.frontier_dropped` | `integer` | 0 |
| `stats.max_depth` | `integer` | 2 |
| `stats.remaining_frontier` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_pages` | `integer` | 1 |
| `stats.robots_applied` | `boolean` | true |
| `url` | `string` | https://example.com/ |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)