Universal Web API Capability

Crawl site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, explicit scope patterns, controlled concurrency, stateless continuation, source-completeness reconciliation, and optional non-destructive AI relevance ranking.

Capability ID web.site.crawl

Cost From $1.25 per 1,000 requests · 1 credit

Maximum runtime 120 seconds

View raw

# Crawl site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, explicit scope patterns, controlled concurrency, stateless continuation, source-completeness reconciliation, and optional non-destructive AI relevance ranking.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.crawl`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer $UPSCRAPE_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "concurrency": 2,
    "detail": "standard",
    "max_depth": 2,
    "max_pages": 5,
    "url": "https://example.com/"
  },
  "capability": "web.site.crawl"
}'
```

## Input


| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `accept_statuses` | `array<integer>` | No |  |
| `ai` | `string` | No | Optional non-destructive relevance ranking for discovered documents. Every deterministic candidate remains in the result; auto leaves candidates unranked if AI is unavailable, while always requires ranking to succeed. Allowed values: `never`, `auto`, `always`. Default: `never`. |
| `allow_browser_fallback` | `boolean` | No | Default: `false`. |
| `allow_truncated` | `boolean` | No | Default: `false`. |
| `concurrency` | `integer` | No | From 1 to 16. Default: `4`. |
| `continuation_token` | `string` | No | Opaque stateless frontier returned by a preceding crawl slice with the same options. A transiently failed URL is retained for at most one bounded retry; permanent failures are not amplified. |
| `delay_ms` | `integer` | No | From 0 to 5000. Default: `0`. |
| `detail` | `string` | No | Allowed values: `summary`, `standard`, `full`. Default: `standard`. |
| `document` | `object` | No |  |
| `document.chunk_chars` | `integer` | No | From 500 to 20000. |
| `document.continuation_token` | `string` | No |  |
| `document.ocr` | `string` | No | Allowed values: `auto`, `never`, `always`. Default: `auto`. |
| `document.outputs` | `array<string>` | No | Requested document outputs. source returns the originally captured bytes: inline through 5.5 MB, otherwise through a hash-verified durable store, and fails if neither delivery path is available. When source is omitted, remote_source is provenance only and contains no bytes. Allowed values: `structured`, `text`, `markdown`, `source`. |
| `document.page_end` | `integer` | No | From 1 to 1000000. |
| `document.page_start` | `integer` | No | From 1 to 1000000. |
| `exclude_patterns` | `array<string>` | No |  |
| `include_patterns` | `array<string>` | No |  |
| `include_subdomains` | `boolean` | No | When false, only the seed host is allowed. When true, hosts are still limited to the seed's registrable domain; unrelated hosts are never added. Default: `false`. |
| `instructions` | `string` | No | Required when ai is auto or always. Describe which documents/files are relevant. Ranking is non-destructive: every deterministic candidate remains in the result. |
| `max_body_bytes` | `integer` | No | From 1 to 104857600. |
| `max_depth` | `integer` | No | From 1 to 10. Default: `2`. |
| `max_pages` | `integer` | No | Maximum number of result pages to fetch. From 1 to 100. Default: `10`. |
| `max_total_bytes` | `integer` | No | From 1 to 104857600. |
| `respect_robots` | `boolean` | No | Default: `true`. |
| `url` | `string` | Yes | Public http(s) site URL to crawl. |

### Example input

```json
{
  "concurrency": 2,
  "detail": "standard",
  "max_depth": 2,
  "max_pages": 5,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`.

Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This example shows one response. Values and optional fields vary with the data available.

```json
{
  "files": [],
  "final_url": "https://example.com/",
  "pages": [
    {
      "capture_backend": "http",
      "content": {
        "bytes_read": 559,
        "category": "page",
        "format": "html",
        "media_type": "text/html",
        "sha256": "[redacted:token]",
        "truncated": false,
        "type_source": "mime"
      },
      "content_type": "text/html",
      "depth": 0,
      "elapsed_ms": 0,
      "files": [],
      "final_url": "https://example.com/",
      "language": "en",
      "links": [
        {
          "text": "Learn more",
          "url": "https://iana.org/domains/example",
          "xpath": "/html[1]/body[1]/div[1]/p[2]/a[1]"
        }
      ],
      "rendered": false,
      "status_code": 200,
      "text": "Example Domain Example Domain This domain is for use in documentation examples without needing permission. Avoid use in operations. Learn more",
      "title": "Example Domain",
      "url": "https://example.com/",
      "warnings": []
    }
  ],
  "stats": {
    "ai_attempts": 0,
    "ai_completion_tokens": 0,
    "ai_mode": "never",
    "ai_prompt_tokens": 0,
    "ai_ranking_mode": "non_destructive",
    "ai_rejected_files": 0,
    "ai_relevant_files": 0,
    "ai_uncertain_files": 0,
    "ai_unselected_files": 0,
    "ai_used": false,
    "concurrency": 2,
    "crawled_pages": 1,
    "effective_delay_ms": 0,
    "elapsed_ms": 2203,
    "estimated_result_bytes": 1151,
    "failed_pages": 0,
    "files_before_compaction": 0,
    "files_found": 0,
    "files_omitted_by_compaction": 0,
    "files_returned": 0,
    "frontier_dropped": 0,
    "max_depth": 2,
    "remaining_frontier": 0,
    "response_compacted": false,
    "returned_pages": 1,
    "robots_applied": true
  },
  "url": "https://example.com/",
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `files` | `array` | 0 items |
| `final_url` | `string` | https://example.com/ |
| `pages` | `array` | 1 items |
| `pages` | `array` | 1 items |
| `stats` | `object` | 26 fields |
| `stats.ai_attempts` | `integer` | 0 |
| `stats.ai_completion_tokens` | `integer` | 0 |
| `stats.ai_mode` | `string` | never |
| `stats.ai_prompt_tokens` | `integer` | 0 |
| `stats.ai_ranking_mode` | `string` | non_destructive |
| `stats.ai_rejected_files` | `integer` | 0 |
| `stats.ai_relevant_files` | `integer` | 0 |
| `stats.ai_uncertain_files` | `integer` | 0 |
| `stats.ai_unselected_files` | `integer` | 0 |
| `stats.ai_used` | `boolean` | false |
| `stats.concurrency` | `integer` | 2 |
| `stats.crawled_pages` | `integer` | 1 |
| `stats.effective_delay_ms` | `integer` | 0 |
| `stats.elapsed_ms` | `integer` | 2203 |
| `stats.estimated_result_bytes` | `integer` | 1151 |
| `stats.failed_pages` | `integer` | 0 |
| `stats.files_before_compaction` | `integer` | 0 |
| `stats.files_found` | `integer` | 0 |
| `stats.files_omitted_by_compaction` | `integer` | 0 |
| `stats.files_returned` | `integer` | 0 |
| `stats.frontier_dropped` | `integer` | 0 |
| `stats.max_depth` | `integer` | 2 |
| `stats.remaining_frontier` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_pages` | `integer` | 1 |
| `stats.robots_applied` | `boolean` | true |
| `url` | `string` | https://example.com/ |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)

POST /execute Shared execution endpoint

Input parameters

accept_statuses array<integer> optional

ai string optional

Optional non-destructive relevance ranking for discovered documents. Every deterministic candidate remains in the result; auto leaves candidates unranked if AI is unavailable, while always requires ranking to succeed. Allowed values: `never`, `auto`, `always`. Default: `never`.

Available values neverautoalways

allow_browser_fallback boolean optional

Default: `false`.

allow_truncated boolean optional

Default: `false`.

concurrency integer optional

From 1 to 16. Default: `4`.

Example 2

continuation_token string optional

Opaque stateless frontier returned by a preceding crawl slice with the same options. A transiently failed URL is retained for at most one bounded retry; permanent failures are not amplified.

delay_ms integer optional

From 0 to 5000. Default: `0`.

detail string optional

Allowed values: `summary`, `standard`, `full`. Default: `standard`.

Example standard

Available values summarystandardfull

document object optional

document.chunk_chars integer optional

From 500 to 20000.

document.continuation_token string optional

document.ocr string optional

Allowed values: `auto`, `never`, `always`. Default: `auto`.

Available values autoneveralways

document.outputs array<string> optional

Requested document outputs. source returns the originally captured bytes: inline through 5.5 MB, otherwise through a hash-verified durable store, and fails if neither delivery path is available. When source is omitted, remote_source is provenance only and contains no bytes. Allowed values: `structured`, `text`, `markdown`, `source`.

Available values structuredtextmarkdownsource

document.page_end integer optional

From 1 to 1000000.

document.page_start integer optional

From 1 to 1000000.

exclude_patterns array<string> optional

include_patterns array<string> optional

include_subdomains boolean optional

When false, only the seed host is allowed. When true, hosts are still limited to the seed's registrable domain; unrelated hosts are never added. Default: `false`.

instructions string optional

Required when ai is auto or always. Describe which documents/files are relevant. Ranking is non-destructive: every deterministic candidate remains in the result.

max_body_bytes integer optional

From 1 to 104857600.

max_depth integer optional

From 1 to 10. Default: `2`.

Example 2

max_pages integer optional

Maximum number of result pages to fetch. From 1 to 100. Default: `10`.

Example 5

max_total_bytes integer optional

From 1 to 104857600.

respect_robots boolean optional

Default: `true`.

url string required

Public http(s) site URL to crawl.

Example https://example.com/

Response

Successful output is returned in results[0].data.

Raw platform output is intentionally open-ended and may evolve with the upstream source.

Illustrative sample The example response is shown in the response panel.

Illustrative output fields

Values illustrate one response. Read the guidance above for missing values and limits; fields can vary with the requested operation and the data available.

Path	Observed type	Example
`files`	`array`	0 items
`final_url`	`string`	https://example.com/
`pages`	`array`	1 items
`pages`	`array`	1 items
`stats`	`object`	26 fields
`stats.ai_attempts`	`integer`	0
`stats.ai_completion_tokens`	`integer`	0
`stats.ai_mode`	`string`	never
`stats.ai_prompt_tokens`	`integer`	0
`stats.ai_ranking_mode`	`string`	non_destructive
`stats.ai_rejected_files`	`integer`	0
`stats.ai_relevant_files`	`integer`	0
`stats.ai_uncertain_files`	`integer`	0
`stats.ai_unselected_files`	`integer`	0
`stats.ai_used`	`boolean`	false
`stats.concurrency`	`integer`	2
`stats.crawled_pages`	`integer`	1
`stats.effective_delay_ms`	`integer`	0
`stats.elapsed_ms`	`integer`	2203
`stats.estimated_result_bytes`	`integer`	1151
`stats.failed_pages`	`integer`	0
`stats.files_before_compaction`	`integer`	0
`stats.files_found`	`integer`	0
`stats.files_omitted_by_compaction`	`integer`	0
`stats.files_returned`	`integer`	0
`stats.frontier_dropped`	`integer`	0
`stats.max_depth`	`integer`	2
`stats.remaining_frontier`	`integer`	0
`stats.response_compacted`	`boolean`	false
`stats.returned_pages`	`integer`	1
`stats.robots_applied`	`boolean`	true
`url`	`string`	https://example.com/
`warnings`	`array`	0 items

Execution behavior

A 200 response completed inline. A 202 response was queued; poll GET /jobs/:id until the job reaches completed or failed.