Universal Web API Capability

Map site

Build a deterministic, robots-aware public URL inventory from the starting page, robots.txt, and bounded recursive sitemap indexes. Classifies pages and files without deeply fetching every discovered URL and supports opaque pagination cursors.

Capability ID web.site.map

Cost From $1.25 per 1,000 requests · 1 credit

Maximum runtime 120 seconds

View raw

# Map site

Build a deterministic, robots-aware public URL inventory from the starting page, robots.txt, and bounded recursive sitemap indexes. Classifies pages and files without deeply fetching every discovered URL and supports opaque pagination cursors.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.map`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer $UPSCRAPE_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "discover_sitemaps": true,
    "max_urls": 100,
    "respect_robots": true,
    "url": "https://example.com/"
  },
  "capability": "web.site.map"
}'
```

## Input


| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `cursor` | `string` | No | Opaque cursor from the preceding map response using the same options. |
| `discover_sitemaps` | `boolean` | No | Default: `true`. |
| `exclude_patterns` | `array<string>` | No |  |
| `include_patterns` | `array<string>` | No |  |
| `include_subdomains` | `boolean` | No | When false, only the seed host is allowed. When true, hosts remain limited to the seed's registrable domain; unrelated hosts are never added. Default: `false`. |
| `max_sitemaps` | `integer` | No | From 1 to 500. Default: `50`. |
| `max_urls` | `integer` | No | Maximum URLs returned in this response page. From 1 to 5000. Default: `1000`. |
| `respect_robots` | `boolean` | No | Default: `true`. |
| `url` | `string` | Yes | Public http(s) site URL to inventory. |

### Example input

```json
{
  "discover_sitemaps": true,
  "max_urls": 100,
  "respect_robots": true,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`.

Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This example shows one response. Values and optional fields vary with the data available.

```json
{
  "final_url": "https://example.com/",
  "origin": "https://example.com",
  "robots": {
    "applied": true,
    "fetched": false,
    "sitemaps": [],
    "status_code": 404,
    "url": "https://example.com/robots.txt",
    "warnings": []
  },
  "sitemaps": [
    {
      "bytes": 559,
      "content_type": "text/html",
      "error": "The requested data could not be retrieved. Try again later.",
      "sha256": "[redacted:token]",
      "status_code": 404,
      "url": "https://example.com/sitemap.xml"
    }
  ],
  "stats": {
    "elapsed_ms": 1484,
    "estimated_result_bytes": 657,
    "inventory_count": 1,
    "offset": 0,
    "response_compacted": false,
    "returned_count": 1,
    "sitemap_count": 1
  },
  "url": "https://example.com/",
  "urls": [
    {
      "allowed": true,
      "format": "unknown",
      "kind": "page",
      "source": "seed",
      "url": "https://example.com/"
    }
  ],
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `final_url` | `string` | https://example.com/ |
| `origin` | `string` | https://example.com |
| `robots` | `object` | 6 fields |
| `robots.applied` | `boolean` | true |
| `robots.fetched` | `boolean` | false |
| `robots.sitemaps` | `array` | 0 items |
| `robots.status_code` | `integer` | 404 |
| `robots.url` | `string` | https://example.com/robots.txt |
| `robots.warnings` | `array` | 0 items |
| `sitemaps` | `array` | 1 items |
| `sitemaps` | `array` | 1 items |
| `stats` | `object` | 7 fields |
| `stats.elapsed_ms` | `integer` | 1484 |
| `stats.estimated_result_bytes` | `integer` | 657 |
| `stats.inventory_count` | `integer` | 1 |
| `stats.offset` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_count` | `integer` | 1 |
| `stats.sitemap_count` | `integer` | 1 |
| `url` | `string` | https://example.com/ |
| `urls` | `array` | 1 items |
| `urls` | `array` | 1 items |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)

POST /execute Shared execution endpoint

Input parameters

cursor string optional

Opaque cursor from the preceding map response using the same options.

discover_sitemaps boolean optional

Default: `true`.

Example true

exclude_patterns array<string> optional

include_patterns array<string> optional

include_subdomains boolean optional

When false, only the seed host is allowed. When true, hosts remain limited to the seed's registrable domain; unrelated hosts are never added. Default: `false`.

max_sitemaps integer optional

From 1 to 500. Default: `50`.

max_urls integer optional

Maximum URLs returned in this response page. From 1 to 5000. Default: `1000`.

Example 100

respect_robots boolean optional

Default: `true`.

Example true

url string required

Public http(s) site URL to inventory.

Example https://example.com/

Response

Successful output is returned in results[0].data.

Raw platform output is intentionally open-ended and may evolve with the upstream source.

Illustrative sample The example response is shown in the response panel.

Illustrative output fields

Values illustrate one response. Read the guidance above for missing values and limits; fields can vary with the requested operation and the data available.

Path	Observed type	Example
`final_url`	`string`	https://example.com/
`origin`	`string`	https://example.com
`robots`	`object`	6 fields
`robots.applied`	`boolean`	true
`robots.fetched`	`boolean`	false
`robots.sitemaps`	`array`	0 items
`robots.status_code`	`integer`	404
`robots.url`	`string`	https://example.com/robots.txt
`robots.warnings`	`array`	0 items
`sitemaps`	`array`	1 items
`sitemaps`	`array`	1 items
`stats`	`object`	7 fields
`stats.elapsed_ms`	`integer`	1484
`stats.estimated_result_bytes`	`integer`	657
`stats.inventory_count`	`integer`	1
`stats.offset`	`integer`	0
`stats.response_compacted`	`boolean`	false
`stats.returned_count`	`integer`	1
`stats.sitemap_count`	`integer`	1
`url`	`string`	https://example.com/
`urls`	`array`	1 items
`urls`	`array`	1 items
`warnings`	`array`	0 items

Execution behavior

A 200 response completed inline. A 202 response was queued; poll GET /jobs/:id until the job reaches completed or failed.

curl --request POST \ --url https://data.upscrape.com/execute \ --header "Authorization: Bearer $UPSCRAPE_API_KEY" \ --header "Content-Type: application/json" \ --header "Prefer: wait=30" \ --data '{ "input": { "discover_sitemaps": true, "max_urls": 100, "respect_robots": true, "url": "https://example.com/" }, "capability": "web.site.map" }'

import json import os import requests payload = json.loads(r'''{ "input": { "discover_sitemaps": true, "max_urls": 100, "respect_robots": true, "url": "https://example.com/" }, "capability": "web.site.map" }''') response = requests.post( "https://data.upscrape.com/execute", headers={ "Authorization": f"Bearer {os.environ['UPSCRAPE_API_KEY']}", "Content-Type": "application/json", "Prefer": "wait=30", }, json=payload, ) response.raise_for_status() print(json.dumps(response.json(), indent=2))

{ "final_url": "https://example.com/", "origin": "https://example.com", "robots": { "applied": true, "fetched": false, "sitemaps": [], "status_code": 404, "url": "https://example.com/robots.txt", "warnings": [] }, "sitemaps": [ { "bytes": 559, "content_type": "text/html", "error": "The requested data could not be retrieved. Try again later.", "sha256": "[redacted:token]", "status_code": 404, "url": "https://example.com/sitemap.xml" } ], "stats": { "elapsed_ms": 1484, "estimated_result_bytes": 657, "inventory_count": 1, "offset": 0, "response_compacted": false, "returned_count": 1, "sitemap_count": 1 }, "url": "https://example.com/", "urls": [ { "allowed": true, "format": "unknown", "kind": "page", "source": "seed", "url": "https://example.com/" } ], "warnings": [] }