Skip to content
QuantCDN

Update a crawler

PATCH
/api/v2/organizations/{organization}/projects/{project}/crawlers/{crawler}
curl --request PATCH \
  --url https://dashboard.quantcdn.io/api/v2/organizations/test-org/projects/test-project/crawlers/00000000-0000-0000-0000-000000000000 \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "name": "Test Crawler", "domain": "test-domain.com", "browser_mode": false, "urls": [ "/", "/about", "/contact" ], "start_urls": [ "/", "/blog" ], "headers": { "Authorization": "Bearer token123", "X-Custom-Header": "value" }, "exclude": [ "/admin/*", "/private/*" ], "include": [ "/blog/*", "/products/*" ], "webhook_url": "https://example.com/webhook", "webhook_auth_header": "Bearer token123", "webhook_extra_vars": "key1=value1&key2=value2", "workers": 2, "delay": 4, "depth": -1, "max_hits": 0, "max_html": 50, "status_ok": [ 200, 201 ], "sitemap": [ { "url": "/sitemap.xml", "recursive": true } ], "allowed_domains": [ "example.com", "assets.example.com" ], "user_agent": "Mozilla/5.0...", "assets": { "network_intercept": { "enabled": true, "timeout": 30, "execute_js": false }, "parser": { "enabled": true } }, "max_errors": 100 }'

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
organization
required
string

Organization identifier

Example
test-org
project
required
string

Project identifier

Example
test-project
crawler
required
string

The UUID of the crawler

Example
00000000-0000-0000-0000-000000000000

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json
object
name

Crawler name

string
Example
Test Crawler
domain
required

Domain to crawl

string
Example
test-domain.com
browser_mode

Enable browser mode

boolean
Example
false
urls

URLs to crawl

Array<string>
Example
[
  "/",
  "/about",
  "/contact"
]
start_urls

Starting URLs for crawl

Array<string>
Example
[
  "/",
  "/blog"
]
headers

Custom headers

object
key
additional properties
string
Example
{
  "Authorization": "Bearer token123",
  "X-Custom-Header": "value"
}
exclude

URL patterns to exclude (regex)

Array<string>
Example
[
  "/admin/*",
  "/private/*"
]
include

URL patterns to include (regex)

Array<string>
Example
[
  "/blog/*",
  "/products/*"
]
webhook_url

Webhook URL for notifications

string
Example
https://example.com/webhook
webhook_auth_header

Authorization header for webhook

string
Example
Bearer token123
webhook_extra_vars

Extra variables for webhook

string
Example
key1=value1&key2=value2
workers

Number of concurrent workers (default: 2, non-default requires verification)

integer
>= 1 <= 20
Example
2
delay

Delay between requests in seconds (default: 4, non-default requires verification)

number format: float
<= 10
Example
4
depth

Maximum crawl depth, -1 for unlimited

integer
>= -1
Example
-1
max_hits

Maximum total requests, 0 for unlimited (default: 0, non-default requires verification)

integer
Example
0
max_html

Maximum HTML pages, 0 for unlimited (default: org limit, non-default requires verification)

integer
Example
50
status_ok

HTTP status codes that will result in content being captured and pushed to Quant

Array<integer>
Example
[
  200,
  201
]
sitemap

Sitemap configuration

Array<object>
object
url

Sitemap URL

string
Example
/sitemap.xml
recursive

Recursively follow sitemap links

boolean
Example
true
Example
[
  {
    "url": "/sitemap.xml",
    "recursive": true
  }
]
allowed_domains

Allowed domains for multi-domain crawling, automatically enables merge_domains

Array<string>
Example
[
  "example.com",
  "assets.example.com"
]
user_agent

Custom user agent, only when browser_mode is false

string
Example
Mozilla/5.0...
assets

Asset harvesting configuration

object
network_intercept

Network intercept configuration for asset collection

object
enabled

Enable network intercept

boolean
Example
true
timeout

Request timeout in seconds

integer
Example
30
execute_js

Execute JavaScript during asset collection

boolean
Example
false
parser

Parser configuration for asset extraction

object
enabled

Enable parser

boolean
Example
true
Example
{
  "network_intercept": {
    "enabled": true,
    "timeout": 30,
    "execute_js": false
  },
  "parser": {
    "enabled": true
  }
}
max_errors

Maximum errors before stopping crawl

integer
Example
100

Responses

Section titled “Responses”

200

Section titled “200”

The request has succeeded.

Media typeapplication/json
object
id
required

Crawler ID

integer
name

Crawler name

string
project_id
required

Project ID

integer
uuid
required

Crawler UUID

string
config
required

Crawler configuration (YAML)

string
domain
required

Crawler domain

string
domain_verified

Domain verification status

integer
0
urls_list

URLs list (YAML)

string
webhook_url

Webhook URL for notifications

string
webhook_auth_header

Authorization header for webhook

string
webhook_extra_vars

Extra variables for webhook

string
browser_mode

Browser mode enabled

boolean
workers

Number of concurrent workers

integer
delay

Delay between requests in seconds

number format: float
depth

Maximum crawl depth

integer
max_hits

Maximum total requests

integer
max_html

Maximum HTML pages

integer
status_ok

HTTP status codes for content capture

Array<integer>
user_agent

Custom user agent

string
max_errors

Maximum errors before stopping

integer
start_urls

Starting URLs

Array<string>
urls

URLs list

Array<string>
headers

Custom headers

object
key
additional properties
string
exclude

URL patterns to exclude

Array<string>
include

URL patterns to include

Array<string>
sitemap

Sitemap configuration

Array<object>
object
url

Sitemap URL

string
recursive

Recursively follow sitemap links

boolean
allowed_domains

Allowed domains

Array<string>
assets

Asset harvesting configuration

object
network_intercept

Network intercept configuration for asset collection

object
enabled

Enable network intercept

boolean
timeout

Request timeout in seconds

integer
execute_js

Execute JavaScript during asset collection

boolean
parser

Parser configuration for asset extraction

object
enabled

Enable parser

boolean
created_at

Creation timestamp

string format: date-time
updated_at

Last update timestamp

string format: date-time
deleted_at

Deletion timestamp

string format: date-time
nullable
Example
{
  "id": 456,
  "name": "Test Crawler",
  "project_id": 789,
  "uuid": "550e8400-e29b-41d4-a716-446655440000",
  "config": "domain: test-domain.com\\nconfig:\\n  max_html: 100\\n  browser_mode: false",
  "domain": "test-domain.com",
  "domain_verified": 0,
  "urls_list": "single_url:\\n  - /\\n  - /about\\n  - /contact",
  "webhook_url": "https://example.com/webhook",
  "webhook_auth_header": "Bearer token123",
  "webhook_extra_vars": "key1=value1&key2=value2",
  "browser_mode": false,
  "workers": 2,
  "delay": 4,
  "depth": -1,
  "max_hits": 0,
  "max_html": 50,
  "status_ok": [
    200
  ],
  "user_agent": "Mozilla/5.0...",
  "max_errors": 100,
  "start_urls": [
    "/",
    "/blog"
  ],
  "urls": [
    "/",
    "/about"
  ],
  "headers": {
    "Authorization": "Bearer token"
  },
  "exclude": [
    "/admin/*"
  ],
  "include": [
    "/blog/*"
  ],
  "sitemap": [
    {
      "url": "/sitemap.xml",
      "recursive": true
    }
  ],
  "allowed_domains": [
    "example.com"
  ],
  "assets": {
    "network_intercept": {
      "enabled": true,
      "timeout": 30,
      "execute_js": false
    },
    "parser": {
      "enabled": true
    }
  },
  "created_at": "2024-01-20T09:15:00Z",
  "updated_at": "2024-10-11T16:45:00Z"
}

400

Section titled “400”

The server could not understand the request due to invalid syntax.

Media typeapplication/json
object
message
required

Error message

string
error
required

Error flag

boolean
Example
{
  "message": "The requested resource was not found",
  "error": true
}

403

Section titled “403”

Access is forbidden.

Media typeapplication/json
object
message
required

Error message

string
error
required

Error flag

boolean
Example
{
  "message": "The requested resource was not found",
  "error": true
}