Get details of a single crawler

GET

/api/v2/organizations/{organization}/projects/{project}/crawlers/{crawler}

const url = 'https://dashboard.quantcdn.io/api/v2/organizations/test-org/projects/test-project/crawlers/00000000-0000-0000-0000-000000000000';
const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request GET \
  --url https://dashboard.quantcdn.io/api/v2/organizations/test-org/projects/test-project/crawlers/00000000-0000-0000-0000-000000000000 \
  --header 'Authorization: Bearer <token>'

Authorizations

BearerAuth

Parameters

Path Parameters

organization

required

string

Organization identifier

Example

test-org

project

required

string

Project identifier

Example

test-project

crawler

required

string

The UUID of the crawler

Example

00000000-0000-0000-0000-000000000000

Responses

200

The request has succeeded.

application/json

object

required

Crawler ID

integer

name

Crawler name

string

project_id

required

Project ID

integer

uuid

required

Crawler UUID

string

config

required

Crawler configuration (YAML)

string

domain

required

Crawler domain

string

domain_verified

Domain verification status

integer

urls_list

URLs list (YAML)

string

webhook_url

Webhook URL for notifications

string

webhook_auth_header

Authorization header for webhook

string

webhook_extra_vars

Extra variables for webhook

string

browser_mode

Browser mode enabled

boolean

workers

Number of concurrent workers

integer

delay

Delay between requests in seconds

number format: float

depth

Maximum crawl depth

integer

max_hits

Maximum total requests

integer

max_html

Maximum HTML pages

integer

status_ok

HTTP status codes for content capture

Array<integer>

user_agent

Custom user agent

string

max_errors

Maximum errors before stopping

integer

start_urls

Starting URLs

Array<string>

urls

URLs list

Array<string>

headers

Custom headers

object

key

additional properties

string

exclude

URL patterns to exclude

Array<string>

include

URL patterns to include

Array<string>

sitemap

Sitemap configuration

Array<object>

object

url

Sitemap URL

string

recursive

Recursively follow sitemap links

boolean

allowed_domains

Allowed domains

Array<string>

assets

Asset harvesting configuration

object

network_intercept

Network intercept configuration for asset collection

object

enabled

Enable network intercept

boolean

timeout

Request timeout in seconds

integer

execute_js

Execute JavaScript during asset collection

boolean

parser

Parser configuration for asset extraction

object

enabled

Enable parser

boolean

created_at

Creation timestamp

string format: date-time

updated_at

Last update timestamp

string format: date-time

deleted_at

Deletion timestamp

string format: date-time

nullable

Example

{
  "id": 456,
  "name": "Test Crawler",
  "project_id": 789,
  "uuid": "550e8400-e29b-41d4-a716-446655440000",
  "config": "domain: test-domain.com\\nconfig:\\n  max_html: 100\\n  browser_mode: false",
  "domain": "test-domain.com",
  "domain_verified": 0,
  "urls_list": "single_url:\\n  - /\\n  - /about\\n  - /contact",
  "webhook_url": "https://example.com/webhook",
  "webhook_auth_header": "Bearer token123",
  "webhook_extra_vars": "key1=value1&key2=value2",
  "browser_mode": false,
  "workers": 2,
  "delay": 4,
  "depth": -1,
  "max_hits": 0,
  "max_html": 50,
  "status_ok": [
    200
  ],
  "user_agent": "Mozilla/5.0...",
  "max_errors": 100,
  "start_urls": [
    "/",
    "/blog"
  ],
  "urls": [
    "/",
    "/about"
  ],
  "headers": {
    "Authorization": "Bearer token"
  },
  "exclude": [
    "/admin/*"
  ],
  "include": [
    "/blog/*"
  ],
  "sitemap": [
    {
      "url": "/sitemap.xml",
      "recursive": true
    }
  ],
  "allowed_domains": [
    "example.com"
  ],
  "assets": {
    "network_intercept": {
      "enabled": true,
      "timeout": 30,
      "execute_js": false
    },
    "parser": {
      "enabled": true
    }
  },
  "created_at": "2024-01-20T09:15:00Z",
  "updated_at": "2024-10-11T16:45:00Z"
}

400

The server could not understand the request due to invalid syntax.

application/json

object

message

required

Error message

string

error

required

Error flag

boolean

Example

{
  "message": "The requested resource was not found",
  "error": true
}

403

Access is forbidden.

application/json

object

message

required

Error message

string

error

required

Error flag

boolean

Example

{
  "message": "The requested resource was not found",
  "error": true
}