Core API

Crawl & jobs

Crawl a controlled set of pages asynchronously and poll durable job state.

Start a crawl

curl -X POST https://app.crawly.cc/api/v1/crawl \
  -H "Authorization: Bearer $CRAWLY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":5,"maxDepth":2}'

Poll the job

The start request returns 202 and a job id. Poll until status is succeeded or failed.

curl https://app.crawly.cc/api/v1/jobs/JOB_ID \
  -H "Authorization: Bearer $CRAWLY_API_KEY"

Limits and billing

Crawl costs one credit per successful page. Plan limits cap pages and depth. Queue attribution preserves the account, API key, policy, and Usage entry.

Note: Do not continuously poll at high frequency. Use a short backoff or configure a job webhook.