Start crawl

POST https://v2-api.scrapegraphai.com/api/crawl

Starts an asynchronous crawl. The response returns a job id immediately; poll GET /api/crawl/:id, fetch page content with GET /api/crawl/:id/pages, or manage the job via the control endpoints.

Request body

string

required

Starting URL to crawl.

array

Output formats captured for each crawled page. Same shape as the Scrape formats array.

integer

Maximum number of pages to crawl.

integer

How many levels of links to follow from the starting URL.

integer

Cap on links expanded per page.

array

Glob-style URL patterns to include, e.g. ["/blog/*"].

array

Glob-style URL patterns to exclude, e.g. ["/admin/*"].

object

Fetch-time options applied to every page. See the Scrape endpoint.

Example request

curl -X POST https://v2-api.scrapegraphai.com/api/crawl \
  -H "SGAI-APIKEY: $SGAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapegraphai.com/",
    "formats": [{ "type": "markdown" }],
    "maxPages": 5,
    "maxDepth": 2,
    "includePatterns": ["/blog/*"],
    "excludePatterns": ["/admin/*"]
  }'

Example response

{
  "id": "79694e03-f2ea-43f2-93cc-7c6fc26f999a",
  "status": "running",
  "total": 3,
  "finished": 0,
  "pages": []
}

Field	Description
`id`	Crawl job identifier used on every follow-up endpoint.
`status`	Lifecycle state: `"running"`, `"completed"`, `"failed"`, or `"stopped"`.
`total`	Total pages the crawler expects to process so far.
`finished`	Pages completed.
`pages`	Per-page results (empty until the job makes progress).

Poll progress: GET /api/crawl/:id
Fetch pages: GET /api/crawl/:id/pages
Stop, resume, or delete: Manage crawl jobs
Service overview: Crawl

API Documentation

Scrape

Extract

Search

Crawl

Monitor

History

Account

Request body

Example request

Example response

​Request body

​Example request

​Example response

​Related

Request body

Example request

Example response

Related