Start Crawl
POST /v1/crawl
Start a new crawl job using SmartCrawler. Choose between AI-powered extraction or cost-effective markdown conversion.
Request Body
Content-Type: application/json
Schema
{
"url": "string",
"prompt": "string",
"extraction_mode": "boolean",
"cache_website": "boolean",
"depth": "integer",
"max_pages": "integer",
"same_domain_only": "boolean",
"batch_size": "integer",
"schema": { /* JSON Schema object */ }
}
Parameters
| Parameter | Type | Required | Default | Description |
|---|
| url | string | Yes | - | The starting URL for the crawl |
| prompt | string | No* | - | Instructions for data extraction (*required when extraction_mode=true) |
| extraction_mode | boolean | No | true | When false, enables markdown conversion mode (NO AI/LLM processing, 2 credits per page) |
| cache_website | boolean | No | false | Whether to cache the website content |
| depth | integer | No | 1 | Maximum crawl depth |
| max_pages | integer | No | 10 | Maximum number of pages to crawl |
| same_domain_only | boolean | No | true | Whether to crawl only the same domain |
| batch_size | integer | No | 1 | Number of pages to process in each batch |
| schema | object | No | - | JSON Schema object for structured output |
| stealth | boolean | No | false | Enable stealth mode to bypass bot protection using advanced anti-detection techniques. Adds +4 credits to the request cost |
Example
{
"url": "https://scrapegraphai.com/",
"prompt": "What does the company do? and I need text content from there privacy and terms",
"cache_website": true,
"depth": 2,
"max_pages": 2,
"same_domain_only": true,
"batch_size": 1,
"stealth": true,
"schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "ScrapeGraphAI Website Content",
"type": "object",
"properties": {
"company": {
"type": "object",
"properties": {
"name": { "type": "string" },
"description": { "type": "string" },
"features": {
"type": "array",
"items": { "type": "string" }
},
"contact_email": { "type": "string", "format": "email" },
"social_links": {
"type": "object",
"properties": {
"github": { "type": "string", "format": "uri" },
"linkedin": { "type": "string", "format": "uri" },
"twitter": { "type": "string", "format": "uri" }
},
"additionalProperties": false
}
},
"required": ["name", "description"]
},
"services": {
"type": "array",
"items": {
"type": "object",
"properties": {
"service_name": { "type": "string" },
"description": { "type": "string" },
"features": {
"type": "array",
"items": { "type": "string" }
}
},
"required": ["service_name", "description"]
}
},
"legal": {
"type": "object",
"properties": {
"privacy_policy": { "type": "string" },
"terms_of_service": { "type": "string" }
},
"required": ["privacy_policy", "terms_of_service"]
}
},
"required": ["company", "services", "legal"]
}
}
Markdown Conversion Example (No AI/LLM)
For cost-effective HTML to markdown conversion without AI processing:
{
"url": "https://scrapegraphai.com/",
"extraction_mode": false,
"depth": 2,
"max_pages": 5,
"same_domain_only": true,
"stealth": true
}
When extraction_mode: false, the prompt parameter is not required. This mode converts HTML to clean markdown with metadata extraction at only 2 credits per page (80% savings compared to AI mode).
Response
- 200 OK: Crawl started successfully. Returns
{ "task_id": "<task_id>" }. Use this task_id to retrieve the crawl result from the Get Crawl Result endpoint.
- 422 Unprocessable Entity: Validation error.
See the Get Crawl Result endpoint for the full response structure.
