Skip to main content
POST
/
v1
/
crawl
Start SmartCrawler
curl --request POST \
  --url https://api.example.com/v1/crawl

​
Start Crawl

POST /v1/crawl Start a new crawl job using SmartCrawler. Choose between AI-powered extraction or cost-effective markdown conversion.

​
Request Body

Content-Type: application/json

​
Schema

{
  "url": "string",
  "prompt": "string",
  "extraction_mode": "boolean",
  "cache_website": "boolean",
  "depth": "integer",
  "max_pages": "integer",
  "same_domain_only": "boolean",
  "batch_size": "integer",
  "schema": { /* JSON Schema object */ },
  "rules": {
    "exclude": ["string"],
    "include_paths": ["string"],
    "exclude_paths": ["string"],
    "same_domain": "boolean"
  },
  "sitemap": "boolean",
  "render_heavy_js": "boolean",
  "stealth": "boolean"
}

​
Parameters

ParameterTypeRequiredDefaultDescription
urlstringYes-The starting URL for the crawl
promptstringNo*-Instructions for data extraction (*required when extraction_mode=true)
extraction_modebooleanNotrueWhen false, enables markdown conversion mode (NO AI/LLM processing, 2 credits per page)
cache_websitebooleanNofalseWhether to cache the website content
depthintegerNo1Maximum crawl depth
max_pagesintegerNo10Maximum number of pages to crawl
same_domain_onlybooleanNotrueWhether to crawl only the same domain
batch_sizeintegerNo1Number of pages to process in each batch
schemaobjectNo-JSON Schema object for structured output
rulesobjectNo-Crawl rules for filtering URLs. Object with optional fields: exclude (array of regex URL patterns), include_paths (array of path patterns to include, supports wildcards * and **), exclude_paths (array of path patterns to exclude, takes precedence over include_paths), same_domain (boolean, default: true). See Rules section below for details.
sitemapbooleanNofalseUse sitemap.xml for discovery
render_heavy_jsbooleanNofalseEnable heavy JavaScript rendering
stealthbooleanNofalseEnable stealth mode to bypass bot protection using advanced anti-detection techniques. Adds +4 credits to the request cost

​
Example

{
  "url": "https://scrapegraphai.com/",
  "prompt": "What does the company do? and I need text content from there privacy and terms",
  "cache_website": true,
  "depth": 2,
  "max_pages": 2,
  "same_domain_only": true,
  "batch_size": 1,
  "stealth": true,
  "rules": {
    "include_paths": ["/privacy", "/terms", "/about"],
    "exclude_paths": ["/admin/*", "/api/**"],
    "same_domain": true
  },
  "schema": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "ScrapeGraphAI Website Content",
    "type": "object",
    "properties": {
      "company": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "description": { "type": "string" },
          "features": {
            "type": "array",
            "items": { "type": "string" }
          },
          "contact_email": { "type": "string", "format": "email" },
          "social_links": {
            "type": "object",
            "properties": {
              "github": { "type": "string", "format": "uri" },
              "linkedin": { "type": "string", "format": "uri" },
              "twitter": { "type": "string", "format": "uri" }
            },
            "additionalProperties": false
          }
        },
        "required": ["name", "description"]
      },
      "services": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "service_name": { "type": "string" },
            "description": { "type": "string" },
            "features": {
              "type": "array",
              "items": { "type": "string" }
            }
          },
          "required": ["service_name", "description"]
        }
      },
      "legal": {
        "type": "object",
        "properties": {
          "privacy_policy": { "type": "string" },
          "terms_of_service": { "type": "string" }
        },
        "required": ["privacy_policy", "terms_of_service"]
      }
    },
    "required": ["company", "services", "legal"]
  }
}

​
Markdown Conversion Example (No AI/LLM)

For cost-effective HTML to markdown conversion without AI processing: 
{
  "url": "https://scrapegraphai.com/",
  "extraction_mode": false,
  "depth": 2,
  "max_pages": 5,
  "same_domain_only": true,
  "stealth": true,
  "rules": {
    "include_paths": ["/docs/*", "/blog/**"],
    "exclude_paths": ["/admin/*", "/api/**"],
    "same_domain": true
  }
}

​
Rules Example

Control which URLs to crawl using the rules object: 
{
  "url": "https://example.com/",
  "prompt": "Extract product information",
  "depth": 3,
  "max_pages": 50,
  "rules": {
    "exclude": ["https://example.com/logout", "https://example.com/admin/*"],
    "include_paths": ["/products/*", "/blog/**"],
    "exclude_paths": ["/admin/*", "/api/**", "/private/*"],
    "same_domain": true
  }
}

​
Rules Parameters

FieldTypeDefaultDescription
excludearray[]List of URL patterns (regex) to exclude from crawling. Matches full URL.
include_pathsarray[](Optional) List of path patterns to include (e.g., ["/products/*", "/blog/**"]). Supports wildcards: * matches any characters, ** matches any path segments. If empty or not specified, all paths are included.
exclude_pathsarray[](Optional) List of path patterns to exclude (e.g., ["/admin/*", "/api/**"]). Supports wildcards: * matches any characters, ** matches any path segments. Takes precedence over include_paths. If empty or not specified, no paths are excluded.
same_domainbooleantrueRestrict crawling to same domain
Both include_paths and exclude_paths are optional. If include_paths is not specified or empty, all paths are included. If exclude_paths is not specified or empty, no paths are excluded. The exclude_paths patterns take precedence over include_paths patterns.
When extraction_mode: false, the prompt parameter is not required. This mode converts HTML to clean markdown with metadata extraction at only 2 credits per page (80% savings compared to AI mode).

​
Response

  • 200 OK: Crawl started successfully. Returns { "task_id": "<task_id>" }. Use this task_id to retrieve the crawl result from the Get Crawl Result endpoint.
  • 422 Unprocessable Entity: Validation error.
See the Get Crawl Result endpoint for the full response structure.