SmartCrawler - ScrapeGraphAI

Legacy v1 service. Use Crawl instead — it is the canonical v2 API. The equivalent MCP tools are crawl_start, crawl_get_status, crawl_stop, and crawl_resume. This page is kept for reference.

Overview

SmartCrawler is our advanced web crawling service that offers two modes:

AI-Powered Extraction: LLM-powered web crawling with intelligent data extraction (10 credits per page)
Markdown Conversion: Cost-effective HTML to markdown conversion without AI/LLM processing (2 credits per page - 80% savings!)

Unlike SmartScraper, which extracts data from a single page, SmartCrawler can traverse multiple pages, follow links, and either extract structured data or convert content to clean markdown from entire websites or sections.

Try SmartCrawler instantly in our interactive playground

Getting Started

Quick Start

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.crawl(
    url="https://scrapegraphai.com/",
    prompt="Extract info about the company",
    depth=2,
    breadth=None,
    max_pages=10,
    rules={
        "include_paths": ["/about", "/services/*"],
        "exclude_paths": ["/admin/*", "/api/**"],
        "same_domain": True
    }
)

import { crawl } from 'scrapegraph-js';

const apiKey = 'your-api-key';

const response = await crawl(apiKey, {
  url: 'https://scrapegraphai.com/',
  prompt: 'Extract info about the company',
  depth: 2,
  max_pages: 10,
  rules: {
    include_paths: ['/about', '/services/*'],
    exclude_paths: ['/admin/*', '/api/**'],
    same_domain: true
  }
});

if (response.status === 'success') {
  console.log(response.data);
}

curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/crawl' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://scrapegraphai.com/",
  "prompt": "Extract info about the company",
  "depth": 2,
  "breadth": null,
  "max_pages": 10,
  "rules": {
    "include_paths": ["/about", "/services/*"],
    "exclude_paths": ["/admin/*", "/api/**"],
    "same_domain": true
  }
}'

just-scrape crawl https://scrapegraphai.com/ -p "Extract info about the company" \
  --depth 2 --max-pages 10 \
  --rules '{"include_paths":["/about","/services/*"],"exclude_paths":["/admin/*","/api/**"],"same_domain":true}'

Required Headers

Header	Description
SGAI-APIKEY	Your API authentication key
Content-Type	application/json

Parameters

Parameter	Type	Required	Description
url	string	Yes	The starting URL for the crawl.
prompt	string	No*	Instructions for what to extract (*required when extraction_mode=true).
extraction_mode	bool	No	When `false`, enables markdown conversion mode (default: true).
depth	int	No	How many link levels to follow (default: 1).
breadth	int	No	Maximum number of links to crawl per depth level. If null/undefined, unlimited (default). Controls the ‘width’ of exploration at each depth. Useful for limiting crawl scope on large sites. Note: max_pages always takes priority. Ignored when sitemap=true.
max_pages	int	No	Maximum number of pages to crawl (default: 20).
schema	object	No	Pydantic or Zod schema for structured output.
rules	object	No	Crawl rules object with optional fields: `exclude` (array of regex URL patterns), `include_paths` (array of path patterns to include, supports wildcards `` and `*`), `exclude_paths` (array of path patterns to exclude, takes precedence over `include_paths`), `same_domain` (boolean, default: true). See below for details.
sitemap	bool	No	Use sitemap.xml for discovery (default: false).
webhook_url	string	No	URL to receive webhook notification on job completion.
wait_ms	int	No	Milliseconds to wait before scraping each page. Useful for pages with heavy JavaScript rendering that need extra time to load (default: 3000).

Get your API key from the dashboard

Markdown Conversion Mode

For cost-effective content archival and when you only need clean markdown without AI processing, use the markdown conversion mode. This mode offers significant cost savings and is perfect for documentation, content migration, and simple data collection.

Benefits

80% Cost Savings: Only 2 credits per page vs 10 credits for AI mode
No AI/LLM Processing: Pure HTML to markdown conversion
Clean Output: Well-formatted markdown with metadata extraction
Fast Processing: No AI inference delays
Perfect for: Documentation, content archival, site migration

Quick Start - Markdown Mode

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

# Markdown conversion mode - no prompt needed
response = client.crawl(
    url="https://scrapegraphai.com/",
    extraction_mode=False,  # False = Markdown conversion (NO AI/LLM)
    depth=2,
    breadth=None,
    max_pages=5,
    rules={
        "include_paths": ["/docs/*", "/blog/**"],
        "exclude_paths": ["/admin/*"],
        "same_domain": True
    }
)

import { crawl } from 'scrapegraph-js';

const apiKey = 'your-api-key';

const response = await crawl(apiKey, {
  url: 'https://scrapegraphai.com/',
  extraction_mode: false,
  depth: 2,
  max_pages: 5,
  rules: {
    include_paths: ['/docs/*', '/blog/**'],
    exclude_paths: ['/admin/*'],
    same_domain: true
  }
});

if (response.status === 'success') {
  console.log(response.data);
}

curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/crawl' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://scrapegraphai.com/",
  "extraction_mode": false,
  "depth": 2,
  "breadth": null,
  "max_pages": 5,
  "rules": {
    "include_paths": ["/docs/*", "/blog/**"],
    "exclude_paths": ["/admin/*"],
    "same_domain": true
  }
}'

just-scrape crawl https://scrapegraphai.com/ --no-extraction \
  --depth 2 --max-pages 5 \
  --rules '{"include_paths":["/docs/*","/blog/**"],"exclude_paths":["/admin/*"],"same_domain":true}'

Markdown Mode Response

Markdown Conversion Response

{
  "status": "success",
  "result": {
    "status": "done",
    "pages_processed": 5,
    "credits_used": 10,
    "crawled_urls": [
      "https://scrapegraphai.com/",
      "https://scrapegraphai.com/about",
      "https://scrapegraphai.com/pricing"
    ],
    "pages": [
      {
        "url": "https://scrapegraphai.com/",
        "title": "ScrapeGraphAI - AI-Powered Web Scraping",
        "markdown": "# Transform Websites into Structured Data\n\nScrapeGraphAI is the most complete web scraping library...",
        "metadata": {
          "word_count": 1250,
          "headers": ["Transform Websites", "Features", "Pricing"],
          "links_count": 25
        }
      }
    ]
  }
}

Crawl Rules

You can control the crawl behavior with the rules object:

Python

rules = {
    "exclude": ["https://example.com/logout", "https://example.com/admin/*"],  # Regex patterns for full URLs to exclude
    "include_paths": ["/products/*", "/blog/**"],  # Path patterns to include
    "exclude_paths": ["/admin/*", "/api/**", "/private/*"],  # Path patterns to exclude (takes precedence)
    "same_domain": True  # Only crawl links on the same domain
}

Field	Type	Default	Description
exclude	list	[]	List of URL patterns (regex) to exclude from crawling. Matches full URL.
include_paths	list	[]	(Optional) List of path patterns to include (e.g., `["/products/", "/blog/"]`). Supports wildcards: `` matches any characters, `**` matches any path segments. If empty or not specified, all paths are included.
exclude_paths	list	[]	(Optional) List of path patterns to exclude (e.g., `["/admin/", "/api/"]`). Supports wildcards: `` matches any characters, `**` matches any path segments. Takes precedence over `include_paths`. If empty or not specified, no paths are excluded.
same_domain	bool	True	Restrict crawl to the same domain

Both include_paths and exclude_paths are optional. If include_paths is not specified or empty, all paths are included. If exclude_paths is not specified or empty, no paths are excluded. The exclude_paths patterns take precedence over include_paths patterns.

Example Response

{
  "status": "success",
  "result": {
    "status": "done",
    "llm_result": {
      "company": {
        "name": "ScrapeGraphAI, Inc",
        "description": "ScrapeGraphAI is a company that provides web scraping services using artificial intelligence...",
        "features": ["AI Agent Ready", "Universal Data Extraction", ...],
        "contact_email": "contact@scrapegraphai.com",
        "social_links": {
          "github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai",
          "linkedin": "https://www.linkedin.com/company/101881123",
          "twitter": "https://x.com/scrapegraphai"
        }
      },
      "services": [
        {"service_name": "Markdownify", ...},
        {"service_name": "Smart Scraper", ...}
      ],
      "legal": {
        "privacy_policy": "https://scrapegraphai.com/privacy",
        "terms_of_service": "https://scrapegraphai.com/terms"
      }
    },
    "crawled_urls": [
      "https://scrapegraphai.com/", ...
    ],
    "pages": [
      {
        "url": "https://scrapegraphai.com/",
        "markdown": "# Transform Websites into Structured Data\n..."
      },
      ...
    ]
  }
}

llm_result: Structured extraction based on your prompt/schema
crawled_urls: List of all URLs visited
pages: List of objects with url and extracted markdown content

Retrieve a Previous Crawl

You can retrieve the result of a crawl job by its task ID:

result = client.get_crawl_result(task_id="your-task-id")

import { history } from 'scrapegraph-js';

const apiKey = 'your_api_key';

const response = await history(apiKey, {
  service: 'crawl',
  page: 1,
  page_size: 10,
});

if (response.status === 'success') {
  console.log(response.data.requests);
}

Parameters

Parameter	Type	Required	Description
apiKey	string	Yes	The ScrapeGraph API Key.
taskId	string	Yes	The crawl job task ID.

Custom Schema Example

Define exactly what data you want to extract from every page:

from pydantic import BaseModel, Field

class CompanyData(BaseModel):
    name: str = Field(description="Company name")
    description: str = Field(description="Description")
    features: list[str] = Field(description="Features")

response = client.crawl(
    url="https://example.com",
    prompt="Extract company info",
    schema=CompanyData,
    depth=1,
    breadth=None,
    max_pages=5,
    rules={
        "include_paths": ["/about", "/company/*"],
        "exclude_paths": ["/admin/*", "/private/**"],
        "same_domain": True
    }
)

import { crawl } from 'scrapegraph-js';
import { z } from 'zod';

const apiKey = 'your-api-key';

const CompanySchema = z.object({
  name: z.string().describe('Company name'),
  description: z.string().describe('Description'),
  features: z.array(z.string()).describe('Features')
});

const response = await crawl(apiKey, {
  url: 'https://example.com',
  prompt: 'Extract company info',
  depth: 1,
  max_pages: 5,
  schema: CompanySchema,
  rules: {
    include_paths: ['/about', '/company/*'],
    exclude_paths: ['/admin/*', '/private/**'],
    same_domain: true
  }
});

if (response.status === 'success') {
  console.log(response.data);
}

Async Support

SmartCrawler supports async execution for large crawls:

import asyncio
from scrapegraph_py import AsyncClient

async def main():
    async with AsyncClient(api_key="your-api-key") as client:
        task = await client.crawl(
            url="https://scrapegraphai.com/",
            prompt="Extract info about the company",
            depth=2,
            breadth=None,
            max_pages=10,
            rules={
                "include_paths": ["/about", "/services/*"],
                "exclude_paths": ["/admin/*", "/api/**"],
                "same_domain": True
            }
        )
        # Poll for result
        result = await client.get_crawl_result(task["task_id"])
        print(result)

if __name__ == "__main__":
    asyncio.run(main())

import { crawl } from 'scrapegraph-js';

const apiKey = 'your-api-key';

const response = await crawl(apiKey, {
  url: 'https://scrapegraphai.com/',
  prompt: 'Extract info about the company',
  depth: 2,
  max_pages: 10,
  rules: {
    include_paths: ['/about', '/services/*'],
    exclude_paths: ['/admin/*', '/api/**'],
    same_domain: true
  }
});

if (response.status === 'success') {
  console.log(response.data);
}

Validation & Error Handling

SmartCrawler performs advanced validation:

Ensures either url or website_html is provided
Validates HTML size (max 2MB)
Checks for valid URLs and HTML structure
Handles empty or invalid prompts
Returns clear error messages for all validation failures

Endpoint Details

POST https://api.scrapegraphai.com/v1/crawl

Required Headers

Header	Description
SGAI-APIKEY	Your API authentication key
Content-Type	application/json

Request Body

Field	Type	Required	Description
url	string	Yes*	Starting URL (*or website_html required)
website_html	string	No	Raw HTML content (max 2MB)
prompt	string	Yes	Extraction instructions
schema	object	No	Output schema
headers	object	No	Custom headers
number_of_scrolls	int	No	Infinite scroll per page
depth	int	No	Crawl depth
breadth	int	No	Maximum number of links to crawl per depth level. If null/undefined, unlimited (default). Controls the ‘width’ of exploration at each depth. Useful for limiting crawl scope on large sites. Note: max_pages always takes priority. Ignored when sitemap=true.
max_pages	int	No	Max pages to crawl
rules	object	No	Crawl rules object with optional fields: `exclude` (regex URL patterns), `include_paths` (path patterns to include), `exclude_paths` (path patterns to exclude), `same_domain` (boolean)
sitemap	bool	No	Use sitemap.xml
wait_ms	int	No	Milliseconds to wait before scraping each page. Useful for pages with heavy JavaScript rendering (default: 3000).

Response Format

{
  "status": "success",
  "result": {
    "status": "done",
    "llm_result": { /* Structured extraction */ },
    "crawled_urls": ["..."],
    "pages": [ { "url": "...", "markdown": "..." }, ... ]
  }
}

Key Features

Multi-Page Extraction

Crawl and extract from entire sites, not just single pages

AI Understanding

Contextual extraction across multiple pages

Markdown Conversion

Cost-effective HTML to markdown conversion (80% savings!)

Crawl Rules

Fine-tune what gets crawled and extracted

Schema Support

Define custom output schemas for structured results

Webhook Notifications

Get notified when crawl jobs complete with signed webhooks

Use Cases

AI Extraction Mode

Site-wide data extraction with smart understanding
Product catalog crawling with structured output
Legal/Privacy/Terms aggregation with AI parsing
Research and competitive analysis with insights
Multi-page blog/news scraping with content analysis

Markdown Conversion Mode

Website documentation archival and migration
Content backup and preservation (80% cheaper!)
Blog/article collection in markdown format
Site content analysis without AI overhead
Fast bulk content conversion for CMS migration

Best Practices

AI Extraction Mode

Be specific in your prompts for better results
Use schemas for structured output validation
Test prompts on single pages first
Include examples in your schema descriptions

Markdown Conversion Mode

Perfect for content archival and documentation
No prompt required - set extraction_mode: false
80% cheaper than AI mode (2 credits vs 10 per page)
Ideal for bulk content migration

General

Set reasonable max_pages and depth limits
Use rules with include_paths and exclude_paths to precisely control which pages to crawl
Use exclude for full URL regex patterns when needed
exclude_paths takes precedence over include_paths
Always handle errors and poll for results
Monitor your credit usage and rate limits

Webhooks

SmartCrawler supports webhook notifications for job completion. When you provide a webhook_url, you’ll receive a POST request when the crawl finishes.

Setting Up Webhooks

Configure your webhook secret in the dashboard
Provide a webhook_url in your crawl request
Verify incoming webhooks using the X-Webhook-Signature header

Python

response = client.crawl(
    url="https://example.com",
    prompt="Extract product data",
    webhook_url="https://your-server.com/webhook"
)

Each webhook request includes an X-Webhook-Signature header for verification. See the API Reference for detailed verification examples.

API Reference

For detailed API documentation, see:

Support & Resources

Documentation

Comprehensive guides and tutorials

API Reference

Detailed API documentation

Community

Join our Discord community

GitHub

Check out our open-source projects

​Overview

​Getting Started

​Quick Start

Required Headers

​Parameters

​Markdown Conversion Mode

​Benefits

​Quick Start - Markdown Mode

​Markdown Mode Response

​Crawl Rules

​Example Response

​Retrieve a Previous Crawl

​Parameters

​Custom Schema Example

​Async Support

​Validation & Error Handling

​Endpoint Details

Required Headers

​Request Body

​Response Format

​Key Features

Multi-Page Extraction

AI Understanding

Markdown Conversion

Crawl Rules

Schema Support

Webhook Notifications

​Use Cases

​AI Extraction Mode

​Markdown Conversion Mode

​Best Practices

​AI Extraction Mode

​Markdown Conversion Mode

​General

​Webhooks

​Setting Up Webhooks

​API Reference

​Support & Resources

Documentation

API Reference

Community

GitHub

Ready to Start Crawling?

Overview

Getting Started

Quick Start

Parameters

Markdown Conversion Mode

Benefits

Quick Start - Markdown Mode

Markdown Mode Response

Crawl Rules

Example Response

Retrieve a Previous Crawl

Parameters

Custom Schema Example

Async Support

Validation & Error Handling

Endpoint Details

Request Body

Response Format

Key Features

Use Cases

AI Extraction Mode

Markdown Conversion Mode

Best Practices

AI Extraction Mode

Markdown Conversion Mode

General

Webhooks

Setting Up Webhooks

API Reference

Support & Resources