> ## Documentation Index > Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt > Use this file to discover all available pages before exploring further. # SmartCrawler > AI-powered website crawling and multi-page extraction **Legacy v1 service.** Use [`Crawl`](/services/crawl) instead — it is the canonical v2 API. The equivalent MCP tools are `crawl_start`, `crawl_get_status`, `crawl_stop`, and `crawl_resume`. This page is kept for reference. ## Overview SmartCrawler is our advanced web crawling service that offers two modes: 1. **AI-Powered Extraction**: LLM-powered web crawling with intelligent data extraction (10 credits per page) 2. **Markdown Conversion**: Cost-effective HTML to markdown conversion without AI/LLM processing (2 credits per page - 80% savings!) Unlike SmartScraper, which extracts data from a single page, SmartCrawler can traverse multiple pages, follow links, and either extract structured data or convert content to clean markdown from entire websites or sections. Try SmartCrawler instantly in our [interactive playground](https://scrapegraphai.com/dashboard) ## Getting Started ### Quick Start ```python Python theme={null} from scrapegraph_py import Client client = Client(api_key="your-api-key") response = client.crawl( url="https://scrapegraphai.com/", prompt="Extract info about the company", depth=2, breadth=None, max_pages=10, rules={ "include_paths": ["/about", "/services/*"], "exclude_paths": ["/admin/*", "/api/**"], "same_domain": True } ) ``` ```javascript JavaScript theme={null} import { crawl } from 'scrapegraph-js'; const apiKey = 'your-api-key'; const response = await crawl(apiKey, { url: 'https://scrapegraphai.com/', prompt: 'Extract info about the company', depth: 2, max_pages: 10, rules: { include_paths: ['/about', '/services/*'], exclude_paths: ['/admin/*', '/api/**'], same_domain: true } }); if (response.status === 'success') { console.log(response.data); } ``` ```bash cURL theme={null} curl -X 'POST' \ 'https://api.scrapegraphai.com/v1/crawl' \ -H 'accept: application/json' \ -H 'SGAI-APIKEY: your-api-key' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://scrapegraphai.com/", "prompt": "Extract info about the company", "depth": 2, "breadth": null, "max_pages": 10, "rules": { "include_paths": ["/about", "/services/*"], "exclude_paths": ["/admin/*", "/api/**"], "same_domain": true } }' ``` ```bash CLI theme={null} just-scrape crawl https://scrapegraphai.com/ -p "Extract info about the company" \ --depth 2 --max-pages 10 \ --rules '{"include_paths":["/about","/services/*"],"exclude_paths":["/admin/*","/api/**"],"same_domain":true}' ``` ##### Required Headers | Header | Description | | ------------ | --------------------------- | | SGAI-APIKEY | Your API authentication key | | Content-Type | application/json | #### Parameters | Parameter | Type | Required | Description | | ---------------- | ------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | url | string | Yes | The starting URL for the crawl. | | prompt | string | No\* | Instructions for what to extract (\*required when extraction\_mode=true). | | extraction\_mode | bool | No | When `false`, enables markdown conversion mode (default: true). | | depth | int | No | How many link levels to follow (default: 1). | | breadth | int | No | Maximum number of links to crawl per depth level. If null/undefined, unlimited (default). Controls the 'width' of exploration at each depth. Useful for limiting crawl scope on large sites. Note: max\_pages always takes priority. Ignored when sitemap=true. | | max\_pages | int | No | Maximum number of pages to crawl (default: 20). | | schema | object | No | Pydantic or Zod schema for structured output. | | rules | object | No | Crawl rules object with optional fields: `exclude` (array of regex URL patterns), `include_paths` (array of path patterns to include, supports wildcards `*` and `**`), `exclude_paths` (array of path patterns to exclude, takes precedence over `include_paths`), `same_domain` (boolean, default: true). See below for details. | | sitemap | bool | No | Use sitemap.xml for discovery (default: false). | | webhook\_url | string | No | URL to receive webhook notification on job completion. | | wait\_ms | int | No | Milliseconds to wait before scraping each page. Useful for pages with heavy JavaScript rendering that need extra time to load (default: 3000). | Get your API key from the [dashboard](https://scrapegraphai.com/dashboard) ## Markdown Conversion Mode For cost-effective content archival and when you only need clean markdown without AI processing, use the markdown conversion mode. This mode offers significant cost savings and is perfect for documentation, content migration, and simple data collection. ### Benefits * **80% Cost Savings**: Only 2 credits per page vs 10 credits for AI mode * **No AI/LLM Processing**: Pure HTML to markdown conversion * **Clean Output**: Well-formatted markdown with metadata extraction * **Fast Processing**: No AI inference delays * **Perfect for**: Documentation, content archival, site migration ### Quick Start - Markdown Mode ```python Python theme={null} from scrapegraph_py import Client client = Client(api_key="your-api-key") # Markdown conversion mode - no prompt needed response = client.crawl( url="https://scrapegraphai.com/", extraction_mode=False, # False = Markdown conversion (NO AI/LLM) depth=2, breadth=None, max_pages=5, rules={ "include_paths": ["/docs/*", "/blog/**"], "exclude_paths": ["/admin/*"], "same_domain": True } ) ``` ```javascript JavaScript theme={null} import { crawl } from 'scrapegraph-js'; const apiKey = 'your-api-key'; const response = await crawl(apiKey, { url: 'https://scrapegraphai.com/', extraction_mode: false, depth: 2, max_pages: 5, rules: { include_paths: ['/docs/*', '/blog/**'], exclude_paths: ['/admin/*'], same_domain: true } }); if (response.status === 'success') { console.log(response.data); } ``` ```bash cURL theme={null} curl -X 'POST' \ 'https://api.scrapegraphai.com/v1/crawl' \ -H 'accept: application/json' \ -H 'SGAI-APIKEY: your-api-key' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://scrapegraphai.com/", "extraction_mode": false, "depth": 2, "breadth": null, "max_pages": 5, "rules": { "include_paths": ["/docs/*", "/blog/**"], "exclude_paths": ["/admin/*"], "same_domain": true } }' ``` ```bash CLI theme={null} just-scrape crawl https://scrapegraphai.com/ --no-extraction \ --depth 2 --max-pages 5 \ --rules '{"include_paths":["/docs/*","/blog/**"],"exclude_paths":["/admin/*"],"same_domain":true}' ``` ### Markdown Mode Response ```json theme={null} { "status": "success", "result": { "status": "done", "pages_processed": 5, "credits_used": 10, "crawled_urls": [ "https://scrapegraphai.com/", "https://scrapegraphai.com/about", "https://scrapegraphai.com/pricing" ], "pages": [ { "url": "https://scrapegraphai.com/", "title": "ScrapeGraphAI - AI-Powered Web Scraping", "markdown": "# Transform Websites into Structured Data\n\nScrapeGraphAI is the most complete web scraping library...", "metadata": { "word_count": 1250, "headers": ["Transform Websites", "Features", "Pricing"], "links_count": 25 } } ] } } ``` ### Crawl Rules You can control the crawl behavior with the `rules` object: ```python Python theme={null} rules = { "exclude": ["https://example.com/logout", "https://example.com/admin/*"], # Regex patterns for full URLs to exclude "include_paths": ["/products/*", "/blog/**"], # Path patterns to include "exclude_paths": ["/admin/*", "/api/**", "/private/*"], # Path patterns to exclude (takes precedence) "same_domain": True # Only crawl links on the same domain } ``` | Field | Type | Default | Description | | -------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | exclude | list | \[] | List of URL patterns (regex) to exclude from crawling. Matches full URL. | | include\_paths | list | \[] | **(Optional)** List of path patterns to include (e.g., `["/products/*", "/blog/**"]`). Supports wildcards: `*` matches any characters, `**` matches any path segments. If empty or not specified, all paths are included. | | exclude\_paths | list | \[] | **(Optional)** List of path patterns to exclude (e.g., `["/admin/*", "/api/**"]`). Supports wildcards: `*` matches any characters, `**` matches any path segments. Takes precedence over `include_paths`. If empty or not specified, no paths are excluded. | | same\_domain | bool | True | Restrict crawl to the same domain | Both `include_paths` and `exclude_paths` are optional. If `include_paths` is not specified or empty, all paths are included. If `exclude_paths` is not specified or empty, no paths are excluded. The `exclude_paths` patterns take precedence over `include_paths` patterns. ### Example Response ```json theme={null} { "status": "success", "result": { "status": "done", "llm_result": { "company": { "name": "ScrapeGraphAI, Inc", "description": "ScrapeGraphAI is a company that provides web scraping services using artificial intelligence...", "features": ["AI Agent Ready", "Universal Data Extraction", ...], "contact_email": "contact@scrapegraphai.com", "social_links": { "github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai", "linkedin": "https://www.linkedin.com/company/101881123", "twitter": "https://x.com/scrapegraphai" } }, "services": [ {"service_name": "Markdownify", ...}, {"service_name": "Smart Scraper", ...} ], "legal": { "privacy_policy": "https://scrapegraphai.com/privacy", "terms_of_service": "https://scrapegraphai.com/terms" } }, "crawled_urls": [ "https://scrapegraphai.com/", ... ], "pages": [ { "url": "https://scrapegraphai.com/", "markdown": "# Transform Websites into Structured Data\n..." }, ... ] } } ``` * `llm_result`: Structured extraction based on your prompt/schema * `crawled_urls`: List of all URLs visited * `pages`: List of objects with `url` and extracted `markdown` content ### Retrieve a Previous Crawl You can retrieve the result of a crawl job by its task ID: ```python Python theme={null} result = client.get_crawl_result(task_id="your-task-id") ``` ```javascript JavaScript theme={null} import { history } from 'scrapegraph-js'; const apiKey = 'your_api_key'; const response = await history(apiKey, { service: 'crawl', page: 1, page_size: 10, }); if (response.status === 'success') { console.log(response.data.requests); } ``` #### Parameters | Parameter | Type | Required | Description | | --------- | ------ | -------- | ------------------------ | | apiKey | string | Yes | The ScrapeGraph API Key. | | taskId | string | Yes | The crawl job task ID. | ### Custom Schema Example Define exactly what data you want to extract from every page: ```python Python theme={null} from pydantic import BaseModel, Field class CompanyData(BaseModel): name: str = Field(description="Company name") description: str = Field(description="Description") features: list[str] = Field(description="Features") response = client.crawl( url="https://example.com", prompt="Extract company info", schema=CompanyData, depth=1, breadth=None, max_pages=5, rules={ "include_paths": ["/about", "/company/*"], "exclude_paths": ["/admin/*", "/private/**"], "same_domain": True } ) ``` ```javascript JavaScript theme={null} import { crawl } from 'scrapegraph-js'; import { z } from 'zod'; const apiKey = 'your-api-key'; const CompanySchema = z.object({ name: z.string().describe('Company name'), description: z.string().describe('Description'), features: z.array(z.string()).describe('Features') }); const response = await crawl(apiKey, { url: 'https://example.com', prompt: 'Extract company info', depth: 1, max_pages: 5, schema: CompanySchema, rules: { include_paths: ['/about', '/company/*'], exclude_paths: ['/admin/*', '/private/**'], same_domain: true } }); if (response.status === 'success') { console.log(response.data); } ``` ### Async Support SmartCrawler supports async execution for large crawls: ```python Python theme={null} import asyncio from scrapegraph_py import AsyncClient async def main(): async with AsyncClient(api_key="your-api-key") as client: task = await client.crawl( url="https://scrapegraphai.com/", prompt="Extract info about the company", depth=2, breadth=None, max_pages=10, rules={ "include_paths": ["/about", "/services/*"], "exclude_paths": ["/admin/*", "/api/**"], "same_domain": True } ) # Poll for result result = await client.get_crawl_result(task["task_id"]) print(result) if __name__ == "__main__": asyncio.run(main()) ``` ```javascript JavaScript theme={null} import { crawl } from 'scrapegraph-js'; const apiKey = 'your-api-key'; const response = await crawl(apiKey, { url: 'https://scrapegraphai.com/', prompt: 'Extract info about the company', depth: 2, max_pages: 10, rules: { include_paths: ['/about', '/services/*'], exclude_paths: ['/admin/*', '/api/**'], same_domain: true } }); if (response.status === 'success') { console.log(response.data); } ``` ### Validation & Error Handling SmartCrawler performs advanced validation: * Ensures either `url` or `website_html` is provided * Validates HTML size (max 2MB) * Checks for valid URLs and HTML structure * Handles empty or invalid prompts * Returns clear error messages for all validation failures ### Endpoint Details ```bash theme={null} POST https://api.scrapegraphai.com/v1/crawl ``` ##### Required Headers | Header | Description | | ------------ | --------------------------- | | SGAI-APIKEY | Your API authentication key | | Content-Type | application/json | #### Request Body | Field | Type | Required | Description | | ------------------- | ------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | url | string | Yes\* | Starting URL (\*or website\_html required) | | website\_html | string | No | Raw HTML content (max 2MB) | | prompt | string | Yes | Extraction instructions | | schema | object | No | Output schema | | headers | object | No | Custom headers | | number\_of\_scrolls | int | No | Infinite scroll per page | | depth | int | No | Crawl depth | | breadth | int | No | Maximum number of links to crawl per depth level. If null/undefined, unlimited (default). Controls the 'width' of exploration at each depth. Useful for limiting crawl scope on large sites. Note: max\_pages always takes priority. Ignored when sitemap=true. | | max\_pages | int | No | Max pages to crawl | | rules | object | No | Crawl rules object with optional fields: `exclude` (regex URL patterns), `include_paths` (path patterns to include), `exclude_paths` (path patterns to exclude), `same_domain` (boolean) | | sitemap | bool | No | Use sitemap.xml | | wait\_ms | int | No | Milliseconds to wait before scraping each page. Useful for pages with heavy JavaScript rendering (default: 3000). | #### Response Format ```json theme={null} { "status": "success", "result": { "status": "done", "llm_result": { /* Structured extraction */ }, "crawled_urls": ["..."], "pages": [ { "url": "...", "markdown": "..." }, ... ] } } ``` ### Key Features Crawl and extract from entire sites, not just single pages Contextual extraction across multiple pages Cost-effective HTML to markdown conversion (80% savings!) Fine-tune what gets crawled and extracted Define custom output schemas for structured results Get notified when crawl jobs complete with signed webhooks ## Use Cases ### AI Extraction Mode * Site-wide data extraction with smart understanding * Product catalog crawling with structured output * Legal/Privacy/Terms aggregation with AI parsing * Research and competitive analysis with insights * Multi-page blog/news scraping with content analysis ### Markdown Conversion Mode * Website documentation archival and migration * Content backup and preservation (80% cheaper!) * Blog/article collection in markdown format * Site content analysis without AI overhead * Fast bulk content conversion for CMS migration ## Best Practices ### AI Extraction Mode * Be specific in your prompts for better results * Use schemas for structured output validation * Test prompts on single pages first * Include examples in your schema descriptions ### Markdown Conversion Mode * Perfect for content archival and documentation * No prompt required - set `extraction_mode: false` * 80% cheaper than AI mode (2 credits vs 10 per page) * Ideal for bulk content migration ### General * Set reasonable `max_pages` and `depth` limits * Use `rules` with `include_paths` and `exclude_paths` to precisely control which pages to crawl * Use `exclude` for full URL regex patterns when needed * `exclude_paths` takes precedence over `include_paths` * Always handle errors and poll for results * Monitor your credit usage and rate limits ## Webhooks SmartCrawler supports webhook notifications for job completion. When you provide a `webhook_url`, you'll receive a POST request when the crawl finishes. ### Setting Up Webhooks 1. Configure your webhook secret in the [dashboard](https://scrapegraphai.com/dashboard) 2. Provide a `webhook_url` in your crawl request 3. Verify incoming webhooks using the `X-Webhook-Signature` header ```python Python theme={null} response = client.crawl( url="https://example.com", prompt="Extract product data", webhook_url="https://your-server.com/webhook" ) ``` Each webhook request includes an `X-Webhook-Signature` header for verification. See the [API Reference](/api-reference/endpoint/smartcrawler/start#webhook-signature-verification) for detailed verification examples. ## API Reference For detailed API documentation, see: * [Start Crawl Job](/api-reference/endpoint/smartcrawler/start) * [Get Crawl Status](/api-reference/endpoint/smartcrawler/get-status) ## Support & Resources Comprehensive guides and tutorials Detailed API documentation Join our Discord community Check out our open-source projects Sign up now and get your API key to begin extracting data with SmartCrawler!