SmartCrawler

​

Overview

1. AI-Powered Extraction: LLM-powered web crawling with intelligent data extraction (10 credits per page)
2. Markdown Conversion: Cost-effective HTML to markdown conversion without AI/LLM processing (2 credits per page - 80% savings!)

​

Getting Started

​

Quick Start

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.smartcrawler(
    url="https://scrapegraphai.com/",
    prompt="Extract info about the company",
    depth=2,
    max_pages=10
)

Required Headers

​

Parameters

​

Markdown Conversion Mode

​

Benefits

• 80% Cost Savings: Only 2 credits per page vs 10 credits for AI mode
• No AI/LLM Processing: Pure HTML to markdown conversion
• Clean Output: Well-formatted markdown with metadata extraction
• Fast Processing: No AI inference delays
• Perfect for: Documentation, content archival, site migration

​

Quick Start - Markdown Mode

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

# Markdown conversion mode - no prompt needed
response = client.crawl(
    url="https://scrapegraphai.com/",
    extraction_mode=False,  # False = Markdown conversion (NO AI/LLM)
    depth=2,
    max_pages=5
)

​

Markdown Mode Response

{
  "status": "success",
  "result": {
    "status": "done",
    "pages_processed": 5,
    "credits_used": 10,
    "crawled_urls": [
      "https://scrapegraphai.com/",
      "https://scrapegraphai.com/about",
      "https://scrapegraphai.com/pricing"
    ],
    "pages": [
      {
        "url": "https://scrapegraphai.com/",
        "title": "ScrapeGraphAI - AI-Powered Web Scraping",
        "markdown": "# Transform Websites into Structured Data\n\nScrapeGraphAI is the most complete web scraping library...",
        "metadata": {
          "word_count": 1250,
          "headers": ["Transform Websites", "Features", "Pricing"],
          "links_count": 25
        }
      }
    ]
  }
}

​

Crawl Rules

rules = {
    "exclude": ["/logout", "/private"],  # List of URL patterns to exclude
    "same_domain": True  # Only crawl links on the same domain
}

​

Example Response

{
  "status": "success",
  "result": {
    "status": "done",
    "llm_result": {
      "company": {
        "name": "ScrapeGraphAI, Inc",
        "description": "ScrapeGraphAI is a company that provides web scraping services using artificial intelligence...",
        "features": ["AI Agent Ready", "Universal Data Extraction", ...],
        "contact_email": "contact@scrapegraphai.com",
        "social_links": {
          "github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai",
          "linkedin": "https://www.linkedin.com/company/101881123",
          "twitter": "https://x.com/scrapegraphai"
        }
      },
      "services": [
        {"service_name": "Markdownify", ...},
        {"service_name": "Smart Scraper", ...}
      ],
      "legal": {
        "privacy_policy": "https://scrapegraphai.com/privacy",
        "terms_of_service": "https://scrapegraphai.com/terms"
      }
    },
    "crawled_urls": [
      "https://scrapegraphai.com/", ...
    ],
    "pages": [
      {
        "url": "https://scrapegraphai.com/",
        "markdown": "# Transform Websites into Structured Data\n..."
      },
      ...
    ]
  }
}

​

Retrieve a Previous Crawl

result = client.get_crawl_result(task_id="your-task-id")

​

Parameters

​

Custom Schema Example

from pydantic import BaseModel, Field

class CompanyData(BaseModel):
    name: str = Field(description="Company name")
    description: str = Field(description="Description")
    features: list[str] = Field(description="Features")

response = client.smartcrawler(
    url="https://example.com",
    prompt="Extract company info",
    schema=CompanyData,
    depth=1,
    max_pages=5
)

​

Async Support

import asyncio
from scrapegraph_py import AsyncClient

async def main():
    async with AsyncClient(api_key="your-api-key") as client:
        task = await client.smartcrawler(
            url="https://scrapegraphai.com/",
            prompt="Extract info about the company",
            depth=2,
            max_pages=10
        )
        # Poll for result
        result = await client.get_crawl_result(task["task_id"])
        print(result)

if __name__ == "__main__":
    asyncio.run(main())

​

Validation & Error Handling

• Ensures either url or website_html is provided
• Validates HTML size (max 2MB)
• Checks for valid URLs and HTML structure
• Handles empty or invalid prompts
• Returns clear error messages for all validation failures

​

Endpoint Details

POST https://api.scrapegraphai.com/v1/crawl

Required Headers

​

Request Body

​

Response Format

{
  "status": "success",
  "result": {
    "status": "done",
    "llm_result": { /* Structured extraction */ },
    "crawled_urls": ["..."],
    "pages": [ { "url": "...", "markdown": "..." }, ... ]
  }
}

​

Key Features

​

Use Cases

​

AI Extraction Mode

• Site-wide data extraction with smart understanding
• Product catalog crawling with structured output
• Legal/Privacy/Terms aggregation with AI parsing
• Research and competitive analysis with insights
• Multi-page blog/news scraping with content analysis

​

Markdown Conversion Mode

• Website documentation archival and migration
• Content backup and preservation (80% cheaper!)
• Blog/article collection in markdown format
• Site content analysis without AI overhead
• Fast bulk content conversion for CMS migration

​

Best Practices

​

AI Extraction Mode

• Be specific in your prompts for better results
• Use schemas for structured output validation
• Test prompts on single pages first
• Include examples in your schema descriptions

​

Markdown Conversion Mode

• Perfect for content archival and documentation
• No prompt required - set extraction_mode: false
• 80% cheaper than AI mode (2 credits vs 10 per page)
• Ideal for bulk content migration

​

General

• Set reasonable max_pages and depth limits
• Use rules to avoid unwanted pages (like /logout)
• Always handle errors and poll for results
• Monitor your credit usage and rate limits

​

API Reference

• Start Crawl Job
• Get Crawl Status

​

Support & Resources