SmartScraper

​

Overview

SmartScraper is our flagship LLM-powered web scraping service that intelligently extracts structured data from any website. Using advanced LLM models, it understands context and content like a human would, making web data extraction more reliable and efficient than ever.

​

Key Features

​

Use Cases

​

Content Aggregation

• News article extraction
• Blog post summarization
• Product information gathering
• Research data collection

​

Data Analysis

• Market research
• Competitor analysis
• Price monitoring
• Trend tracking

​

AI Training

• Dataset creation
• Training data collection
• Content classification
• Knowledge base building

​

Getting Started

​

Quick Start

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.smartscraper(
    website_url="https://scrapegraphai.com/",
    user_prompt="Extract info about the company"
)

{
  "request_id": "sg-req-abc123",
  "status": "completed",
  "website_url": "https://scrapegraphai.com/",
  "user_prompt": "Extract info about the company",
  "result": {
    "company_name": "ScrapeGraphAI",
    "description": "ScrapeGraphAI is a powerful AI scraping API designed for efficient web data extraction to power LLM applications and AI agents...",
    "features": [
      "Effortless, cost-effective, and AI-powered data extraction",
      "Handles proxy rotation and rate limits",
      "Supports a wide variety of websites"
    ],
    "contact_email": "contact@scrapegraphai.com",
    "social_links": {
      "github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai",
      "linkedin": "https://www.linkedin.com/company/101881123",
      "twitter": "https://x.com/scrapegraphai"
    }
  },
  "error": ""
}

​

Advanced Usage

​

Custom Schema Example

Define exactly what data you want to extract:

​

Async Support

For applications requiring asynchronous execution, SmartScraper provides async support through the AsyncClient:

from scrapegraph_py import AsyncClient
import asyncio

async def main():
    async with AsyncClient(api_key="your-api-key") as client:
        response = await client.smartscraper(
            website_url="https://example.com",
            user_prompt="Extract the main content"
        )
        print(response)

# Run the async function
asyncio.run(main())

​

Integration Options

​

Official SDKs

• Python SDK - Perfect for data science and backend applications
• JavaScript SDK - Ideal for web applications and Node.js

​

AI Framework Integrations

• LangChain Integration - Use SmartScraper in your LLM workflows
• LlamaIndex Integration - Build powerful search and QA systems

​

Best Practices

​

Optimizing Extraction

1. Be specific in your prompts
2. Use schemas for structured data
3. Handle pagination for multi-page content
4. Implement error handling and retries

​

Rate Limiting

• Implement reasonable delays between requests
• Use async clients for better performance
• Monitor your API usage

​

Example Projects

Check out our cookbook for real-world examples:

• E-commerce product scraping
• News aggregation
• Research data collection
• Content monitoring

​

API Reference

For detailed API documentation, see:

• Start Scraping Job
• Get Job Status

​

Support & Resources