Skip to main content

​
Prerequisites

​
Python SDK

pip install scrapegraph-py
Usage: 
from scrapegraph_py import Client

client = Client(api_key="your-api-key-here")

# Scrape a website
response = client.smartscraper(
    website_url="https://scrapegraphai.com",
    user_prompt="Extract information about the company"
)
print(response)
You can also set the SGAI_API_KEY environment variable and initialize the client without parameters: client = Client()
For more advanced usage, see the Python SDK documentation.

​
JavaScript SDK

Install using npm, pnpm, yarn, or bun: 
# Using npm
npm i scrapegraph-js

# Using pnpm
pnpm i scrapegraph-js

# Using yarn
yarn add scrapegraph-js

# Using bun
bun add scrapegraph-js
Usage: 
import { smartScraper } from "scrapegraph-js";

const apiKey = "your-api-key-here";
const websiteUrl = "https://scrapegraphai.com";
const prompt = "What does the company do?";

try {
  const response = await smartScraper(apiKey, websiteUrl, prompt);
  console.log(response.result);
} catch (error) {
  console.error("Error:", error);
}
Store your API keys securely in environment variables. Use .env files and libraries like dotenv to load them into your app.
For more advanced usage, see the JavaScript SDK documentation.

​
Key Concepts

​
SmartScraper

Extract specific information from any webpage using AI. Provide a URL and a prompt describing what you want to extract. Learn more

​
SearchScraper

Search and extract information from multiple web sources using AI. Start with just a prompt - SearchScraper will find relevant websites and extract the information you need. Learn more

​
SmartCrawler

AI-powered extraction for any webpage with crawl capabilities. Automatically navigate and extract data from multiple pages. Learn more

​
Markdownify

Convert any webpage into clean, formatted markdown. Perfect for content aggregation and processing. Learn more

​
Structured Output with Schemas

Both SDKs support structured output using schemas:
  • Python: Use Pydantic models
  • JavaScript: Use Zod schemas

​
Example: Extract Structured Data with Schema

​
Python Example

from scrapegraph_py import Client
from pydantic import BaseModel, Field

class CompanyInfo(BaseModel):
    company_name: str = Field(description="The company name")
    description: str = Field(description="Company description")
    website: str = Field(description="Company website URL")
    industry: str = Field(description="Industry sector")

client = Client(api_key="your-api-key")
result = client.smartscraper(
    website_url="https://scrapegraphai.com",
    user_prompt="Extract company information",
    output_schema=CompanyInfo
)
print(result)

​
JavaScript Example

import { smartScraper } from "scrapegraph-js";
import { z } from "zod";

const CompanySchema = z.object({
  company_name: z.string().describe("The company name"),
  description: z.string().describe("Company description"),
  website: z.string().url().describe("Company website URL"),
  industry: z.string().describe("Industry sector"),
});

const apiKey = "your-api-key";
const response = await smartScraper(
  apiKey,
  "https://scrapegraphai.com",
  "Extract company information",
  CompanySchema
);
console.log(response.result);

​
Next Steps