LlamaIndex - ScrapeGraphAI

Overview

LlamaIndex is a data framework for building LLM-powered agents and RAG applications. This page shows how to wire scrapegraph-py ≥ 2.0.1 into LlamaIndex as a set of FunctionTools so your agents can scrape pages, extract structured data, search the web, run asynchronous crawls, and manage scheduled monitors.

Official LlamaIndex Documentation

Learn more about building agents and RAG pipelines with LlamaIndex

Which package? LlamaIndex also ships a pre-built tool spec at llama-index-tools-scrapegraphai, but it currently depends on scrapegraph-py<2 and targets the legacy v1 backend. New v2 API keys are rejected by that path. The recipes below use the v2 SDK directly — they work with the current dashboard and every v2 endpoint (scrape, extract, search, crawl, monitor).

Installation

pip install -U llama-index
pip install "scrapegraph-py>=2.0.1"

Set your API key:

export SGAI_API_KEY="your-api-key"

Quick Start

Initialize the v2 client and expose a tool to any LlamaIndex agent:

from scrapegraph_py import ScrapeGraphAI
from llama_index.core.tools import FunctionTool
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

sgai = ScrapeGraphAI()  # reads SGAI_API_KEY

def scrape(url: str) -> str:
    """Fetch a page and return its markdown content."""
    result = sgai.scrape(url)
    if result.status == "error":
        raise RuntimeError(result.error)
    return result.data.results.get("markdown", {}).get("data", [""])[0]

agent = FunctionAgent(
    tools=[FunctionTool.from_defaults(fn=scrape)],
    llm=OpenAI(model="gpt-4o"),
)

Cookbook recipes

The following recipes are ported from the official scrapegraph-py cookbook notebooks, swapped to call the v2 extract endpoint so they run against the current dashboard API key.

1. Extract company info

Pull founders, pricing plans, and social links off a company homepage. Based on cookbook/company-info/.

from pydantic import BaseModel, Field
from typing import List
from scrapegraph_py import ScrapeGraphAI

class FounderSchema(BaseModel):
    name: str = Field(description="Name of the founder")
    role: str = Field(description="Role of the founder in the company")
    linkedin: str = Field(description="LinkedIn profile of the founder")

class PricingPlanSchema(BaseModel):
    tier: str = Field(description="Name of the pricing tier")
    price: str = Field(description="Price of the plan")
    credits: int = Field(description="Number of credits included in the plan")

class SocialLinksSchema(BaseModel):
    linkedin: str
    twitter: str
    github: str

class CompanyInfoSchema(BaseModel):
    company_name: str
    description: str
    founders: List[FounderSchema] = Field(default_factory=list)
    pricing_plans: List[PricingPlanSchema] = Field(default_factory=list)
    social_links: SocialLinksSchema

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract info about the company",
    url="https://scrapegraphai.com/",
    schema=CompanyInfoSchema.model_json_schema(),
)

if res.status == "success":
    print(res.data.json_data)

Pull a ranked list of trending repositories. Based on cookbook/github-trending/.

from pydantic import BaseModel, Field
from typing import List
from scrapegraph_py import ScrapeGraphAI

class RepositorySchema(BaseModel):
    name: str = Field(description="Name of the repository (e.g. 'owner/repo')")
    description: str = Field(description="Description of the repository")
    stars: int = Field(description="Star count")
    forks: int = Field(description="Fork count")
    today_stars: int = Field(description="Stars gained today")
    language: str = Field(description="Programming language used")

class ListRepositoriesSchema(BaseModel):
    repositories: List[RepositorySchema]

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract only the first ten trending repositories",
    url="https://github.com/trending",
    schema=ListRepositoriesSchema.model_json_schema(),
)

if res.status == "success":
    for repo in res.data.json_data["repositories"]:
        print(f"{repo['name']} — {repo['stars']} ★")

3. Extract a news feed

Pull headlines from a news section. Based on cookbook/wired-news/.

from pydantic import BaseModel, Field
from typing import List
from scrapegraph_py import ScrapeGraphAI

class NewsItemSchema(BaseModel):
    category: str = Field(description="Category of the news (e.g. 'Health', 'Environment')")
    title: str = Field(description="Title of the news article")
    link: str = Field(description="URL to the news article")
    author: str = Field(description="Author of the news article")

class ListNewsSchema(BaseModel):
    news: List[NewsItemSchema]

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract the first 10 news articles on the page",
    url="https://www.wired.com/category/science/",
    schema=ListNewsSchema.model_json_schema(),
)

if res.status == "success":
    for item in res.data.json_data["news"]:
        print(f"[{item['category']}] {item['title']}")

4. Extract real-estate listings

Pull house listings with price, address, and tags. Based on cookbook/homes-forsale/.

from pydantic import BaseModel, Field
from typing import List
from scrapegraph_py import ScrapeGraphAI, FetchConfig

class HouseListingSchema(BaseModel):
    price: int = Field(description="Price of the house in USD")
    bedrooms: int
    bathrooms: int
    square_feet: int = Field(description="Total square footage of the house")
    address: str
    city: str
    state: str
    zip_code: str
    tags: List[str] = Field(description="Tags like 'New construction' or 'Large garage'")
    agent_name: str
    agency: str

class HousesListingsSchema(BaseModel):
    houses: List[HouseListingSchema]

sgai = ScrapeGraphAI()

# Anti-bot heavy sites need stealth + JS rendering
res = sgai.extract(
    "Extract information about houses for sale",
    url="https://www.zillow.com/san-francisco-ca/",
    schema=HousesListingsSchema.model_json_schema(),
    fetch_config=FetchConfig(mode="js", stealth=True, wait=2000),
)

5. Research agent with `ReActAgent`

Combine scrape + extract into a LlamaIndex ReActAgent so the LLM decides which tool to call per step. Based on cookbook/research-agent/.

from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig
from llama_index.core.tools import FunctionTool
from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI

sgai = ScrapeGraphAI()

def scrape(url: str) -> str:
    """Fetch a page and return its markdown content."""
    res = sgai.scrape(url, formats=[MarkdownFormatConfig()])
    if res.status != "success":
        return res.error or ""
    return res.data.results.get("markdown", {}).get("data", [""])[0]

def extract(url: str, prompt: str) -> dict:
    """Extract structured data from a URL using the given prompt."""
    res = sgai.extract(prompt, url=url)
    return res.data.json_data if res.status == "success" else {"error": res.error}

tools = [FunctionTool.from_defaults(fn=f) for f in (scrape, extract)]

agent = ReActAgent.from_tools(
    tools,
    llm=OpenAI(model="gpt-4o"),
    verbose=True,
)

response = agent.chat(
    "Extract all the keyboard names and prices from "
    "https://www.ebay.com/sch/i.html?_nkw=keyboards"
)
print(response)

Usage Reference

Scrape tool

from scrapegraph_py import (
    ScrapeGraphAI,
    MarkdownFormatConfig, HtmlFormatConfig, JsonFormatConfig,
)
from llama_index.core.tools import FunctionTool

sgai = ScrapeGraphAI()

def scrape(url: str, format: str = "markdown") -> dict:
    """Fetch `url` and return the requested format.

    format: one of "markdown", "html", "json".
    """
    entries = {
        "markdown": MarkdownFormatConfig(mode="reader"),
        "html": HtmlFormatConfig(),
        "json": JsonFormatConfig(prompt="Extract the main content"),
    }
    result = sgai.scrape(url, formats=[entries[format]])
    if result.status == "error":
        return {"error": result.error}
    return result.data.results

scrape_tool = FunctionTool.from_defaults(fn=scrape)

Extract tool

from scrapegraph_py import ScrapeGraphAI
from llama_index.core.tools import FunctionTool

sgai = ScrapeGraphAI()

def extract(url: str, prompt: str, schema: dict | None = None) -> dict:
    """Extract structured data from `url` per `prompt`."""
    result = sgai.extract(prompt, url=url, schema=schema)
    if result.status == "error":
        return {"error": result.error}
    return result.data.json_data

extract_tool = FunctionTool.from_defaults(fn=extract)

Search tool

from scrapegraph_py import ScrapeGraphAI
from llama_index.core.tools import FunctionTool

sgai = ScrapeGraphAI()

def search(
    query: str,
    num_results: int = 5,
    prompt: str | None = None,
    time_range: str | None = None,
    country: str | None = None,
) -> dict:
    """Search the web and return structured results.

    time_range: "past_hour", "past_24_hours", "past_week", "past_month", "past_year".
    country: two-letter ISO country code (e.g. "us", "it").
    """
    result = sgai.search(
        query,
        num_results=num_results,
        prompt=prompt,
        time_range=time_range,
        location_geo_code=country,
    )
    if result.status == "error":
        return {"error": result.error}
    return {
        "results": [{"title": r.title, "url": r.url} for r in result.data.results],
        "json_data": result.data.json_data,
    }

search_tool = FunctionTool.from_defaults(fn=search)

Crawl tool

Crawls are asynchronous — poll sgai.crawl.get(id) until status in ("completed", "failed", "stopped").

import time
from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig
from llama_index.core.tools import FunctionTool

sgai = ScrapeGraphAI()

def crawl(
    url: str,
    max_depth: int = 2,
    max_pages: int = 50,
    include_patterns: list[str] | None = None,
    exclude_patterns: list[str] | None = None,
) -> dict:
    """Crawl a site and return pages as markdown once the job completes."""
    start = sgai.crawl.start(
        url,
        formats=[MarkdownFormatConfig()],
        max_depth=max_depth,
        max_pages=max_pages,
        include_patterns=include_patterns,
        exclude_patterns=exclude_patterns,
    )
    if start.status == "error":
        return {"error": start.error}

    crawl_id = start.data.id
    while True:
        status = sgai.crawl.get(crawl_id)
        if status.data.status in ("completed", "failed", "stopped"):
            return status.data.model_dump()
        time.sleep(2)

crawl_tool = FunctionTool.from_defaults(fn=crawl)

Monitor tool

from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig
from llama_index.core.tools import FunctionTool

sgai = ScrapeGraphAI()

def create_monitor(
    url: str,
    name: str,
    interval: str,
    webhook_url: str | None = None,
) -> dict:
    """Create a recurring monitor (cron `interval`) that tracks changes on `url`."""
    result = sgai.monitor.create(
        url,
        interval,
        name=name,
        formats=[MarkdownFormatConfig()],
        webhook_url=webhook_url,
    )
    if result.status == "error":
        return {"error": result.error}
    return {"cron_id": result.data.cron_id}

monitor_tool = FunctionTool.from_defaults(fn=create_monitor)

Configuration Options

The v2 ScrapeGraphAI client accepts:

Parameter	Type	Default	Description
`api_key`	`str \| None`	`None`	Falls back to `SGAI_API_KEY`.
`base_url`	`str`	`https://v2-api.scrapegraphai.com/api`	Override via `SGAI_API_URL`.
`timeout`	`int`	`120`	Request timeout in seconds. Override via `SGAI_TIMEOUT`.

Each v2 resource maps 1:1 to a LlamaIndex tool:

SDK call	Endpoint	First positional arg
`sgai.scrape(url, ...)`	Scrape	`url`
`sgai.extract(prompt, url=..., ...)`	Extract	`prompt`
`sgai.search(query, ...)`	Search	`query`
`sgai.crawl.start(url, ...)`, `.get/.stop/.resume/.delete(id)`	Crawl	`url` / `id`
`sgai.monitor.create(url, interval, ...)`, `.list/.get/.update/.pause/.resume/.delete/.activity(...)`	Monitor	`url`, `interval`

Every call returns an ApiResult[T] with status, data, error, and elapsed_ms — so tools can surface errors without exceptions.

Advanced Usage

Combining every endpoint in one agent

Hand the full tool list to an agent and let it pick the right tool per step:

from scrapegraph_py import ScrapeGraphAI, MarkdownFormatConfig
from llama_index.core.tools import FunctionTool
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

sgai = ScrapeGraphAI()

def scrape(url: str) -> str:
    res = sgai.scrape(url)
    if res.status != "success":
        return res.error or ""
    return res.data.results.get("markdown", {}).get("data", [""])[0]

def extract(url: str, prompt: str) -> dict:
    res = sgai.extract(prompt, url=url)
    return res.data.json_data if res.status == "success" else {"error": res.error}

def search(query: str, num_results: int = 5) -> list[dict]:
    res = sgai.search(query, num_results=num_results)
    if res.status == "error":
        return [{"error": res.error}]
    return [{"title": r.title, "url": r.url} for r in res.data.results]

def crawl(url: str, max_pages: int = 20) -> dict:
    res = sgai.crawl.start(url, formats=[MarkdownFormatConfig()], max_pages=max_pages)
    return {"crawl_id": res.data.id} if res.status == "success" else {"error": res.error}

def create_monitor(url: str, name: str, interval: str) -> dict:
    res = sgai.monitor.create(
        url, interval, name=name, formats=[MarkdownFormatConfig()],
    )
    return {"cron_id": res.data.cron_id} if res.status == "success" else {"error": res.error}

tools = [FunctionTool.from_defaults(fn=f) for f in (
    scrape, extract, search, crawl, create_monitor,
)]

agent = FunctionAgent(
    tools=tools,
    llm=OpenAI(model="gpt-4o"),
    system_prompt=(
        "You are a web research assistant powered by ScrapeGraphAI v2. "
        "Pick the most specific tool for the job: scrape for a single page, "
        "extract for structured data, search for open-web questions, "
        "crawl for multi-page jobs, and create_monitor for recurring jobs."
    ),
)

response = await agent.run(
    "Research the latest blog posts on scrapegraphai.com and summarize them."
)
print(response)

Async client

Every resource has an async twin via AsyncScrapeGraphAI:

from scrapegraph_py import AsyncScrapeGraphAI
from llama_index.core.tools import FunctionTool

async def scrape(url: str) -> str:
    async with AsyncScrapeGraphAI() as sgai:
        res = await sgai.scrape(url)
        if res.status == "error":
            raise RuntimeError(res.error)
        return res.data.results.get("markdown", {}).get("data", [""])[0]

scrape_tool = FunctionTool.from_defaults(async_fn=scrape)

Custom agent configuration

Plug the tools into any LlamaIndex agent — ReActAgent, workflow-based, or third-party:

from llama_index.core.agent.workflow import ReActAgent
from llama_index.llms.anthropic import Anthropic

agent = ReActAgent(
    tools=tools,
    llm=Anthropic(model="claude-sonnet-4-6"),
    verbose=True,
)

Features

Scrape

Fetch pages as markdown, HTML, screenshots, JSON, links, images, summary, or branding

Extract

Structured extraction with a prompt and a JSON schema

Search

AI-powered web search with optional structured output

Crawl

Asynchronous multi-page crawls with start / stop / resume controls

Monitor

Cron-scheduled jobs with webhook notifications on change

Typed Requests

Pydantic request models and ApiResult[T] responses — no surprises

Async-Ready

AsyncScrapeGraphAI mirrors every resource for parallel pipelines

Agent-Ready

Every endpoint exposed as a drop-in LlamaIndex FunctionTool

Best Practices

Tool selection — pass only the tools the agent actually needs; a shorter tool list keeps prompts tighter and routing more accurate.
Schema design — when calling extract or search, pass a concrete JSON schema (YourSchema.model_json_schema()) so the extractor has a clear target.
Format entries — scrape accepts a list of format entries; combine MarkdownFormatConfig, ScreenshotFormatConfig, and JsonFormatConfig in one call to avoid multiple round-trips.
Async crawls — sgai.crawl.start returns immediately; always poll sgai.crawl.get(id) until status in ("completed", "failed", "stopped").
ApiResult — branch on result.status instead of wrapping calls in try/except; the SDK never raises on API-level errors.
Hard pages — stealth mode + mode="js" fetch config handles most anti-bot sites (see the Zillow recipe above).

Support

LlamaIndex Discord

Join the LlamaIndex community for support and discussions

scrapegraph-py cookbook

Browse the full set of notebook examples

ScrapeGraphAI Discord

Get help with ScrapeGraphAI features

Documentation

Explore the full API reference

Get Started

Services

Official SDKs

LLM SDKs

Frameworks

No-code

Contribute

Documentation Index

​Overview

Official LlamaIndex Documentation

​Installation

​Quick Start

​Cookbook recipes

​1. Extract company info

​2. Extract GitHub trending repos

​3. Extract a news feed

​4. Extract real-estate listings

​5. Research agent with ReActAgent

​Usage Reference

​Scrape tool

​Extract tool

​Search tool

​Crawl tool

​Monitor tool

​Configuration Options

​Advanced Usage

​Combining every endpoint in one agent

​Async client

​Custom agent configuration

​Features

Scrape

Extract

Search

Crawl

Monitor

Typed Requests

Async-Ready

Agent-Ready

​Best Practices

​Support

LlamaIndex Discord

scrapegraph-py cookbook

ScrapeGraphAI Discord

Documentation

Overview

Installation

Quick Start

Cookbook recipes

1. Extract company info

2. Extract GitHub trending repos

3. Extract a news feed

4. Extract real-estate listings

5. Research agent with `ReActAgent`

Usage Reference

Scrape tool

Extract tool

Search tool

Crawl tool

Monitor tool

Configuration Options

Advanced Usage

Combining every endpoint in one agent

Async client

Custom agent configuration

Features

Best Practices

Support