Skip to main content

​
Overview

The Google ADK (Agent Development Kit) integration enables your agents to leverage ScrapeGraphAI’s powerful web scraping, crawling, and data extraction capabilities. This integration allows you to build sophisticated AI agents that can understand and process web content intelligently.

Official Google ADK Documentation

View the integration in Google ADK’s official documentation

​
Installation

Install the ScrapeGraphAI MCP server package (requires Python 3.13 or higher): 
pip install scrapegraph-mcp

​
Use with Agent

​
Local MCP Server

from google.adk.agents import Agent
from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams
from mcp import StdioServerParameters

SGAI_API_KEY = "YOUR_SCRAPEGRAPHAI_API_KEY"

root_agent = Agent(
    model="gemini-2.5-pro",
    name="scrapegraph_assistant_agent",
    instruction="""Help the user with web scraping and data extraction using
ScrapeGraph AI. You can convert webpages to markdown, extract
structured data using AI, perform web searches, crawl
multiple pages, and automate complex scraping workflows.""",
    tools=[
        MCPToolset(
            connection_params=StdioConnectionParams(
                server_params=StdioServerParameters(
                    # The following CLI command is available
                    # from `pip install scrapegraph-mcp`
                    command="scrapegraph-mcp",
                    env={
                        "SGAI_API_KEY": SGAI_API_KEY,
                    },
                ),
                timeout=300,
            ),
            # Optional: Filter which tools from the MCP server are exposed
            # tool_filter=["markdownify", "smartscraper", "searchscraper"]
        ),
    ],
)

​
Available Tools

ToolDescription
markdownifyTransform any webpage into clean, structured markdown format
smartscraperLeverage AI to extract structured data from any webpage with support for infinite scrolling
searchscraperExecute AI-powered web searches with structured, actionable results
scrapeBasic scraping endpoint to fetch page content with optional heavy JavaScript rendering
sitemapExtract sitemap URLs and structure for any website
smartcrawler_initiateInitiate intelligent multi-page web crawling (asynchronous operation)
smartcrawler_fetch_resultsRetrieve results from asynchronous crawling operations
agentic_scrapperRun advanced agentic scraping workflows with customizable steps and structured output schemas

​
Configuration

Set your ScrapeGraphAI API key: 
SGAI_API_KEY = "your-api-key-here"
Get your API key from the dashboard

​
Tool Filtering

You can optionally filter which tools from the MCP server are exposed to your agent: 
MCPToolset(
    connection_params=StdioConnectionParams(...),
    tool_filter=["markdownify", "smartscraper", "searchscraper"]
)
This allows you to limit the agent’s capabilities to only the tools you need for your specific use case.

​
Use Cases

Research Agents

Create agents that gather and analyze web data intelligently

Data Collection

Automate structured data extraction from websites

Content Processing

Convert web content into markdown for further processing

Web Crawling

Automate multi-page crawling and data extraction workflows

​
Additional Resources

​
Support

Need help with the integration?

GitHub Issues

Report bugs and request features

Discord Community

Get help from our community