🦜 LangChain

Overview

The LangChain integration enables your agents to extract structured data from websites using natural language. This powerful combination allows you to build sophisticated AI agents that can understand and process web content intelligently.

Official LangChain Documentation

View the integration in LangChain’s official documentation

Installation

Install the package using pip:

pip install langchain-scrapegraph

Available Tools

SmartScraperTool

Extract structured data from any webpage using natural language prompts:

from langchain_scrapegraph.tools import SmartScraperTool

# Initialize the tool (uses SGAI_API_KEY from environment)
tool = SmartscraperTool()

# Extract information using natural language
result = tool.invoke({
    "website_url": "https://www.example.com",
    "user_prompt": "Extract the main heading and first paragraph"
})

Define the structure of the output using Pydantic models:

from typing import List
from pydantic import BaseModel, Field
from langchain_scrapegraph.tools import SmartScraperTool

class WebsiteInfo(BaseModel):
    title: str = Field(description="The main title of the webpage")
    description: str = Field(description="The main description or first paragraph")
    urls: List[str] = Field(description="The URLs inside the webpage")

# Initialize with schema
tool = SmartScraperTool(llm_output_schema=WebsiteInfo)

result = tool.invoke({
    "website_url": "https://www.example.com",
    "user_prompt": "Extract the website information"
})

LocalScraperTool

Process HTML content directly with AI extraction:

from langchain_scrapegraph.tools import LocalScraperTool

tool = LocalScraperTool()
result = tool.invoke({
    "user_prompt": "Extract all contact information",
    "website_html": "<html>...</html>"
})

from typing import Optional
from pydantic import BaseModel, Field
from langchain_scrapegraph.tools import LocalScraperTool

class CompanyInfo(BaseModel):
    name: str = Field(description="The company name")
    description: str = Field(description="The company description")
    email: Optional[str] = Field(description="Contact email if available")
    phone: Optional[str] = Field(description="Contact phone if available")

tool = LocalScraperTool(llm_output_schema=CompanyInfo)

html_content = """
<html>
    <body>
        <h1>TechCorp Solutions</h1>
        <p>We are a leading AI technology company.</p>
        <div class="contact">
            <p>Email: contact@techcorp.com</p>
            <p>Phone: (555) 123-4567</p>
        </div>
    </body>
</html>
"""

result = tool.invoke({
    "website_html": html_content,
    "user_prompt": "Extract the company information"
})

MarkdownifyTool

Convert any webpage into clean, formatted markdown:

from langchain_scrapegraph.tools import MarkdownifyTool

tool = MarkdownifyTool()
markdown = tool.invoke({"website_url": "https://example.com"})

Example Agent

Create a research agent that can gather and analyze web data:

from langchain.agents import initialize_agent, AgentType
from langchain_scrapegraph.tools import SmartScraperTool
from langchain_openai import ChatOpenAI

# Initialize tools
tools = [
    SmartScraperTool(),
]

# Create an agent
agent = initialize_agent(
    tools=tools,
    llm=ChatOpenAI(temperature=0),
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# Use the agent
response = agent.run("""
    Visit example.com, make a summary of the content and extract the main heading and first paragraph
""")

Configuration

Set your ScrapeGraph API key in your environment:

export SGAI_API_KEY="your-api-key-here"

Or set it programmatically:

import os
os.environ["SGAI_API_KEY"] = "your-api-key-here"

Get your API key from the dashboard

Use Cases

Research Agents

Create agents that gather and analyze web data

Data Collection

Automate structured data extraction from websites

Content Processing

Convert web content into markdown for further processing

Information Extraction

Extract specific data points using natural language

Support

Need help with the integration?

GitHub Issues

Report bugs and request features

Discord Community

Get help from our community

Get Started

Services

Official SDKs

Integrations

Contribute

Resources

Overview

Official LangChain Documentation

Installation

Available Tools

SmartScraperTool

LocalScraperTool

MarkdownifyTool

Example Agent

Configuration

Use Cases

Research Agents

Data Collection

Content Processing

Information Extraction

Support

GitHub Issues

Discord Community

Get Started

Services

Official SDKs

Integrations

Contribute

Resources

​Overview

Official LangChain Documentation

​Installation

​Available Tools

​SmartScraperTool

​LocalScraperTool

​MarkdownifyTool

​Example Agent

​Configuration

​Use Cases

Research Agents

Data Collection

Content Processing

Information Extraction

​Support

GitHub Issues

Discord Community

Overview

Installation

Available Tools

SmartScraperTool

LocalScraperTool

MarkdownifyTool

Example Agent

Configuration

Use Cases

Support