Agentic Scraper

Removed in API v2 — no direct replacement. For multi-step browser automation, see Crawl (with cookies, headers, and wait) or contact us about custom workflows. This page is kept for reference.

Overview

Agentic Scraper is our most advanced service for automating browser actions and scraping data from any website, including those protected by login or requiring complex interactions. It allows you to:

Navigate websites
Perform sequences of actions (login, clicks, form filling, scrolling, etc.)
Extract the resulting content as markdown
Optionally use AI to extract structured data according to a schema

Try it instantly in our interactive playground – no coding required!

Difference: With vs Without AI Extraction

Without AI Extraction (ai_extraction: false): You get the page content (after actions) as markdown/raw. Ideal for archiving, backup, or manual analysis.
With AI Extraction (ai_extraction: true): AI extracts structured data according to a provided schema (e.g., JSON with specific fields). Perfect for automation, integration, and data analysis.

Getting Started

Quick Start

import { agenticScraper } from 'scrapegraph-js';
import 'dotenv/config';

const apiKey = process.env.SGAI_APIKEY;

// Basic scraping without AI extraction
const response = await agenticScraper(apiKey, {
  url: 'https://scrapegraphai.com/dashboard',
  steps: [
    'Type email@gmail.com in email input box',
    'Type test-password@123 in password inputbox',
    'click on login'
  ],
  use_session: true,
  ai_extraction: false,
});
console.log(response.data);

// With AI extraction
const aiResponse = await agenticScraper(apiKey, {
  url: 'https://scrapegraphai.com/dashboard',
  steps: [
    'Type email@gmail.com in email input box',
    'Type test-password@123 in password inputbox',
    'click on login',
    'wait for dashboard to load completely'
  ],
  use_session: true,
  ai_extraction: true,
  user_prompt: 'Extract user info, dashboard sections, and remaining credits',
  output_schema: {
    dashboard_info: {
      type: "object",
      properties: {
        username: { type: "string" },
        email: { type: "string" },
        dashboard_sections: { type: "array", items: { type: "string" } },
        credits_remaining: { type: "number" }
      },
      required: ["username", "dashboard_sections"]
    }
  },
});
console.log(aiResponse.data);

# Basic scraping without AI extraction
curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/agentic-scrapper' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://scrapegraphai.com/dashboard",
  "use_session": true,
  "steps": ["Type email@gmail.com in email input box", "Type test-password@123 in password inputbox", "click on login"],
  "ai_extraction": false
}'

# With AI extraction
curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/agentic-scrapper' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://scrapegraphai.com/dashboard",
  "use_session": true,
  "steps": ["Type email@gmail.com in email input box", "Type test-password@123 in password inputbox", "click on login", "wait for dashboard to load completely"],
  "user_prompt": "Extract user info, dashboard sections, and remaining credits",
  "output_schema": {
    "dashboard_info": {
      "type": "object",
      "properties": {
        "username": {"type": "string"},
        "email": {"type": "string"},
        "dashboard_sections": {"type": "array", "items": {"type": "string"}},
        "credits_remaining": {"type": "number"}
      },
      "required": ["username", "dashboard_sections"]
    }
  },
  "ai_extraction": true
}'

import os
from dotenv import load_dotenv
from scrapegraph_py import Client
from scrapegraph_py.logger import sgai_logger

load_dotenv()
sgai_logger.set_logging(level="INFO")
api_key = os.getenv("SGAI_API_KEY")
client = Client(api_key=api_key)

# Basic example: login and scrape without AI
response = client.agenticscraper(
    url="https://scrapegraphai.com/dashboard",
    use_session=True,
    steps=[
        "Type email@gmail.com in email input box",
        "Type test-password@123 in password inputbox",
        "click on login"
    ],
    ai_extraction=False
)
print(response)

# Example with AI extraction and schema
output_schema = {
    "dashboard_info": {
        "type": "object",
        "properties": {
            "username": {"type": "string"},
            "email": {"type": "string"},
            "dashboard_sections": {"type": "array", "items": {"type": "string"}},
            "credits_remaining": {"type": "number"}
        },
        "required": ["username", "dashboard_sections"]
    }
}
ai_response = client.agenticscraper(
    url="https://scrapegraphai.com/dashboard",
    use_session=True,
    steps=[
        "Type email@gmail.com in email input box",
        "Type test-password@123 in password inputbox",
        "click on login",
        "wait for dashboard to load completely"
    ],
    user_prompt="Extract user info, dashboard sections, and remaining credits",
    output_schema=output_schema,
    ai_extraction=True
)
print(ai_response)
client.close()

# Basic scraping without AI extraction
just-scrape agentic-scraper https://scrapegraphai.com/dashboard \
  -s "Type email@gmail.com in email input box,Type test-password@123 in password inputbox,Click login" \
  --use-session

# With AI extraction
just-scrape agentic-scraper https://scrapegraphai.com/dashboard \
  -s "Type email@gmail.com in email input box,Type test-password@123 in password inputbox,Click login,wait for dashboard to load" \
  --ai-extraction -p "Extract user info, dashboard sections, and remaining credits" \
  --use-session

Main Parameters

Parameter	Type	Required	Description
apiKey	string	Yes	Your ScrapeGraph API Key
url	string	Yes	Target page URL
use_session	bool	No	Keep browser session (useful for login)
steps	array	Yes	Sequence of actions to perform (click, type, etc.)
user_prompt	string	AI only	Prompt for AI extraction
output_schema	object	AI only	JSON schema for structured output
ai_extraction	bool	No	true = AI extraction, false = raw content only

Get your API key from the dashboard

Use Cases

Login and scraping of private dashboards
Automation of forms and data collection
Extraction of structured data from complex portals
Archiving content after dynamic interactions
End-to-end testing of web flows

Best Practices

Carefully define the steps sequence to simulate user actions
Use ai_extraction: false for backup/archiving, true for structured data
Provide a detailed output_schema for reliable AI results
Handle errors and poll for request status if the response is asynchronous

API Reference

For technical details see:

Support & Resources

Documentation

Comprehensive guides and tutorials

API Reference

Detailed API documentation

Community

Join our Discord community

GitHub

Open-source projects

Main Website

Official website

Ready to try?

Get your API key and start using Agentic Scraper now!

Get Started

Services

Official SDKs

LLM SDKs

Frameworks

Personal AI Agents

No-code

Contribute

Overview

Difference: With vs Without AI Extraction

Getting Started

Quick Start

Main Parameters

Use Cases

Best Practices

API Reference

Support & Resources

Documentation

API Reference

Community

GitHub

Main Website

Ready to try?

​Overview

​Difference: With vs Without AI Extraction

​Getting Started

​Quick Start

​Main Parameters

​Use Cases

​Best Practices

​API Reference

​Support & Resources

Documentation

API Reference

Community

GitHub

Main Website

Ready to try?

Overview

Difference: With vs Without AI Extraction

Getting Started

Quick Start

Main Parameters

Use Cases

Best Practices

API Reference

Support & Resources