> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Agentic Scraper
> Automate browser actions and scrape data with or without AI, even on login-protected sites.
**Removed in API v2 — no direct replacement.** For multi-step browser automation, see [`Crawl`](/services/crawl) (with `cookies`, `headers`, and `wait`) or contact us about custom workflows. This page is kept for reference.
## Overview
Agentic Scraper is our most advanced service for automating browser actions and scraping data from any website, including those protected by login or requiring complex interactions. It allows you to:
* Navigate websites
* Perform sequences of actions (login, clicks, form filling, scrolling, etc.)
* Extract the resulting content as markdown
* **Optionally** use AI to extract structured data according to a schema
Try it instantly in our [interactive playground](https://scrapegraphai.com/dashboard) – no coding required!
## Difference: With vs Without AI Extraction
* **Without AI Extraction (`ai_extraction: false`)**: You get the page content (after actions) as markdown/raw. Ideal for archiving, backup, or manual analysis.
* **With AI Extraction (`ai_extraction: true`)**: AI extracts structured data according to a provided schema (e.g., JSON with specific fields). Perfect for automation, integration, and data analysis.
## Getting Started
### Quick Start
```javascript JavaScript theme={null}
import { agenticScraper } from 'scrapegraph-js';
import 'dotenv/config';
const apiKey = process.env.SGAI_APIKEY;
// Basic scraping without AI extraction
const response = await agenticScraper(apiKey, {
url: 'https://scrapegraphai.com/dashboard',
steps: [
'Type email@gmail.com in email input box',
'Type test-password@123 in password inputbox',
'click on login'
],
use_session: true,
ai_extraction: false,
});
console.log(response.data);
// With AI extraction
const aiResponse = await agenticScraper(apiKey, {
url: 'https://scrapegraphai.com/dashboard',
steps: [
'Type email@gmail.com in email input box',
'Type test-password@123 in password inputbox',
'click on login',
'wait for dashboard to load completely'
],
use_session: true,
ai_extraction: true,
user_prompt: 'Extract user info, dashboard sections, and remaining credits',
output_schema: {
dashboard_info: {
type: "object",
properties: {
username: { type: "string" },
email: { type: "string" },
dashboard_sections: { type: "array", items: { type: "string" } },
credits_remaining: { type: "number" }
},
required: ["username", "dashboard_sections"]
}
},
});
console.log(aiResponse.data);
```
```bash cURL theme={null}
# Basic scraping without AI extraction
curl -X 'POST' \
'https://api.scrapegraphai.com/v1/agentic-scrapper' \
-H 'accept: application/json' \
-H 'SGAI-APIKEY: your-api-key' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://scrapegraphai.com/dashboard",
"use_session": true,
"steps": ["Type email@gmail.com in email input box", "Type test-password@123 in password inputbox", "click on login"],
"ai_extraction": false
}'
# With AI extraction
curl -X 'POST' \
'https://api.scrapegraphai.com/v1/agentic-scrapper' \
-H 'accept: application/json' \
-H 'SGAI-APIKEY: your-api-key' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://scrapegraphai.com/dashboard",
"use_session": true,
"steps": ["Type email@gmail.com in email input box", "Type test-password@123 in password inputbox", "click on login", "wait for dashboard to load completely"],
"user_prompt": "Extract user info, dashboard sections, and remaining credits",
"output_schema": {
"dashboard_info": {
"type": "object",
"properties": {
"username": {"type": "string"},
"email": {"type": "string"},
"dashboard_sections": {"type": "array", "items": {"type": "string"}},
"credits_remaining": {"type": "number"}
},
"required": ["username", "dashboard_sections"]
}
},
"ai_extraction": true
}'
```
```python Python theme={null}
import os
from dotenv import load_dotenv
from scrapegraph_py import Client
from scrapegraph_py.logger import sgai_logger
load_dotenv()
sgai_logger.set_logging(level="INFO")
api_key = os.getenv("SGAI_API_KEY")
client = Client(api_key=api_key)
# Basic example: login and scrape without AI
response = client.agenticscraper(
url="https://scrapegraphai.com/dashboard",
use_session=True,
steps=[
"Type email@gmail.com in email input box",
"Type test-password@123 in password inputbox",
"click on login"
],
ai_extraction=False
)
print(response)
# Example with AI extraction and schema
output_schema = {
"dashboard_info": {
"type": "object",
"properties": {
"username": {"type": "string"},
"email": {"type": "string"},
"dashboard_sections": {"type": "array", "items": {"type": "string"}},
"credits_remaining": {"type": "number"}
},
"required": ["username", "dashboard_sections"]
}
}
ai_response = client.agenticscraper(
url="https://scrapegraphai.com/dashboard",
use_session=True,
steps=[
"Type email@gmail.com in email input box",
"Type test-password@123 in password inputbox",
"click on login",
"wait for dashboard to load completely"
],
user_prompt="Extract user info, dashboard sections, and remaining credits",
output_schema=output_schema,
ai_extraction=True
)
print(ai_response)
client.close()
```
```bash CLI theme={null}
# Basic scraping without AI extraction
just-scrape agentic-scraper https://scrapegraphai.com/dashboard \
-s "Type email@gmail.com in email input box,Type test-password@123 in password inputbox,Click login" \
--use-session
# With AI extraction
just-scrape agentic-scraper https://scrapegraphai.com/dashboard \
-s "Type email@gmail.com in email input box,Type test-password@123 in password inputbox,Click login,wait for dashboard to load" \
--ai-extraction -p "Extract user info, dashboard sections, and remaining credits" \
--use-session
```
### Main Parameters
| Parameter | Type | Required | Description |
| -------------- | ------ | -------- | -------------------------------------------------- |
| apiKey | string | Yes | Your ScrapeGraph API Key |
| url | string | Yes | Target page URL |
| use\_session | bool | No | Keep browser session (useful for login) |
| steps | array | Yes | Sequence of actions to perform (click, type, etc.) |
| user\_prompt | string | AI only | Prompt for AI extraction |
| output\_schema | object | AI only | JSON schema for structured output |
| ai\_extraction | bool | No | true = AI extraction, false = raw content only |
Get your API key from the [dashboard](https://scrapegraphai.com/dashboard)
## Use Cases
* Login and scraping of private dashboards
* Automation of forms and data collection
* Extraction of structured data from complex portals
* Archiving content after dynamic interactions
* End-to-end testing of web flows
## Best Practices
* Carefully define the `steps` sequence to simulate user actions
* Use `ai_extraction: false` for backup/archiving, `true` for structured data
* Provide a detailed `output_schema` for reliable AI results
* Handle errors and poll for request status if the response is asynchronous
## API Reference
For technical details see:
* [Start Agentic Scraper Job](/api-reference/endpoint/agenticscraper/start)
* [Get Job Status](/api-reference/endpoint/agenticscraper/get-status)
## Support & Resources
Comprehensive guides and tutorials
Detailed API documentation
Join our Discord community
Open-source projects
Official website
Get your API key and start using Agentic Scraper now!