Agentic Scraper

​

Overview

• Navigate websites
• Perform sequences of actions (login, clicks, form filling, scrolling, etc.)
• Extract the resulting content as markdown
• Optionally use AI to extract structured data according to a schema

​

Difference: With vs Without AI Extraction

• Without AI Extraction (ai_extraction: false): You get the page content (after actions) as markdown/raw. Ideal for archiving, backup, or manual analysis.
• With AI Extraction (ai_extraction: true): AI extracts structured data according to a provided schema (e.g., JSON with specific fields). Perfect for automation, integration, and data analysis.

​

Getting Started

​

Quick Start

import { agenticScraper } from 'scrapegraph-js';
import 'dotenv/config';

const apiKey = process.env.SGAI_APIKEY;
const url = 'https://dashboard.scrapegraphai.com/';
const steps = [
  'Type email@gmail.com in email input box',
  'Type test-password@123 in password inputbox',
  'click on login'
];

try {
  // Basic scraping without AI extraction
  const response = await agenticScraper(apiKey, url, steps, true, null, null, false);
  console.log(response);
} catch (error) {
  console.error(error);
}

// With AI extraction
const schema = {
  dashboard_info: {
    type: "object",
    properties: {
      username: { type: "string" },
      email: { type: "string" },
      dashboard_sections: { type: "array", items: { type: "string" } },
      credits_remaining: { type: "number" }
    },
    required: ["username", "dashboard_sections"]
  }
};

try {
  const aiResponse = await agenticScraper(
    apiKey,
    url,
    [...steps, 'wait for dashboard to load completely'],
    true,
    'Extract user info, dashboard sections, and remaining credits',
    schema,
    true
  );
  console.log(aiResponse);
} catch (error) {
  console.error(error);
}

​

Main Parameters

​

Use Cases

• Login and scraping of private dashboards
• Automation of forms and data collection
• Extraction of structured data from complex portals
• Archiving content after dynamic interactions
• End-to-end testing of web flows

​

Best Practices

• Carefully define the steps sequence to simulate user actions
• Use ai_extraction: false for backup/archiving, true for structured data
• Provide a detailed output_schema for reliable AI results
• Handle errors and poll for request status if the response is asynchronous

​

API Reference

• Start Agentic Scraper Job
• Get Job Status

​

Support & Resources