Mocking & Testing

Standard Libraries

Use familiar testing tools for mocking

No Credit Usage

Test without consuming API credits

Overview

In v2, the built-in mock mode (mock=True, mock_handler, mock_responses) has been removed from the SDKs. Instead, use standard mocking libraries for your language to test ScrapeGraphAI integrations without making real API calls or consuming credits.

If you’re migrating from v1, replace Client(mock=True) with standard mocking patterns shown below.

Python SDK Testing

Using `unittest.mock`

from unittest.mock import patch, MagicMock
from scrapegraph_py import Client

def test_extract():
    client = Client(api_key="test-key")

    mock_response = {
        "data": {
            "title": "Test Page",
            "content": "This is test content"
        },
        "request_id": "test-request-123"
    }

    with patch.object(client, "extract", return_value=mock_response):
        response = client.extract(
            url="https://example.com",
            prompt="Extract title and content"
        )

        assert response["data"]["title"] == "Test Page"
        assert response["request_id"] == "test-request-123"

Using `responses` Library

Mock HTTP requests at the transport layer:

import responses
from scrapegraph_py import Client

@responses.activate
def test_extract_http():
    responses.post(
        "https://api.scrapegraphai.com/api/v2/extract",
        json={
            "data": {"title": "Mock Title"},
            "request_id": "mock-123"
        },
        status=200,
    )

    client = Client(api_key="test-key")
    response = client.extract(
        url="https://example.com",
        prompt="Extract the title"
    )

    assert response["data"]["title"] == "Mock Title"

Using `pytest` Fixtures

import pytest
from unittest.mock import MagicMock
from scrapegraph_py import Client

@pytest.fixture
def mock_client():
    client = Client(api_key="test-key")
    client.extract = MagicMock(return_value={
        "data": {"title": "Mock Title"},
        "request_id": "mock-123"
    })
    client.search = MagicMock(return_value={
        "data": {"results": []},
        "request_id": "mock-456"
    })
    client.credits = MagicMock(return_value={
        "remaining_credits": 100,
        "total_credits_used": 0
    })
    return client

def test_extract(mock_client):
    response = mock_client.extract(
        url="https://example.com",
        prompt="Extract the title"
    )
    assert response["data"]["title"] == "Mock Title"

def test_credits(mock_client):
    credits = mock_client.credits()
    assert credits["remaining_credits"] == 100

Async Testing with `aioresponses`

import pytest
import asyncio
from aioresponses import aioresponses
from scrapegraph_py import AsyncClient

@pytest.mark.asyncio
async def test_async_extract():
    with aioresponses() as mocked:
        mocked.post(
            "https://api.scrapegraphai.com/api/v2/extract",
            payload={
                "data": {"title": "Async Mock"},
                "request_id": "async-123"
            },
        )

        async with AsyncClient(api_key="test-key") as client:
            response = await client.extract(
                url="https://example.com",
                prompt="Extract data"
            )

            assert response["data"]["title"] == "Async Mock"

JavaScript SDK Testing

Using Jest / Vitest

import { describe, it, expect, vi } from "vitest";
import { extract, getCredits, search } from "scrapegraph-js";

// Mock the module
vi.mock("scrapegraph-js", () => ({
  extract: vi.fn().mockResolvedValue({
    status: "success",
    data: { raw: null, json: { title: "Mock Title" }, usage: {}, metadata: {} },
    elapsedMs: 100,
  }),
    search: vi.fn().mockResolvedValue({
      status: "success",
      data: { results: [] },
      elapsedMs: 100,
    }),
    getCredits: vi.fn().mockResolvedValue({
      status: "success",
      data: { remaining: 100, used: 50, plan: "pro" },
      elapsedMs: 50,
    }),
  };
}));

describe("ScrapeGraphAI", () => {
  it("should extract data", async () => {
    const result = await extract("test-key", {
      url: "https://example.com",
      prompt: "Extract the title",
    });
    expect(result.data?.json?.title).toBe("Mock Title");
  });

  it("should check credits", async () => {
    const result = await getCredits("test-key");
    expect(result.data?.remaining).toBe(100);
  });
});

Using MSW (Mock Service Worker)

Mock at the network level for more realistic testing:

import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";
import { extract } from "scrapegraph-js";

const server = setupServer(
  http.post("https://api.scrapegraphai.com/v2/extract", () => {
    return HttpResponse.json({
      raw: null,
      json: { title: "MSW Mock Title" },
      usage: { promptTokens: 100, completionTokens: 50 },
      metadata: { chunker: { chunks: [] } },
    });
  })
);

beforeAll(() => server.listen());
afterAll(() => server.close());
afterEach(() => server.resetHandlers());

test("extract returns mocked data", async () => {
  const result = await extract("test-key", {
    url: "https://example.com",
    prompt: "Extract the title",
  });
  expect(result.data?.json?.title).toBe("MSW Mock Title");
});

Testing with cURL

Test API endpoints directly using cURL against a local mock server or staging environment:

# Test extract endpoint
curl -X POST "https://api.scrapegraphai.com/api/v2/extract" \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "prompt": "Extract the title"
  }'

# Test credits endpoint
curl -X GET "https://api.scrapegraphai.com/api/v2/credits" \
  -H "Authorization: Bearer your-api-key"

SDK Comparison

Feature	Python	JavaScript
Mock library	`unittest.mock`, `responses`	Jest/Vitest mocks, MSW
HTTP-level mocking	`responses`, `aioresponses`	MSW (Mock Service Worker)
Async mocking	`aioresponses`, `unittest.mock`	Native async/await
Fixture support	pytest fixtures	beforeEach/afterEach

Best Practices

Mock at the client method level for unit tests (fastest, simplest)
Mock at the HTTP level for integration tests (validates request/response shapes)
Use fixtures to share mock configurations across tests
Keep mock responses realistic - match the actual API response structure
Test both success and error scenarios

Support

GitHub Issues

Report bugs or request features

Documentation

Python SDK documentation

Need help with testing? Join our Discord community for support.

Get Started

Services

Official SDKs

LLM SDKs

Frameworks

Personal AI Agents

No-code

Contribute

Standard Libraries

No Credit Usage

Overview

Python SDK Testing

Using `unittest.mock`

Using `responses` Library

Using `pytest` Fixtures

Async Testing with `aioresponses`

JavaScript SDK Testing

Using Jest / Vitest

Using MSW (Mock Service Worker)

Testing with cURL

SDK Comparison

Best Practices

Support

GitHub Issues

Documentation

Standard Libraries

No Credit Usage

​Overview

​Python SDK Testing

​Using unittest.mock

​Using responses Library

​Using pytest Fixtures

​Async Testing with aioresponses

​JavaScript SDK Testing

​Using Jest / Vitest

​Using MSW (Mock Service Worker)

​Testing with cURL

​SDK Comparison

​Best Practices

​Support

GitHub Issues

Documentation

Overview

Python SDK Testing

Using `unittest.mock`

Using `responses` Library

Using `pytest` Fixtures

Async Testing with `aioresponses`

JavaScript SDK Testing

Using Jest / Vitest

Using MSW (Mock Service Worker)

Testing with cURL

SDK Comparison

Best Practices

Support