Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt

Use this file to discover all available pages before exploring further.

​
What is rate limiting?

Rate limiting restricts the number of API requests you can make within a given time window. ScrapeGraphAI enforces limits to ensure fair usage and stable performance for all users.

​
Rate limit response

When you exceed the rate limit, the API returns an HTTP 429 Too Many Requests response. In the v2 SDK this surfaces as res.status === "error" with res.error describing the failure — no exception is raised. 
{
  "error": "rate_limit_exceeded",
  "message": "Too many requests. Please slow down and retry after a few seconds."
}

​
Limits by plan

PlanRequests per minuteConcurrent jobs
Free51
Starter305
Pro10020
EnterpriseCustomCustom
Check the dashboard for up-to-date limits for your current plan.

​
How to handle rate limits in code

​
Python — with exponential backoff

import time
from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()

def extract_with_retry(url: str, prompt: str, max_retries: int = 3):
    for attempt in range(max_retries):
        res = sgai.extract(prompt, url=url)
        if res.status == "success":
            return res
        if "rate_limit" not in (res.error or "").lower():
            return res  # non-rate-limit error — don't retry
        wait = 2 ** attempt  # 1s, 2s, 4s
        print(f"Rate limited. Retrying in {wait}s...")
        time.sleep(wait)
    raise RuntimeError("Max retries exceeded")

​
JavaScript — with retry

import { ScrapeGraphAI } from "scrapegraph-js";

const sgai = ScrapeGraphAI();

async function extractWithRetry(url, prompt, retries = 3) {
  for (let i = 0; i < retries; i++) {
    const res = await sgai.extract({ url, prompt });
    if (res.status === "success") return res;
    if (!/rate.?limit/i.test(res.error ?? "")) return res;
    const wait = Math.pow(2, i) * 1000;
    console.log(`Rate limited. Retrying in ${wait}ms...`);
    await new Promise((r) => setTimeout(r, wait));
  }
  throw new Error("Max retries exceeded");
}

​
Tips to avoid hitting rate limits

  • Batch requests — process URLs in batches with a small delay between each batch rather than sending them all at once.
  • Cache results — if you are scraping the same URLs repeatedly, store the results and only re-scrape when the data needs to be fresh.
  • Use crawl.start for multi-page jobs — one crawl job counts as one concurrent job, not one per page.
  • Upgrade your plan — if your use case requires higher throughput, consider upgrading to a plan with higher limits.