Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt

Use this file to discover all available pages before exploring further.

ScrapeGraphAI enforces rate limits to ensure reliable performance for all users. Limits vary by plan.

​
Limits overview

PlanRequests per minuteConcurrent crawlsMonitorsMonthly credits
Free1011500
Starter1003510,000
Growth5001525100,000
Pro5,00050100750,000
EnterpriseCustomCustomCustomCustom
For full pricing details, see Plans & Pricing.
Contact support for custom limits or high-volume plans.

​
What counts as a request?

Each API call to any v2 endpoint (/api/extract, /api/scrape, /api/search, /api/crawl, /api/monitor, …) counts as one request toward your rate limit. Polling a crawl or monitor status endpoint does not count toward the limit.

​
Rate limit headers

Every API response includes headers that show your current rate limit status: 
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 87
X-RateLimit-Reset: 1709123456
  • X-RateLimit-Limit — maximum requests allowed per minute
  • X-RateLimit-Remaining — requests remaining in the current window
  • X-RateLimit-Reset — Unix timestamp when the limit resets

​
Handling the 429 response

When you exceed the rate limit, the API returns HTTP 429 Too Many Requests. In the v2 SDK this surfaces as res.status === "error" — the SDK does not raise. Retry with exponential backoff: 
import time
from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()

def extract_with_backoff(url, prompt, max_retries=5):
    for i in range(max_retries):
        res = sgai.extract(prompt, url=url)
        if res.status == "success":
            return res
        if "rate_limit" not in (res.error or "").lower():
            return res  # non-rate-limit error — don't retry
        wait = 2 ** i
        print(f"Rate limited — retrying in {wait}s")
        time.sleep(wait)
    raise RuntimeError("Exceeded max retries")
See the rate limiting troubleshooting guide for a JavaScript example and more tips.

​
Increasing your limits

  • Upgrade your plan from the dashboard to get higher limits immediately.
  • Enterprise customers can request custom rate limit configurations by contacting support.