Skip to main content

What is rate limiting?

Rate limiting restricts the number of API requests you can make within a given time window. ScrapeGraphAI enforces limits to ensure fair usage and stable performance for all users.

Rate limit response

When you exceed the rate limit, the API returns an HTTP 429 Too Many Requests response. In the v2 SDK this surfaces as res.status === "error" with res.error describing the failure — no exception is raised.

Limits by plan

Check the dashboard for up-to-date limits for your current plan.

How to handle rate limits in code

Python — with exponential backoff

JavaScript — with retry

Tips to avoid hitting rate limits

  • Batch requests — process URLs in batches with a small delay between each batch rather than sending them all at once.
  • Cache results — if you are scraping the same URLs repeatedly, store the results and only re-scrape when the data needs to be fresh.
  • Use crawl.start for multi-page jobs — one crawl job counts as one concurrent job, not one per page.
  • Upgrade your plan — if your use case requires higher throughput, consider upgrading to a plan with higher limits.