Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt

Use this file to discover all available pages before exploring further.

In v2, all proxy and fetch behaviour is controlled through the FetchConfig object. You can set the proxy strategy (mode), country-based geotargeting (country), wait times, scrolling, custom headers, cookies, and more. FetchConfig is accepted by sgai.extract(), sgai.scrape(), sgai.search(), and sgai.crawl.start().

​
Choosing a fetch mode

The mode parameter controls how pages are retrieved:
ModeDescription
autoAutomatically selects the best strategy (default)
fastDirect HTTP fetch, no JS rendering — fastest option
jsHeadless browser for JavaScript-heavy pages
Set stealth: true alongside any mode to enable residential proxy with anti-bot headers.

​
Examples

​
Geo-targeted content

Access content from a specific country using the country parameter: 
from scrapegraph_py import ScrapeGraphAI, FetchConfig

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract the main content",
    url="https://example.com",
    fetch_config=FetchConfig(country="de"),  # Route through Germany
)

​
Stealth mode for protected sites

Use stealth mode to bypass anti-bot protections: 
from scrapegraph_py import ScrapeGraphAI, FetchConfig, MarkdownFormatConfig

sgai = ScrapeGraphAI()

res = sgai.scrape(
    "https://protected-site.com",
    formats=[MarkdownFormatConfig()],
    fetch_config=FetchConfig(
        mode="js",
        stealth=True,
        wait=3000,
        scrolls=3,
        country="us",
    ),
)

​
Custom headers and cookies

Pass custom HTTP headers or cookies with your requests: 
from scrapegraph_py import ScrapeGraphAI, FetchConfig

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract product details",
    url="https://example.com",
    fetch_config=FetchConfig(
        headers={"Accept-Language": "en-US"},
        cookies={"session": "abc123"},
    ),
)

​
Tips

  • Start with mode: "auto" and only switch to a specific mode if you need to.
  • Set stealth: true for sites with strong anti-bot protections (combine with mode: "js" for dynamic sites).
  • Add wait time for pages that load content dynamically after the initial render.
  • Use scrolls to trigger lazy-loaded content on infinite-scroll pages.
  • The country parameter doesn’t affect pricing — credits are charged the same regardless of proxy location.