Proxy & Fetch Configuration

In v2, all proxy and fetch behaviour is controlled through the FetchConfig object. You can set the proxy strategy (mode), country-based geotargeting (country), wait times, scrolling, custom headers, cookies, and more. FetchConfig is accepted by sgai.extract(), sgai.scrape(), sgai.search(), and sgai.crawl.start().

Choosing a fetch mode

The mode parameter controls how pages are retrieved:

Mode	Description
`auto`	Automatically selects the best strategy (default)
`fast`	Direct HTTP fetch, no JS rendering — fastest option
`js`	Headless browser for JavaScript-heavy pages

Set stealth: true alongside any mode to enable residential proxy with anti-bot headers.

Examples

Geo-targeted content

Access content from a specific country using the country parameter:

from scrapegraph_py import ScrapeGraphAI, FetchConfig

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract the main content",
    url="https://example.com",
    fetch_config=FetchConfig(country="de"),  # Route through Germany
)

import { ScrapeGraphAI } from 'scrapegraph-js';

const sgai = ScrapeGraphAI();

const res = await sgai.extract({
  url: 'https://example.com',
  prompt: 'Extract the main content',
  fetchConfig: { country: 'de' },
});

Stealth mode for protected sites

Use stealth mode to bypass anti-bot protections:

from scrapegraph_py import ScrapeGraphAI, FetchConfig, MarkdownFormatConfig

sgai = ScrapeGraphAI()

res = sgai.scrape(
    "https://protected-site.com",
    formats=[MarkdownFormatConfig()],
    fetch_config=FetchConfig(
        mode="js",
        stealth=True,
        wait=3000,
        scrolls=3,
        country="us",
    ),
)

import { ScrapeGraphAI } from 'scrapegraph-js';

const sgai = ScrapeGraphAI();

const res = await sgai.scrape({
  url: 'https://protected-site.com',
  formats: [{ type: 'markdown' }],
  fetchConfig: {
    mode: 'js',
    stealth: true,
    wait: 3000,
    scrolls: 3,
    country: 'us',
  },
});

Custom headers and cookies

Pass custom HTTP headers or cookies with your requests:

from scrapegraph_py import ScrapeGraphAI, FetchConfig

sgai = ScrapeGraphAI()

res = sgai.extract(
    "Extract product details",
    url="https://example.com",
    fetch_config=FetchConfig(
        headers={"Accept-Language": "en-US"},
        cookies={"session": "abc123"},
    ),
)

import { ScrapeGraphAI } from 'scrapegraph-js';

const sgai = ScrapeGraphAI();

const res = await sgai.extract({
  url: 'https://example.com',
  prompt: 'Extract product details',
  fetchConfig: {
    headers: { 'Accept-Language': 'en-US' },
    cookies: { session: 'abc123' },
  },
});

Tips

Start with mode: "auto" and only switch to a specific mode if you need to.
Set stealth: true for sites with strong anti-bot protections (combine with mode: "js" for dynamic sites).
Add wait time for pages that load content dynamically after the initial render.
Use scrolls to trigger lazy-loaded content on infinite-scroll pages.
The country parameter doesn’t affect pricing — credits are charged the same regardless of proxy location.

Knowledge Base

Scraping Tools

CLI

Troubleshooting

Scraping Guides

Account & Credits

Proxy & Fetch Configuration

Choosing a fetch mode

Examples

Geo-targeted content

Stealth mode for protected sites

Custom headers and cookies

Tips

​Choosing a fetch mode

​Examples

​Geo-targeted content

​Stealth mode for protected sites

​Custom headers and cookies

​Tips

Choosing a fetch mode

Examples

Geo-targeted content

Stealth mode for protected sites

Custom headers and cookies

Tips