Sitemap

Removed in API v2 — no direct replacement. For multi-page URL discovery, use Crawl with extraction_mode="links", or fetch a sitemap.xml directly via Scrape. This page is kept for reference.

Overview

Sitemap is our service that extracts all URLs from a website’s sitemap.xml file automatically. The API discovers the sitemap from robots.txt, common locations like /sitemap.xml, or sitemap index files—perfect for discovering pages for bulk scraping, content inventory, or combining with other endpoints.

Try Sitemap instantly in our interactive playground - no coding required!

Getting Started

Quick Start

from scrapegraph_py import Client

# Initialize the client
client = Client(api_key="YOUR_API_KEY")

# Sitemap request
response = client.sitemap(
    website_url="https://scrapegraphai.com"
)

print("Result:", response)

import { sitemap } from 'scrapegraph-js';

const apiKey = 'YOUR_API_KEY';

const response = await sitemap(apiKey, {
  website_url: 'https://scrapegraphai.com',
});

if (response.status === 'error') {
  console.error('Error:', response.error);
} else {
  console.log('Result:', response.data);
}

curl -X POST https://api.scrapegraphai.com/v1/sitemap \
  -H "Content-Type: application/json" \
  -H "SGAI-APIKEY: YOUR_API_KEY" \
  -d '{
  "website_url": "https://scrapegraphai.com"
}'

just-scrape sitemap https://scrapegraphai.com

Parameters

Parameter	Type	Required	Description
apiKey	string	Yes	The ScrapeGraph API Key.
website_url	string	Yes	The URL of the website to extract the sitemap from. The API will automatically locate the sitemap.xml file.
headers	object	No	Optional headers (user agent, cookies, etc.).
mock	boolean	No	Enable mock mode for testing. Default: false
stealth	boolean	No	Enable stealth mode for anti-bot protection. Adds +5 credits. Default: false

Get your API key from the dashboard

Example Response

{
  "request_id": "65401e0d-8cd6-4d6a-88f6-e21255d1c06a",
  "status": "completed",
  "website_url": "https://scrapegraphai.com",
  "urls": [
    "https://scrapegraphai.com/",
    "https://scrapegraphai.com/about",
    "https://scrapegraphai.com/blog",
    "https://scrapegraphai.com/docs",
    "https://scrapegraphai.com/pricing",
    "https://scrapegraphai.com/contact"
  ],
  "error": ""
}

The response includes:

request_id: Unique identifier for tracking your request
status: Current status of the extraction
website_url: The website that was processed
urls: Array of all URLs found in the sitemap
error: Error message (if any)

Key Features

Automatic Discovery

Finds sitemap from robots.txt or common locations

Sitemap Index Support

Handles sitemap index files with multiple sitemaps

Fast Extraction

Retrieves all URLs without scraping each page

Integration Ready

Combine with SmartScraper for bulk operations

Use Cases

Discover all pages on a website for bulk scraping
Build content inventory from a website
Monitor website structure changes
Combine with SmartScraper to scrape multiple pages
Create site maps for SEO analysis

API Reference

For full request/response details and async status polling, see the Sitemap API reference.

Get Started

Services

Official SDKs

LLM SDKs

Frameworks

Personal AI Agents

No-code

Contribute

Overview

Getting Started

Quick Start

Parameters

Key Features

Automatic Discovery

Sitemap Index Support

Fast Extraction

Integration Ready

Use Cases

API Reference

​Overview

​Getting Started

​Quick Start

​Parameters

​Key Features

Automatic Discovery

Sitemap Index Support

Fast Extraction

Integration Ready

​Use Cases

​API Reference

Overview

Getting Started

Quick Start

Parameters

Key Features

Use Cases

API Reference