Skip to main content
Sitemap Service

​
Overview

Sitemap is our service that extracts all URLs from a website’s sitemap.xml file automatically. The API discovers the sitemap from robots.txt, common locations like /sitemap.xml, or sitemap index files—perfect for discovering pages for bulk scraping, content inventory, or combining with other endpoints.
Try Sitemap instantly in our interactive playground - no coding required!

​
Getting Started

​
Quick Start

from scrapegraph_py import Client

# Initialize the client
client = Client(api_key="YOUR_API_KEY")

# Sitemap request
response = client.sitemap(
    website_url="https://scrapegraphai.com"
)

print("Result:", response)

​
Parameters

ParameterTypeRequiredDescription
apiKeystringYesThe ScrapeGraph API Key.
websiteUrlstringYesThe URL of the website to extract the sitemap from. The API will automatically locate the sitemap.xml file.
headersobjectNoOptional headers (user agent, cookies, etc.).
mockbooleanNoEnable mock mode for testing. Default: false
stealthbooleanNoEnable stealth mode for anti-bot protection. Adds +4 credits. Default: false
Get your API key from the dashboard
{
  "request_id": "65401e0d-8cd6-4d6a-88f6-e21255d1c06a",
  "status": "completed",
  "website_url": "https://scrapegraphai.com",
  "urls": [
    "https://scrapegraphai.com/",
    "https://scrapegraphai.com/about",
    "https://scrapegraphai.com/blog",
    "https://scrapegraphai.com/docs",
    "https://scrapegraphai.com/pricing",
    "https://scrapegraphai.com/contact"
  ],
  "error": ""
}
The response includes:
  • request_id: Unique identifier for tracking your request
  • status: Current status of the extraction
  • website_url: The website that was processed
  • urls: Array of all URLs found in the sitemap
  • error: Error message (if any)

​
Key Features

Automatic Discovery

Finds sitemap from robots.txt or common locations

Sitemap Index Support

Handles sitemap index files with multiple sitemaps

Fast Extraction

Retrieves all URLs without scraping each page

Integration Ready

Combine with SmartScraper for bulk operations

​
Use Cases

  • Discover all pages on a website for bulk scraping
  • Build content inventory from a website
  • Monitor website structure changes
  • Combine with SmartScraper to scrape multiple pages
  • Create site maps for SEO analysis

​
API Reference

For full request/response details and async status polling, see the Sitemap API reference.