AI-powered web scraping for any website
Parameter | Type | Required | Description |
---|---|---|---|
apiKey | string | Yes | The ScrapeGraph API Key. |
websiteUrl | string | Yes | The URL of the webpage that needs to be scraped. |
prompt | string | Yes | A textual description of what you want to achieve. |
schema | object | No | The Pydantic or Zod object that describes the structure and format of the response. |
Example Response
request_id
: Unique identifier for tracking your requeststatus
: Current status of the extraction (“completed”, “running”, “failed”)result
: The extracted data in structured JSON formaterror
: Error message (if any occurred during extraction)Using Your Own HTML
website_url
and website_html
are provided, website_html
takes precedence and will be used for extraction.Parameter | Type | Required | Description |
---|---|---|---|
apiKey | string | Yes | The ScrapeGraph API Key. |
requestId | string | Yes | The request ID associated with the output of a previous smartScraper request. |
AsyncClient
:
Parameter | Type | Required | Description |
---|---|---|---|
number_of_scrolls | number | No | Number of times to scroll down to load more content (default: 0) |
Header | Description |
---|---|
SGAI-APIKEY | Your API authentication key |
Content-Type | application/json |
Field | Type | Required | Description |
---|---|---|---|
website_url | string | Yes* | URL to scrape (*either this or website_html required) |
website_html | string | No | Raw HTML content to process |
user_prompt | string | Yes | Instructions for data extraction |
output_schema | object | No | Pydantic or Zod schema for response validation |