Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt

Use this file to discover all available pages before exploring further.

just-scrape can be installed as a skill for AI coding agents via Vercel’s skills.sh. This lets agents like Claude Code, Cursor, and others call ScrapeGraphAI commands directly during a coding session.

​
Install the skill

bunx skills add https://github.com/ScrapeGraphAI/just-scrape
Browse the skill: skills.sh/scrapegraphai/just-scrape/just-scrape

​
What this enables

Once installed, your coding agent can:
  • Extract structured data from any website using AI
  • Convert documentation pages to markdown for context
  • Search the web and extract structured results
  • Crawl multiple pages and collect data
  • Check your credit balance mid-session
  • Browse request history

​
How agents use it

Agents call just-scrape in --json mode for clean, token-efficient output: 
just-scrape extract https://api.example.com/docs \
  -p "Extract all endpoint names, methods, and descriptions" \
  --json

just-scrape search "latest release notes for react-query" \
  --num-results 3 --json

​
Using with Claude Code

Claude Code is Anthropic’s agentic coding tool that runs directly in your terminal. Since Claude Code can execute shell commands, it works seamlessly with just-scrape.

​
Setup

1

Install just-scrape globally

npm install -g just-scrape
2

Set your API key

Add the key to your shell profile (~/.zshrc, ~/.bashrc, etc.) so Claude Code inherits it automatically:
export SGAI_API_KEY="sgai-..."
3

Start Claude Code and scrape

Launch Claude Code and ask it to scrape any website:
claude
> Scrape https://news.ycombinator.com and extract the top 10 stories with titles and URLs using just-scrape

​
Add just-scrape to CLAUDE.md

To make Claude Code aware of just-scrape in every session, add instructions to the CLAUDE.md file in your project root (or ~/.claude/CLAUDE.md for global access):
CLAUDE.md
## Web Scraping

This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
The API key is set via the SGAI_API_KEY environment variable.

Available commands (always use --json flag):
- `just-scrape extract <url> -p <prompt> --json` — AI extraction from a URL
- `just-scrape search <query> --json` — search the web and extract data
- `just-scrape scrape <url> --json` — get page content (markdown is default; also html, screenshot, branding, links, images, summary, json)
- `just-scrape crawl <url> --json` — crawl multiple pages
- `just-scrape credits --json` — check credit balance and per-job quotas
- `just-scrape validate --json` — verify the API key

Use --schema to enforce a JSON schema on the output.
Use -m js --stealth for sites with anti-bot protection (fetch modes: auto, fast, js).
Use -f to pick scrape format(s), e.g. -f markdown,links,images for multi-format.
Use --country and --time-range with search for geo/time filtering.

​
Example prompts for Claude Code

Once configured, you can ask Claude Code things like: 
> Scrape the pricing page at https://example.com/pricing and create a comparison table in pricing.md

> Search for "best practices for REST API pagination" and summarize the top 5 results

> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md

> Crawl https://competitor.com/blog and extract all article titles, dates, and summaries into a JSON file
Claude Code will run the appropriate just-scrape command, parse the JSON output, and use the data to complete your task.

​
Claude Code + just-scrape in CI

You can also use Claude Code with just-scrape in non-interactive mode for automated workflows: 
claude -p "Use just-scrape to scrape https://example.com/changelog \
  and extract the latest 5 releases with version numbers, dates, and highlights. \
  Save the result as CHANGELOG_SUMMARY.md"

​
Tips

  • Set SGAI_API_KEY in your shell profile so the skill picks it up automatically.
  • Always use --json — agents don’t need spinners or banners.
  • Pass --schema with a JSON schema to get typed, predictable output:
just-scrape extract https://example.com \
  -p "Extract company info" \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"founded":{"type":"number"}}}' \
  --json

​
MCP Server alternative

For a deeper integration in Cursor or Claude Desktop, use the MCP Server instead, which exposes ScrapeGraphAI as native tools available to the AI model.