Skip to main content
just-scrape can be installed as a skill for AI coding agents via Vercel’s skills.sh. This lets agents like Claude, Cursor, and others call ScrapeGraphAI commands directly during a coding session.

​
Install the skill

bunx skills add https://github.com/ScrapeGraphAI/just-scrape
Browse the skill page: skills.sh/scrapegraphai/just-scrape/just-scrape

​
What this enables

Once installed, your coding agent can:
  • Scrape a website to gather data needed for a task
  • Convert documentation pages to markdown for context
  • Search the web and extract structured results
  • Check your credit balance mid-session
  • Browse request history

​
How agents use it

Agents invoke the skill in --json mode so output is clean and token-efficient: 
just-scrape smart-scraper https://api.example.com/docs \
  -p "Extract all endpoint names, methods, and descriptions" \
  --json

just-scrape search-scraper "latest release notes for react-query" \
  --num-results 3 --json

​
Using with Claude Code

Claude Code is Anthropic’s agentic coding tool that runs in your terminal. Since it can execute shell commands, it works seamlessly with just-scrape.

​
Setup

  1. Install just-scrape globally: npm install -g just-scrape
  2. Set SGAI_API_KEY in your shell profile (~/.zshrc, ~/.bashrc)
  3. Launch Claude Code and ask it to scrape anything

​
Add just-scrape to CLAUDE.md

Add scraping instructions to your CLAUDE.md (project root or ~/.claude/CLAUDE.md for global):
CLAUDE.md
## Web Scraping

This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
The API key is set via the SGAI_API_KEY environment variable.

Available commands (always use --json flag):
- `just-scrape smart-scraper <url> -p <prompt> --json` — AI extraction from a URL
- `just-scrape search-scraper <prompt> --json` — search the web and extract data
- `just-scrape markdownify <url> --json` — convert a page to markdown
- `just-scrape crawl <url> -p <prompt> --json` — crawl multiple pages
- `just-scrape scrape <url> --json` — get raw HTML
- `just-scrape sitemap <url> --json` — get all URLs from a sitemap

Use --schema to enforce a JSON schema on the output.
Use --stealth for sites with anti-bot protection.

​
Example prompts

> Scrape the pricing page at https://example.com/pricing and create a comparison table

> Search for "best practices for REST API pagination" and summarize the top results

> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md

​
Non-interactive / CI usage

claude -p "Use just-scrape to scrape https://example.com/changelog \
  and extract the latest 5 releases. Save as CHANGELOG_SUMMARY.md"

​
Manual setup with Cursor

If you are using Cursor without the skills.sh integration, configure just-scrape via the MCP Server for the best experience. Alternatively, add a script to your project that Cursor can call: 
# .cursor/scrape.sh
#!/bin/bash
just-scrape smart-scraper "$1" -p "$2" --json
Then tell Cursor: “Run .cursor/scrape.sh <url> <prompt> to scrape a page.”

​
Tips

  • Set SGAI_API_KEY in your shell profile so the skill picks it up automatically across all agent sessions.
  • Use --json every time — agents don’t need spinners or banners.
  • Pass --schema with a JSON schema to get typed, predictable output that agents can parse reliably.
just-scrape smart-scraper https://example.com \
  -p "Extract company info" \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"founded":{"type":"number"},"employees":{"type":"string"}}}' \
  --json