AI Agent Skill

just-scrape can be installed as a skill for AI coding agents via Vercel’s skills.sh. This lets agents like Claude Code, Cursor, and others call ScrapeGraphAI commands directly during a coding session.

Install the skill

bunx skills add https://github.com/ScrapeGraphAI/just-scrape

Browse the skill: skills.sh/scrapegraphai/just-scrape/just-scrape

What this enables

Once installed, your coding agent can:

Extract structured data from any website using AI
Convert documentation pages to markdown for context
Search the web and extract structured results
Crawl multiple pages and collect data
Check your credit balance mid-session
Browse request history

How agents use it

Agents call just-scrape in --json mode for clean, token-efficient output:

just-scrape extract https://api.example.com/docs \
  -p "Extract all endpoint names, methods, and descriptions" \
  --json

just-scrape search "latest release notes for react-query" \
  --num-results 3 --json

Using with Claude Code

Claude Code is Anthropic’s agentic coding tool that runs directly in your terminal. Since Claude Code can execute shell commands, it works seamlessly with just-scrape.

Setup

Install just-scrape globally

npm install -g just-scrape

Set your API key

Add the key to your shell profile (~/.zshrc, ~/.bashrc, etc.) so Claude Code inherits it automatically:

export SGAI_API_KEY="sgai-..."

Start Claude Code and scrape

Launch Claude Code and ask it to scrape any website:

claude
> Scrape https://news.ycombinator.com and extract the top 10 stories with titles and URLs using just-scrape

Add just-scrape to CLAUDE.md

To make Claude Code aware of just-scrape in every session, add instructions to the CLAUDE.md file in your project root (or ~/.claude/CLAUDE.md for global access):

CLAUDE.md

## Web Scraping

This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
The API key is set via the SGAI_API_KEY environment variable.

Available commands (always use --json flag):
- `just-scrape extract <url> -p <prompt> --json` — AI extraction from a URL
- `just-scrape search <query> --json` — search the web and extract data
- `just-scrape scrape <url> --json` — get page content (markdown is default; also html, screenshot, branding, links, images, summary, json)
- `just-scrape crawl <url> --json` — crawl multiple pages
- `just-scrape credits --json` — check credit balance and per-job quotas
- `just-scrape validate --json` — verify the API key

Use --schema to enforce a JSON schema on the output.
Use -m js --stealth for sites with anti-bot protection (fetch modes: auto, fast, js).
Use -f to pick scrape format(s), e.g. -f markdown,links,images for multi-format.
Use --country and --time-range with search for geo/time filtering.

Example prompts for Claude Code

Once configured, you can ask Claude Code things like:

> Scrape the pricing page at https://example.com/pricing and create a comparison table in pricing.md

> Search for "best practices for REST API pagination" and summarize the top 5 results

> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md

> Crawl https://competitor.com/blog and extract all article titles, dates, and summaries into a JSON file

Claude Code will run the appropriate just-scrape command, parse the JSON output, and use the data to complete your task.

Claude Code + just-scrape in CI

You can also use Claude Code with just-scrape in non-interactive mode for automated workflows:

claude -p "Use just-scrape to scrape https://example.com/changelog \
  and extract the latest 5 releases with version numbers, dates, and highlights. \
  Save the result as CHANGELOG_SUMMARY.md"

Tips

Set SGAI_API_KEY in your shell profile so the skill picks it up automatically.
Always use --json — agents don’t need spinners or banners.
Pass --schema with a JSON schema to get typed, predictable output:

just-scrape extract https://example.com \
  -p "Extract company info" \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"founded":{"type":"number"}}}' \
  --json

MCP Server alternative

For a deeper integration in Cursor or Claude Desktop, use the MCP Server instead, which exposes ScrapeGraphAI as native tools available to the AI model.

Get Started

Services

Official SDKs

LLM SDKs

Frameworks

Personal AI Agents

No-code

Contribute

Install the skill

What this enables

How agents use it

Using with Claude Code

Setup

Add just-scrape to CLAUDE.md

Example prompts for Claude Code

Claude Code + just-scrape in CI

Tips

MCP Server alternative

​Install the skill

​What this enables

​How agents use it

​Using with Claude Code

​Setup

​Add just-scrape to CLAUDE.md

​Example prompts for Claude Code

​Claude Code + just-scrape in CI

​Tips

​MCP Server alternative

Install the skill

What this enables

How agents use it

Using with Claude Code

Setup

Add just-scrape to CLAUDE.md

Example prompts for Claude Code

Claude Code + just-scrape in CI

Tips

MCP Server alternative