Using just-scrape as a coding agent skill

just-scrape can be installed as a skill for AI coding agents via Vercel’s skills.sh. This lets agents like Claude, Cursor, and others call ScrapeGraphAI commands directly during a coding session.

Install the skill

bunx skills add https://github.com/ScrapeGraphAI/just-scrape

Browse the skill page: skills.sh/scrapegraphai/just-scrape/just-scrape

What this enables

Once installed, your coding agent can:

Scrape a website to gather data needed for a task
Convert documentation pages to markdown for context
Search the web and extract structured results
Check your credit balance mid-session
Browse request history

How agents use it

Agents invoke the skill in --json mode so output is clean and token-efficient:

just-scrape extract https://api.example.com/docs \
  -p "Extract all endpoint names, methods, and descriptions" \
  --json

just-scrape search "latest release notes for react-query" \
  --num-results 3 --json

Using with Claude Code

Claude Code is Anthropic’s agentic coding tool that runs in your terminal. Since it can execute shell commands, it works seamlessly with just-scrape.

Setup

Install just-scrape globally: npm install -g just-scrape
Set SGAI_API_KEY in your shell profile (~/.zshrc, ~/.bashrc)
Launch Claude Code and ask it to scrape anything

Add just-scrape to CLAUDE.md

Add scraping instructions to your CLAUDE.md (project root or ~/.claude/CLAUDE.md for global):

CLAUDE.md

## Web Scraping

This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
The API key is set via the SGAI_API_KEY environment variable.

Available commands (always use --json flag):
- `just-scrape extract <url> -p <prompt> --json` — AI structured extraction from a URL
- `just-scrape search <query> --json` — search the web and extract data from results
- `just-scrape scrape <url> --json` — fetch a page (markdown default; also html, screenshot, branding, links, images, summary, json)
- `just-scrape crawl <url> --json` — crawl multiple pages (polls until done)
- `just-scrape credits --json` / `just-scrape validate --json` — balance and key health

Use --schema to enforce a JSON schema on the output.
Use --stealth (with -m js if needed) for sites with anti-bot protection.

Example prompts

> Scrape the pricing page at https://example.com/pricing and create a comparison table

> Search for "best practices for REST API pagination" and summarize the top results

> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md

Non-interactive / CI usage

claude -p "Use just-scrape to scrape https://example.com/changelog \
  and extract the latest 5 releases. Save as CHANGELOG_SUMMARY.md"

Manual setup with Cursor

If you are using Cursor without the skills.sh integration, configure just-scrape via the MCP Server for the best experience. Alternatively, add a script to your project that Cursor can call:

# .cursor/scrape.sh
#!/bin/bash
just-scrape extract "$1" -p "$2" --json

Then tell Cursor: “Run .cursor/scrape.sh <url> <prompt> to scrape a page.”

Tips

Set SGAI_API_KEY in your shell profile so the skill picks it up automatically across all agent sessions.
Use --json every time — agents don’t need spinners or banners.
Pass --schema with a JSON schema to get typed, predictable output that agents can parse reliably.

just-scrape extract https://example.com \
  -p "Extract company info" \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"founded":{"type":"number"},"employees":{"type":"string"}}}' \
  --json

Knowledge Base

Scraping Tools

CLI

Troubleshooting

Scraping Guides

Account & Credits

Using just-scrape as a coding agent skill

Install the skill

What this enables

How agents use it

Using with Claude Code

Setup

Add just-scrape to CLAUDE.md

Example prompts

Non-interactive / CI usage

Manual setup with Cursor

Tips

​Install the skill

​What this enables

​How agents use it

​Using with Claude Code

​Setup

​Add just-scrape to CLAUDE.md

​Example prompts

​Non-interactive / CI usage

​Manual setup with Cursor

​Tips

Install the skill

What this enables

How agents use it

Using with Claude Code

Setup

Add just-scrape to CLAUDE.md

Example prompts

Non-interactive / CI usage

Manual setup with Cursor

Tips