> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Using just-scrape as a coding agent skill

> Give AI coding agents access to web scraping through the just-scrape skill

`just-scrape` can be installed as a **skill** for AI coding agents via [Vercel's skills.sh](https://skills.sh). This lets agents like Claude, Cursor, and others call ScrapeGraphAI commands directly during a coding session.

## Install the skill

```bash theme={null}
bunx skills add https://github.com/ScrapeGraphAI/just-scrape
```

Browse the skill page: [skills.sh/scrapegraphai/just-scrape/just-scrape](https://skills.sh/scrapegraphai/just-scrape/just-scrape)

## What this enables

Once installed, your coding agent can:

* Scrape a website to gather data needed for a task
* Convert documentation pages to markdown for context
* Search the web and extract structured results
* Check your credit balance mid-session
* Browse request history

## How agents use it

Agents invoke the skill in `--json` mode so output is clean and token-efficient:

```bash theme={null}
just-scrape extract https://api.example.com/docs \
  -p "Extract all endpoint names, methods, and descriptions" \
  --json
```

```bash theme={null}
just-scrape search "latest release notes for react-query" \
  --num-results 3 --json
```

## Using with Claude Code

[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) is Anthropic's agentic coding tool that runs in your terminal. Since it can execute shell commands, it works seamlessly with `just-scrape`.

### Setup

1. Install `just-scrape` globally: `npm install -g just-scrape`
2. Set `SGAI_API_KEY` in your shell profile (`~/.zshrc`, `~/.bashrc`)
3. Launch Claude Code and ask it to scrape anything

### Add just-scrape to CLAUDE.md

Add scraping instructions to your `CLAUDE.md` (project root or `~/.claude/CLAUDE.md` for global):

```markdown CLAUDE.md theme={null}
## Web Scraping

This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
The API key is set via the SGAI_API_KEY environment variable.

Available commands (always use --json flag):
- `just-scrape extract <url> -p <prompt> --json` — AI structured extraction from a URL
- `just-scrape search <query> --json` — search the web and extract data from results
- `just-scrape scrape <url> --json` — fetch a page (markdown default; also html, screenshot, branding, links, images, summary, json)
- `just-scrape crawl <url> --json` — crawl multiple pages (polls until done)
- `just-scrape credits --json` / `just-scrape validate --json` — balance and key health

Use --schema to enforce a JSON schema on the output.
Use --stealth (with -m js if needed) for sites with anti-bot protection.
```

### Example prompts

```
> Scrape the pricing page at https://example.com/pricing and create a comparison table

> Search for "best practices for REST API pagination" and summarize the top results

> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md
```

### Non-interactive / CI usage

```bash theme={null}
claude -p "Use just-scrape to scrape https://example.com/changelog \
  and extract the latest 5 releases. Save as CHANGELOG_SUMMARY.md"
```

## Manual setup with Cursor

If you are using Cursor without the skills.sh integration, configure `just-scrape` via the [MCP Server](/services/mcp-server/cursor) for the best experience.

Alternatively, add a script to your project that Cursor can call:

```bash theme={null}
# .cursor/scrape.sh
#!/bin/bash
just-scrape extract "$1" -p "$2" --json
```

Then tell Cursor: *"Run `.cursor/scrape.sh <url> <prompt>` to scrape a page."*

## Tips

* Set `SGAI_API_KEY` in your shell profile so the skill picks it up automatically across all agent sessions.
* Use `--json` every time — agents don't need spinners or banners.
* Pass `--schema` with a JSON schema to get typed, predictable output that agents can parse reliably.

```bash theme={null}
just-scrape extract https://example.com \
  -p "Extract company info" \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"founded":{"type":"number"},"employees":{"type":"string"}}}' \
  --json
```