> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scrapegraphai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Agent Skill

> Give AI coding agents direct access to web scraping through just-scrape

`just-scrape` can be installed as a **skill** for AI coding agents via [Vercel's skills.sh](https://skills.sh). This lets agents like Claude Code, Cursor, and others call ScrapeGraphAI commands directly during a coding session.

## Install the skill

```bash theme={null}
bunx skills add https://github.com/ScrapeGraphAI/just-scrape
```

Browse the skill: [skills.sh/scrapegraphai/just-scrape/just-scrape](https://skills.sh/scrapegraphai/just-scrape/just-scrape)

## What this enables

Once installed, your coding agent can:

* Extract structured data from any website using AI
* Convert documentation pages to markdown for context
* Search the web and extract structured results
* Crawl multiple pages and collect data
* Check your credit balance mid-session
* Browse request history

## How agents use it

Agents call `just-scrape` in `--json` mode for clean, token-efficient output:

```bash theme={null}
just-scrape extract https://api.example.com/docs \
  -p "Extract all endpoint names, methods, and descriptions" \
  --json
```

```bash theme={null}
just-scrape search "latest release notes for react-query" \
  --num-results 3 --json
```

## Using with Claude Code

[Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) is Anthropic's agentic coding tool that runs directly in your terminal. Since Claude Code can execute shell commands, it works seamlessly with `just-scrape`.

### Setup

<Steps>
  <Step title="Install just-scrape globally">
    ```bash theme={null}
    npm install -g just-scrape
    ```
  </Step>

  <Step title="Set your API key">
    Add the key to your shell profile (`~/.zshrc`, `~/.bashrc`, etc.) so Claude Code inherits it automatically:

    ```bash theme={null}
    export SGAI_API_KEY="sgai-..."
    ```
  </Step>

  <Step title="Start Claude Code and scrape">
    Launch Claude Code and ask it to scrape any website:

    ```
    claude
    > Scrape https://news.ycombinator.com and extract the top 10 stories with titles and URLs using just-scrape
    ```
  </Step>
</Steps>

### Add just-scrape to CLAUDE.md

To make Claude Code aware of `just-scrape` in every session, add instructions to the `CLAUDE.md` file in your project root (or `~/.claude/CLAUDE.md` for global access):

```markdown CLAUDE.md theme={null}
## Web Scraping

This project uses `just-scrape` (ScrapeGraph AI CLI) for web scraping.
The API key is set via the SGAI_API_KEY environment variable.

Available commands (always use --json flag):
- `just-scrape extract <url> -p <prompt> --json` — AI extraction from a URL
- `just-scrape search <query> --json` — search the web and extract data
- `just-scrape scrape <url> --json` — get page content (markdown is default; also html, screenshot, branding, links, images, summary, json)
- `just-scrape crawl <url> --json` — crawl multiple pages
- `just-scrape credits --json` — check credit balance and per-job quotas
- `just-scrape validate --json` — verify the API key

Use --schema to enforce a JSON schema on the output.
Use -m js --stealth for sites with anti-bot protection (fetch modes: auto, fast, js).
Use -f to pick scrape format(s), e.g. -f markdown,links,images for multi-format.
Use --country and --time-range with search for geo/time filtering.
```

### Example prompts for Claude Code

Once configured, you can ask Claude Code things like:

```
> Scrape the pricing page at https://example.com/pricing and create a comparison table in pricing.md

> Search for "best practices for REST API pagination" and summarize the top 5 results

> Convert https://docs.example.com/api/authentication to markdown and save it as docs/auth.md

> Crawl https://competitor.com/blog and extract all article titles, dates, and summaries into a JSON file
```

Claude Code will run the appropriate `just-scrape` command, parse the JSON output, and use the data to complete your task.

### Claude Code + just-scrape in CI

You can also use Claude Code with `just-scrape` in non-interactive mode for automated workflows:

```bash theme={null}
claude -p "Use just-scrape to scrape https://example.com/changelog \
  and extract the latest 5 releases with version numbers, dates, and highlights. \
  Save the result as CHANGELOG_SUMMARY.md"
```

## Tips

* Set `SGAI_API_KEY` in your shell profile so the skill picks it up automatically.
* Always use `--json` — agents don't need spinners or banners.
* Pass `--schema` with a JSON schema to get typed, predictable output:

```bash theme={null}
just-scrape extract https://example.com \
  -p "Extract company info" \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"founded":{"type":"number"}}}' \
  --json
```

## MCP Server alternative

For a deeper integration in Cursor or Claude Desktop, use the [MCP Server](/services/mcp-server/introduction) instead, which exposes ScrapeGraphAI as native tools available to the AI model.