@robot-resources/scraper-mcp
MCP server for Scraper — context compression for AI agents.
What is Robot Resources?
Human Resources, but for your AI agents.
Robot Resources gives AI agents two superpowers:
- Router — Routes each LLM call to the cheapest capable model. 60-90% cost savings across OpenAI, Anthropic, and Google.
- Scraper — Compresses web pages to clean markdown. 70-80% fewer tokens per page.
Both run locally. Your API keys never leave your machine. Free, unlimited, no tiers.
Install the full suite
npx robot-resources
One command sets up everything. Learn more at robotresources.ai
About this MCP server
This package gives AI agents two tools to compress web content into token-efficient markdown via the Model Context Protocol: single-page compression and multi-page BFS crawling.
Installation
npx @robot-resources/scraper-mcp
Or install globally:
npm install -g @robot-resources/scraper-mcp
Claude Desktop Configuration
Add to your claude_desktop_config.json:
{
"mcpServers": {
"scraper": {
"command": "npx",
"args": ["-y", "@robot-resources/scraper-mcp"]
}
}
}
Tools
scraper_compress_url
Compress a single web page into markdown with 70-90% fewer tokens.
Parameters:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | yes | — | URL to compress |
mode | string | no | 'auto' | 'fast', 'stealth', 'render', or 'auto' |
timeout | number | no | 10000 | Fetch timeout in milliseconds |
maxRetries | number | no | 3 | Max retry attempts (0-10) |
Example prompt: "Compress https://docs.example.com/getting-started"
scraper_crawl_url
Crawl multiple pages from a starting URL using BFS link discovery.
Parameters:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | yes | — | Starting URL to crawl |
maxPages | number | no | 10 | Max pages to crawl (1-100) |
maxDepth | number | no | 2 | Max link depth (0-5) |
mode | string | no | 'auto' | 'fast', 'stealth', 'render', or 'auto' |
include | string[] | no | — | URL patterns to include (glob) |
exclude | string[] | no | — | URL patterns to exclude (glob) |
timeout | number | no | 10000 | Per-page timeout in milliseconds |
Example prompt: "Crawl the docs at https://docs.example.com with max 20 pages"
Fetch Modes
| Mode | How | Use when |
|---|---|---|
'fast' | Plain HTTP | Default sites, APIs, docs |
'stealth' | TLS fingerprint impersonation | Anti-bot protected sites |
'render' | Headless browser (Playwright) | JS-rendered SPAs |
'auto' | Fast → stealth fallback on 403/challenge | Unknown sites (default) |
Stealth requires impit and render requires playwright as peer dependencies of @robot-resources/scraper.
Requirements
- Node.js 18+
Related
- @robot-resources/scraper - Core compression library
- @robot-resources/router-mcp - MCP server for LLM cost optimization
- Robot Resources - Human Resources, but for your AI agents
License
MIT