screaming-frog-mcp
MCP server that lets Claude run Screaming Frog SEO Spider headless crawls, export data, and manage crawl storage — without anyone opening the GUI.
Type a URL into Claude. Screaming Frog runs in the background. You get the data back. That's it.
Forked from bzsasson/screaming-frog-mcp v0.1.0 with bug fixes. The original had issues that made it unusable in practice — pipe deadlocks that hung crawls, false GUI detection that blocked everything after the first run, a delete command that could wipe your entire crawl database. All fixed.
What's fixed
| Bug | Fix |
|---|---|
| Pipe deadlock | stdout/stderr redirected to log files instead of PIPE. Crawls no longer hang when SF produces large output. |
| GUI detection | Uses psutil instead of ps aux. Works on Mac and Windows. Headless CLI processes no longer get mistaken for the GUI. |
| Stale crawl cleanup | SF leaves a temp crawl.seospider file inside its own app bundle when a crawl gets interrupted. Every crawl after that fails. Now auto-cleaned before each run. |
| Delete safety | delete_crawl(".") used to resolve to the root data directory and wipe everything. Fixed. |
| Export dir leak | Failed exports left temp directories on disk. Now cleaned up. |
| Input validation | Stricter character allowlists for CLI arguments and db_id. |
Requirements
- Screaming Frog SEO Spider with a paid license — headless crawls require a license
- Python 3.10+
- uv (recommended) or pip
Installation
Mac
uvx --from git+https://github.com/marykovziridze/screaming-frog-mcp screaming-frog-mcp
Add to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"screaming-frog": {
"command": "uvx",
"args": ["--from", "git+https://github.com/marykovziridze/screaming-frog-mcp", "screaming-frog-mcp"]
}
}
}
Windows
Install uv first:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Add to C:\Users\[name]\AppData\Roaming\Claude\claude_desktop_config.json:
{
"mcpServers": {
"screaming-frog": {
"command": "uvx",
"args": ["--from", "git+https://github.com/marykovziridze/screaming-frog-mcp", "screaming-frog-mcp"],
"env": {
"SF_CLI_PATH": "C:\\Program Files (x86)\\Screaming Frog SEO Spider\\ScreamingFrogSEOSpiderCli.exe"
}
}
}
}
Restart Claude Desktop after editing the config.
Tools
| Tool | What it does |
|---|---|
sf_check | Verify SF is installed and licensed |
crawl_site | Start a headless crawl |
crawl_status | Check crawl progress |
list_crawls | List saved crawls in SF's database |
export_crawl | Export crawl data as CSV |
read_crawl_data | Read and filter exported CSV data |
delete_crawl | Delete a saved crawl |
storage_summary | Show disk usage of crawl storage |
Configuration
| Variable | Default | Notes |
|---|---|---|
SF_CLI_PATH | Mac: auto-detected | Set manually on Windows or custom installs |
Known limitations
- Windows stale crawl path — auto-cleanup works on Mac. On Windows, if crawls fail after an interruption, check for a
crawl.seospiderfile in your SF install directory and delete it manually. - No crawl progress percentage — SF's headless CLI doesn't report progress mid-crawl. You know when it starts and when it finishes.
- Large sites — tested on sites up to ~160 pages. Not stress-tested on 10k+ page sites.
License
MIT — see LICENSE
Credits
Original MCP server by Boaz Sasson.