MCP WebScout

A Model Context Protocol (MCP) server providing web search (DuckDuckGo) and intelligent content extraction with LLM-powered analysis.

Features

• search: Search the web using DuckDuckGo
• fetch: Advanced web fetching with Crawl4AI and LLM extraction

System Requirements

Python Dependencies (14 packages)

Quick Start

Get started in 5 steps:

1. Clone and Setup Environment

git clone <repository>
cd mcp-webscout
python -m venv .venv

On Windows:

.venv\Scripts\activate

On macOS/Linux:

source .venv/bin/activate

2. Install Dependencies

pip install -e ".[dev]"

3. Install Playwright Browsers

playwright install chromium

4. Configure Environment Variables

cp .env.example .env

Edit .env and add your configuration:

# Required for LLM extraction
DEEPSEEK_API_KEY=sk-your-actual-key-here

# Required for mainland China users
PROXY_URL=http://127.0.0.1:7890
USE_PROXY=true

5. Verify Installation

# Run tests
pytest tests/ -v

# Test the server
python -m mcp_webscout --help

Detailed Configuration

For detailed environment setup instructions, see ENV_SETUP.md.

Usage

As a Command

mcp-webscout

As a Python Module

python -m mcp_webscout

With Claude Desktop

Add to your claude_desktop_config.json:

Basic Configuration

{
  "mcpServers": {
    "webscout": {
      "command": "mcp-webscout"
    }
  }
}

With Environment Variables (Recommended)

{
  "mcpServers": {
    "webscout": {
      "command": "mcp-webscout",
      "env": {
        "DEEPSEEK_API_KEY": "sk-your-key-here",
        "PROXY_URL": "http://127.0.0.1:7890",
        "USE_PROXY": "true",
        "DEFAULT_MAX_LENGTH": "5000",
        "PYTHONUTF8": "1"
      }
    }
  }
}

Windows Configuration

{
  "mcpServers": {
    "webscout": {
      "command": "python",
      "args": ["-m", "mcp_webscout"],
      "env": {
        "DEEPSEEK_API_KEY": "sk-your-key-here",
        "PROXY_URL": "http://127.0.0.1:7890",
        "USE_PROXY": "true",
        "PYTHONUTF8": "1"
      }
    }
  }
}

Tools

search

Search the web using DuckDuckGo.

Parameters:

Returns:

Formatted search results with titles, URLs, and snippets.

Example:

{
  "query": "Python programming",
  "max_results": 3
}

fetch

Advanced web fetching with Crawl4AI and LLM extraction.

Parameters:

Returns:

Fetched and optionally extracted content.