MCP HTTP Fetcher Server

A Model Context Protocol (MCP) server that fetches web pages and converts them to Markdown format. This server supports both SSE (Server-Sent Events) and Stdio protocols, making it suitable for web deployments, Kubernetes environments, desktop clients, and sidecar patterns.

Features

• ✅ Dual Protocol Support: SSE (default) and Stdio protocols
• ✅ HTTP/HTTPS URL fetching: Download content from any web URL
• ✅ HTML to Markdown conversion: Clean conversion using html2text
• ✅ MCP Protocol compliance: Follows MCP server standards
• ✅ Kubernetes ready: SSE protocol with health checks and graceful scaling
• ✅ Sidecar friendly: Stdio protocol for process-to-process communication
• ✅ Error handling: Robust error handling for network issues
• ✅ Docker support: Ready-to-run Docker container
• ✅ Async operation: Non-blocking HTTP requests
• ✅ Modular architecture: Clean separation of concerns for maintainability

Quick Start

SSE Protocol (Default - Web/Kubernetes Deployments)

# Start with SSE protocol (default)
python app/server.py

# Custom host/port for Kubernetes
python app/server.py --protocol sse --host 0.0.0.0 --port 8000

Stdio Protocol (Desktop Clients/Sidecars)

# Start with stdio protocol
python app/server.py --protocol stdio

Protocol Overview

For detailed protocol documentation, see docs/PROTOCOLS.md.

Project Structure

The project follows Python best practices with a clean, modular architecture:

mcp_fetcher_http/
├── app/                    # Main application package
│   ├── core/              # Core business logic
│   │   ├── fetcher.py     # URL fetching functionality
│   │   └── converter.py   # HTML to Markdown conversion
│   ├── protocols/         # Protocol implementations
│   │   ├── base.py        # Abstract protocol interface
│   │   ├── stdio.py       # Standard I/O protocol implementation
│   │   └── sse.py         # Server-Sent Events protocol implementation
│   └── server.py          # New modular server entry point
├── docs/                  # Documentation
│   └── PROTOCOLS.md       # Comprehensive protocol guide
├── tests/                 # Test suite
│   ├── test_fetcher.py    # Tests for URL fetching
│   ├── test_converter.py  # Tests for HTML conversion
│   └── test_server.py     # Integration tests
├── examples/              # Example usage and demos
│   ├── demo.py           # Functionality demonstration
│   └── mcp_config_example.json
├── server.py             # Backward-compatible entry point
├── requirements.txt      # Python dependencies
└── Dockerfile           # Container configuration

Entry Points

• python app/server.py - Primary entry point with full protocol support
• python server.py - Legacy entry point (limited protocol options)
• ./run_server.sh - Convenience script with dependency management

Installation

Option 1: Direct Python Installation

1. Clone the repository:

git clone https://github.com/photonn/mcp_fetcher_http.git
cd mcp_fetcher_http

2. Install dependencies:

pip install -r requirements.txt

3. Run the server:

# SSE protocol (default - for web/Kubernetes deployments)
python app/server.py

# Stdio protocol (for desktop clients)
python app/server.py --protocol stdio

# Custom SSE configuration
python app/server.py --protocol sse --host 0.0.0.0 --port 8080

# OR use the convenience script:
./run_server.sh

Option 2: Docker Installation

1. Build the Docker image:

docker build -t mcp-fetcher-http .

2. Run the container:

# SSE protocol (default)
docker run -p 8000:8000 mcp-fetcher-http

# Stdio protocol
docker run -it mcp-fetcher-http python app/server.py --protocol stdio

Note: Docker build may require internet access to install Python packages. In restricted environments, use the direct Python installation method.

Usage

The server exposes a single MCP tool called fetch_url that accepts a URL parameter and returns the page content converted to Markdown.

MCP Client Configuration

For SSE Protocol (Web/Kubernetes deployments)

{
  "mcpServers": {
    "http-fetcher": {
      "command": "curl",
      "args": ["-N", "-H", "Accept: text/event-stream", "http://localhost:8000/sse"],
      "description": "HTTP fetcher using SSE protocol"
    }
  }
}

For Stdio Protocol (Desktop clients)

{
  "mcpServers": {
    "http-fetcher": {
      "command": "python",
      "args": ["/path/to/mcp_fetcher_http/app/server.py", "--protocol", "stdio"],
      "description": "HTTP fetcher using stdio protocol"
    }
  }
}

Command Line Options

python app/server.py --help

Available options:

• --protocol {stdio,sse} - Communication protocol (default: sse)
• --host HOST - Host to bind SSE server (default: localhost)
• --port PORT - Port for SSE server (default: 8000)
• --endpoint ENDPOINT - SSE message endpoint (default: /messages)
• --server-name NAME - Server identifier
• --server-version VERSION - Server version

Tool Schema

{
  "name": "fetch_url",
  "description": "Fetch a web page and convert it to Markdown format",
  "inputSchema": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The URL of the web page to fetch and convert to Markdown"
      }
    },
    "required": ["url"]
  }
}

Example Usage

When connected to an MCP client, you can use the tool like this:

{
  "name": "fetch_url",
  "arguments": {
    "url": "https://example.com"
  }
}

The server will return the page content converted to Markdown format.

Testing

Run the included test scripts to verify functionality:

# Run all tests (recommended)
python tests/test_server.py      # Integration tests
python tests/test_fetcher.py     # URL fetching tests  
python tests/test_converter.py   # HTML conversion tests

# Run demo to see HTML to Markdown conversion
python examples/demo.py

Testing the MCP Server

The server communicates via stdin/stdout using the MCP protocol. For basic testing:

1. Start the server: python server.py (or python app/server.py)
2. The server will wait for MCP protocol messages on stdin
3. Send a list_tools request to see available tools
4. Send a call_tool request with the fetch_url tool

For easier testing, use an MCP client like Claude Desktop or develop a simple test client.

Configuration

The server includes the following configurable aspects:

• Timeout: 30-second timeout for HTTP requests
• Content types: Accepts HTML and XML content types
• Markdown conversion: Preserves links and images in output

Error Handling

The server handles various error conditions:

• Invalid URLs
• Network connectivity issues
• HTTP error responses (4xx, 5xx)
• Content parsing errors
• Timeout conditions

Security Considerations

• The server validates URLs before processing
• Runs in a non-root Docker container
• Has reasonable timeout limits to prevent hanging
• Does not execute or process JavaScript content

Dependencies

• mcp>=1.0.0 - Model Context Protocol server framework
• aiohttp>=3.9.0 - Async HTTP client for fetching URLs
• html2text>=2020.1.16 - HTML to Markdown conversion
• typing-extensions>=4.0.0 - Type hints support

License

Licensed under the Apache License, Version 2.0. See the LICENSE file for details.