📚 Marcus Local MCP Server

A Model Context Protocol (MCP) server that indexes documentation sites and local code repositories for semantic search by AI assistants.

🎯 What Is This?

This is a local MCP server that enables AI assistants (Cursor, Claude Desktop, ChatGPT) to semantically search through:

• Documentation websites - Crawled and indexed from any docs site
• Local code repositories - All text files from your projects

It uses OpenAI embeddings to create a vector database (ChromaDB) that AI assistants can query through the Model Context Protocol.

Think of it as: Giving your AI assistant instant access to searchable documentation and your entire codebase.

📋 How It Works

┌─────────────────┐
│  AI Assistant   │ (Cursor, Claude, ChatGPT, etc.)
│  (via MCP)      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   MCP Server    │ (Python - stdio)
│   main.py       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐      ┌──────────────┐
│   ChromaDB      │◄─────┤   OpenAI     │
│  (Vector Store) │      │  Embeddings  │
└────────┬────────┘      └──────────────┘
         │
         ▼
┌──────────────────────┐
│   Indexed Sources    │
│  • Documentation     │
│    - Moca Network    │
│    - Your Docs       │
│  • Repositories      │
│    - Your Codebase   │
│    - Local Projects  │
└──────────────────────┘

The Flow:

1. Index - Crawl docs OR read local repo files
2. Chunk - Split content into 800-token chunks
3. Embed - Create OpenAI embeddings (batched for speed)
4. Store - Save in ChromaDB vector database
5. Search - AI assistant queries via MCP protocol
6. Retrieve - Return relevant chunks from docs/code

🚀 How to Run It

1. Setup

# Clone repository
git clone <your-repo>
cd crawl4ai_test

# Install Node.js dependencies
npm install

# Setup Python virtual environment
python3 -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install Python dependencies
pip install -r mcp-docs-server/requirements.txt

# Install Crawl4AI
pip install -U crawl4ai
crawl4ai-setup

2. Configure

Create .env file in mcp-docs-server/:

OPENAI_API_KEY=your_openai_api_key_here
EMBEDDING_MODEL=text-embedding-3-small
DEFAULT_RESULTS=5

3. Run the Web UI

# Start Next.js server
npm run dev

# Open browser
open http://localhost:3030

4. Connect to Cursor/Claude

Add to your AI assistant config:

For Cursor (~/.cursor/mcp.json or project config):

{
  "mcpServers": {
    "marcus-mcp-server": {
      "command": "/path/to/your/venv/bin/python3",
      "args": ["/path/to/crawl4ai_test/mcp-docs-server/server/main.py"]
    }
  }
}

For Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "marcus-docs": {
      "command": "/path/to/your/venv/bin/python",
      "args": ["/path/to/crawl4ai_test/mcp-docs-server/server/main.py"]
    }
  }
}

📖 How to Use It

Adding Documentation

Via Web UI:

1. Go to http://localhost:3030
2. Click "Add New Docs"
3. Enter:
   • URL: https://docs.example.com
   • Source Name: Example Docs
   • Max Pages: 50 (or unlimited)
4. Click "Start Indexing"
5. Wait for completion

Via Command Line:

cd mcp-docs-server
source ../venv/bin/activate

python scripts/crawler.py https://docs.example.com "Example Docs" 50
python scripts/indexer_multi.py "Example Docs"

Adding Repositories

Via Web UI:

1. Go to http://localhost:3030
2. Click "Add Repository"
3. Enter:
   • Repository Path: Drag-and-drop folder OR paste path
   • Source Name: Auto-generated from folder name
4. Click "Start Indexing"
5. Watch live progress

What gets indexed:

• ✅ All text files (.js, .py, .md, .tsx, .json, .css, etc.)
• ✅ Auto-skips: node_modules, .git, venv, build, .next, etc.
• ✅ Batched embeddings (50-100x faster)

Via Command Line:

cd mcp-docs-server
source ../venv/bin/activate

python scripts/repo_indexer.py "/path/to/your/repo" "My Project"

Searching

From Web UI:

1. Enter query: "How do I initialize the SDK?"
2. Select source (Docs, Repos, or All)
3. Click "Search Documentation"
4. View results

From AI Assistant:

Search all sources:

@marcus-mcp-server search for "authentication flow"

Filter by specific source:

@marcus-mcp-server search for "BorrowInterface component" 
with source="Credo Protocol"

Example usage in Cursor:

User: Using my marcus-mcp-server, show me how authentication 
      is implemented in the Credo Protocol repository

AI: [Searches indexed repository and returns relevant code chunks]

Pro Tip: Always filter by source name to get focused results and save context tokens.

Managing Sources

View Sources:

• See all indexed docs and repos on the main page
• Filter by "Docs" or "Repos" tabs
• Expand to see individual pages/files

Delete Sources:

• Click trash icon next to any source
• Confirm deletion
• Source and all chunks are removed

📁 Project Structure

crawl4ai_test/
├── pages/                        # Next.js UI
│   ├── index.js                 # Main page (search + sources)
│   ├── add.js                   # Add documentation
│   ├── add-repo.js              # Add repository
│   └── api/                     # API routes
│       ├── mcp-search.js        # Search endpoint
│       ├── mcp-info.js          # Get index info
│       ├── add-docs-crawl.js    # Crawl docs
│       ├── add-docs-index.js    # Index docs
│       ├── add-repo-index.js    # Index repository
│       └── mcp-delete-source.js # Delete source
├── components/
│   ├── ui/                      # shadcn/ui components
│   └── home/                    # Page components
├── mcp-docs-server/             # MCP Server
│   ├── server/
│   │   └── main.py             # MCP server (stdio)
│   ├── scripts/
│   │   ├── crawler.py          # Crawl docs with Crawl4AI
│   │   ├── indexer_multi.py    # Index docs
│   │   ├── repo_indexer.py     # Index repositories
│   │   ├── get_source_pages.py # Get pages/files
│   │   ├── search.py           # Search
│   │   └── delete_source.py    # Delete sources
│   ├── data/
│   │   ├── chroma_db/          # Vector database
│   │   ├── chunks/             # Metadata
│   │   └── raw/                # Crawled JSON
│   └── requirements.txt
└── venv/                        # Python environment

🎨 Built With

• Frontend: Next.js 15 + shadcn/ui + Tailwind CSS
• Backend: Python 3.13 + MCP Protocol
• Crawler: Crawl4AI
• Vector DB: ChromaDB
• Embeddings: OpenAI (text-embedding-3-small)

Status: ✅ Fully Operational | 🤖 MCP Ready | 🔍 Search Enabled