Ubuntu Desktop Control MCP Server

An MCP (Model Context Protocol) server that enables LLMs to control your Ubuntu desktop by taking screenshots and sending mouse clicks. This allows AI assistants to visually interact with your desktop applications.

⚡ NEW: Optimized Production Workflow

5x faster, 5x more accurate! Now using the same optimization techniques as Anthropic's Computer Use API:

• 📸 Smart Screenshots: Auto-downsampled to 1280x720 (5x smaller)
• 🎯 Numbered Elements: See what's clickable at a glance with overlaid IDs
• 🤖 AT-SPI Integration: Automatic UI element detection using accessibility API
• 📐 Percentage Coords: Resolution-agnostic positioning (no more pixel hunting!)
• ⚡ Workflow Batching: Execute multiple actions in one MCP call
• 🎪 Element Cache: Direct element interaction - "click element #5"

Example - Old way (8+ calls, ~15s):

take_screenshot() → analyze → grid overlay → zoom quadrant → find pixel → click → miss

Example - New way (1 call, ~3s):

take_screenshot() → "I see Pinta is element #5" → click_screen(element_id=5) → ✓

See README.md for full details.

Features

• 📸 Screenshot Capture: Annotated screenshots with automatic element detection
• 🔢 Element Detection: AT-SPI + CV fallback for robust UI element identification
• 🖱️ Smart Clicking: Click by element ID or percentage coordinates
• ⌨️ Keyboard Control: Type text and press keys/hotkeys
• 🎯 Mouse Movement: Smooth cursor positioning with animation
• 🚀 Workflow Batching: Execute multi-step tasks in single MCP call
• 📊 Diagnostics: Display scaling detection, warnings, and recommendations

Quick Start

1. Prerequisites

• Ubuntu Linux (X11 required, Wayland not fully supported)
• Python 3.9+

2. Installation

From PyPI (Recommended)

pip install ubuntu-desktop-control

From Source

# Clone repository
git clone https://github.com/charettep/ubuntu-desktop-control-mcp.git
cd ubuntu-desktop-control-mcp

# Install system dependencies (requires sudo)
chmod +x scripts/install.sh
./scripts/install.sh

# Install Python dependencies
pip install -e .

Configuration

Claude Code

claude mcp add --transport stdio ubuntu-desktop-control -- \
  ubuntu-desktop-control

{
  "mcpServers": {
    "ubuntu-desktop-control": {
      "command": "ubuntu-desktop-control",
      "args": []
    }
  }
}

VS Code Insiders

{
  "servers": {
    "ubuntu-desktop-control": {
      "type": "stdio",
      "command": "ubuntu-desktop-control",
      "args": []
    }
  }
}

Codex CLI

codex mcp add ubuntu-desktop-control -- \
  ubuntu-desktop-control

[mcp_servers.ubuntu-desktop-control]
type = "stdio"
command = "ubuntu-desktop-control"
args = []

Tools

Core Capabilities

Prompt Rendering Tools

These tools allow clients without native prompt support (like Codex CLI) to render prompt templates as text.

Prompts

Configuration & Customization

Environment Variables

The server relies on standard Linux/X11 environment variables to locate and interact with the desktop session.

Passing Environment Variables

You can customize these variables in your MCP client configuration.

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "ubuntu-desktop-control": {
      "command": "ubuntu-desktop-control",
      "args": [],
      "env": {
        "DISPLAY": ":0",
        "XAUTHORITY": "/home/user/.Xauthority"
      }
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "ubuntu-desktop-control": {
      "command": "ubuntu-desktop-control",
      "args": [],
      "env": {
        "DISPLAY": ":0"
      }
    }
  }
}

Display Scaling & Coordinates

If clicks land in the wrong place, you likely have a HiDPI display scaling mismatch (e.g., logical 1920x1080 vs physical 3840x2160).

Solutions:

1. Auto-scale: Use click_screen(..., auto_scale=True) to let the server handle it.
2. Diagnostics: Run get_display_diagnostics() to see the scaling factor.
3. Element IDs: Use take_screenshot(detect_elements=True) and click via element_id or percentage coordinates.

Troubleshooting

Security

⚠️ Warning: This server gives LLMs full control over your mouse and visibility of your screen.

• Only use with trusted clients.
• Be aware screenshots may capture sensitive data.
• Automated clicks can be destructive.

License

MIT License