uitars-mcp
MCP server that gives AI coding agents local GUI grounding — the ability to find any UI element on screen and return its exact pixel coordinates.
Powered by UI-TARS-2B, ByteDance's 2B parameter GUI grounding model.
Why
Claude Code's built-in computer-use sends every screenshot to the cloud for analysis. This MCP server runs a local vision model instead:
- ~1.2s per element find (vs cloud round-trip latency)
- 4.1GB VRAM (runs on any modern GPU)
- Fully offline — no API keys, no cloud dependency
- 90.7% accuracy on ScreenSpot desktop-text benchmark
- Native pixel coordinates — returns exact click targets
Setup
1. Download UI-TARS-2B
# Requires ~4.5GB disk space
huggingface-cli download bytedance-research/UI-TARS-2B-SFT --local-dir ./ui-tars-2b
2. Install PyTorch with CUDA
# Install CUDA-enabled PyTorch first (adjust cu126 to your CUDA version)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
3. Install uitars-mcp
pip install uitars-mcp
# or from source:
pip install -e .
4. Configure Claude Code
Add to your Claude Code MCP settings (~/.claude/settings.json):
{
"mcpServers": {
"uitars-mcp": {
"command": "uitars-mcp",
"env": {
"UITARS_MODEL": "/path/to/ui-tars-2b"
}
}
}
}
If installed in a venv, use the full path to the executable:
{
"mcpServers": {
"uitars-mcp": {
"command": "/path/to/venv/bin/uitars-mcp",
"env": {
"UITARS_MODEL": "/path/to/ui-tars-2b"
}
}
}
}
Tools
| Tool | What it does | Latency |
|---|---|---|
find_element | Find a UI element by description, returns click coordinates | ~1.2s |
describe_screen | Describe everything visible on screen | ~2s |
read_screen_text | OCR — read all text on screen | ~3s |
check_element | Check element state (enabled, value, etc.) | ~1s |
verify_action | Verify an action worked by checking screen state | ~1.5s |
suggest_action | Suggest next action to achieve a goal | ~1.5s |
benchmark | Measure end-to-end latency | varies |
How it works
- Takes a screenshot via
mss(fast, cross-platform) - Resizes to 1344px wide (optimal vision token count)
- Runs UI-TARS-2B inference on GPU
- Converts model's 0-1000 normalized coordinates to native screen pixels
- Returns coordinates ready for
computer-useclick tools
The model is lazy-loaded on first call (~3s), then stays in VRAM for subsequent calls.
Environment variables
| Variable | Default | Description |
|---|---|---|
UITARS_MODEL | (required) | Path to UI-TARS-2B model directory |
Requirements
- Python 3.10+
- NVIDIA GPU with 4.1GB+ VRAM
- CUDA-enabled PyTorch
- Windows or Linux (macOS untested)