app-screen-mcp

MCP server for reliable iOS Simulator automation.

Control simulators, read accessibility UI trees, capture screenshots, and run grounded agent actions through one Model Context Protocol server.

Why this exists

Most mobile automation breaks when scripts act without observing current screen state.

app-screen-mcp fixes that by combining:

• Accessibility structure from idb ui describe-all
• Real pixels from Simulator screenshots
• Deterministic interactions (tap, type_text, swipe, hardware buttons)

This gives agents a closed loop: observe, reason, act, verify.

What you get

• Simulator discovery and boot control
• App launch and termination by bundle ID
• Full normalized accessibility tree
• JPEG screenshots with size/quality controls
• Image-hash suppression to skip unchanged screenshots
• Semantic actions by text or accessibility ID
• Relative-coordinate taps for resolution-independent flows
• One-call screen summary (UI tree + optional screenshot)

Architecture

MCP Client / Agent
        |
        v
   app-screen-mcp
        |
        +--> xcrun simctl (devices, app lifecycle, screenshots)
        |
        +--> idb (UI tree, tap, swipe, type, hardware buttons)
        |
        v
   iOS Simulator

Prerequisites

• macOS with Xcode + iOS Simulator
• Node.js 18+
• idb tooling

Manual install for idb:

brew tap facebook/fb
brew install idb-companion
pip3 install fb-idb

Install

Option 1: one-step installer (recommended)

bash <(curl -fsSL https://raw.githubusercontent.com/xmuweili/app-screen-mcp/main/install.sh)

The script checks/installs:

• Xcode Command Line Tools
• Homebrew
• Node.js (18+)
• idb-companion
• fb-idb
• app-screen-mcp (global npm package)

Option 2: global npm install

npm install -g app-screen-mcp

Option 3: from source

git clone https://github.com/xmuweili/app-screen-mcp.git
cd app-screen-mcp
npm install
npm run build

Configure your MCP client

Claude Desktop

Config file: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "ios-simulator": {
      "command": "node",
      "args": ["/absolute/path/to/app-screen-mcp/dist/index.js"]
    }
  }
}

Cursor / VS Code MCP

Common config paths:

• .cursor/mcp.json
• .vscode/mcp.json

Use the key your client expects: mcpServers or mcp.servers.

{
  "mcpServers": {
    "ios-simulator": {
      "command": "node",
      "args": ["/absolute/path/to/app-screen-mcp/dist/index.js"]
    }
  }
}

Restart your MCP client after updating config.

Tool reference (15 tools)

Quick usage patterns

1) Observe without image (cheap)

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": false,
    "compact_tree": true
  }
}

2) Add image when visual confirmation is required

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": true,
    "max_dim": 720,
    "quality": 55
  }
}

3) Skip unchanged screenshots

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": true,
    "only_if_changed": true,
    "previous_image_hash": "<last_hash>"
  }
}

4) Stable interaction flow

1. get_screen_summary(include_image=false)
2. find_elements("Sign In")
3. tap_id("login_email") or tap_text("Email")
4. type_text("user@example.com")
5. tap_text("Sign In")
6. get_screen_summary(include_image=true, only_if_changed=true)

Permission prompts

Permission behavior is enforced by the MCP client, not this server.

Claude Code (CLI)

Allow this server in ~/.claude/settings.json:

{
  "permissions": {
    "allow": ["mcp__ios-simulator__*"]
  }
}

ios-simulator must match your MCP server name.

Codex CLI

Codex uses command-level approvals. Use “always allow” for recurring safe command prefixes when prompted.

Local development

npm install
npm run build
npm start

Main implementation:

• src/index.ts

Testing

The integration tests run against a real booted iOS Simulator.

npm test

Before running tests:

• Boot an iOS Simulator device.
• Build/install demo app under demo-app/ if not already installed.

Test helpers:

• tests/helpers/client.ts
• tests/helpers/ui.ts
• tests/helpers/vision.ts

Troubleshooting

• No iOS simulator is currently running
  • Boot a simulator first, or call boot_simulator.
• idb command failures
  • Verify idb-companion and fb-idb installation and PATH.
• Missing/weak element matches
  • Improve app accessibility labels and identifiers.
• GUI client cannot find node/module path
  • Use an absolute path in args instead of relying on shell PATH.

Contributing

Contributions are welcome.

For tool changes:

• Update tool schema in src/index.ts
• Implement tool handler in src/index.ts
• Add integration coverage in tests/demo-app.test.ts
• Update Tool Reference in this README

License

MIT