@yawlabs/mcp-compliance

Test any MCP server for spec compliance. 69-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the MCP specification. CLI, MCP server, and programmatic API.

Built and maintained by Yaw Labs.

Why this tool?

MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.

This tool solves that:

• 69 tests across 8 categories — transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. No gaps.
• Capability-driven — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
• Graded scoring — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
• CI-ready — --strict mode exits with code 1 on required test failures. Drop it into any pipeline.
• Spec-referenced — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
• Three interfaces — CLI for humans, MCP server for AI assistants, programmatic API for integration.
• Published specification — the testing methodology and rule catalog are open (CC BY 4.0) so anyone can implement compatible tooling.

Quick start

Run against any MCP server:

npx @yawlabs/mcp-compliance test https://my-server.com/mcp

Or install globally:

npm install -g @yawlabs/mcp-compliance
mcp-compliance test https://my-server.com/mcp

That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.

CLI usage

# Terminal output with colors and grade
mcp-compliance test https://my-server.com/mcp

# JSON output (for scripting)
mcp-compliance test https://my-server.com/mcp --format json

# SARIF output (for GitHub Code Scanning)
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif

# Strict mode — exits with code 1 on required test failure (for CI)
mcp-compliance test https://my-server.com/mcp --strict

# With authentication
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"

# Custom headers (repeatable)
mcp-compliance test https://my-server.com/mcp -H "Authorization: Bearer tok123" -H "X-Api-Key: abc"

# Custom timeout (default: 15000ms)
mcp-compliance test https://my-server.com/mcp --timeout 30000

# Retry failed tests
mcp-compliance test https://my-server.com/mcp --retries 2

# Only run specific categories or test IDs
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
mcp-compliance test https://my-server.com/mcp --only lifecycle-init,tools-list

# Skip specific categories or test IDs
mcp-compliance test https://my-server.com/mcp --skip prompts,resources

# Verbose mode — print each test result as it runs
mcp-compliance test https://my-server.com/mcp --verbose

Options

Get badge markdown

mcp-compliance badge https://my-server.com/mcp

Outputs the markdown embed for a compliance badge hosted at mcp.hosting.

What the 69 tests check

Grading

Required tests are worth 70% of the score, optional tests 30%. See the full scoring algorithm in the specification.

CI integration

# GitHub Actions example
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict

# With JSON output for parsing
- name: MCP Compliance Check
  run: |
    npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format json > compliance.json
    cat compliance.json | jq '.grade'

# With retries for flaky network conditions
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000

# SARIF output for GitHub Code Scanning
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif
- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: compliance.sarif

MCP server (for Claude Code, Cursor, etc.)

This package also exposes an MCP server with 3 tools that can be used from Claude Code, Cursor, or any MCP client.

Setup

Claude Code (one-liner):

claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance mcp

Or create .mcp.json in your project root:

macOS / Linux / WSL:

{
  "mcpServers": {
    "mcp-compliance": {
      "command": "npx",
      "args": ["-y", "@yawlabs/mcp-compliance", "mcp"]
    }
  }
}

Windows:

{
  "mcpServers": {
    "mcp-compliance": {
      "command": "cmd",
      "args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance", "mcp"]
    }
  }
}

Tip: This file is safe to commit — it contains no secrets.

Restart your MCP client and approve the server when prompted.

Tools

• mcp_compliance_test — Run the full 69-test suite against a URL. Supports auth, custom headers, timeout, retries, and category/test filtering. Returns grade, score, and detailed results.
• mcp_compliance_badge — Get the badge markdown/HTML for a server. Supports auth and custom headers.
• mcp_compliance_explain — Explain what a specific test ID checks and why it matters.

All tools have MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) so MCP clients can skip confirmation dialogs for safe operations.

Programmatic usage

import { runComplianceSuite } from '@yawlabs/mcp-compliance';

const report = await runComplianceSuite('https://my-server.com/mcp');
console.log(`Grade: ${report.grade} (${report.score}%)`);

// With options
const report2 = await runComplianceSuite('https://my-server.com/mcp', {
  headers: { 'Authorization': 'Bearer tok123' },
  timeout: 30000,
  retries: 1,
  only: ['transport', 'lifecycle'],
});

Specification

The compliance testing methodology is published as an open specification:

• MCP Compliance Testing Specification — test execution model, scoring algorithm, all 69 test rules with pass/fail criteria (CC BY 4.0)
• Machine-readable rule catalog — JSON Schema-compliant catalog for programmatic consumption

These are complementary to (not competing with) the official MCP specification. The MCP spec defines what servers must do; this spec defines how to verify compliance.

Requirements

• Node.js 18+

Contributing

git clone https://github.com/YawLabs/mcp-compliance.git
cd mcp-compliance
npm install
npm run build
npm test

Development commands:

Links

• mcp.hosting — Hosted MCP server infrastructure
• MCP Specification
• MCP Compliance Testing Spec
• Yaw Labs

License

MIT — see LICENSE.