mcp-quality-gate

Quality gate for MCP servers. Like npm audit for packages, but for Model Context Protocol servers.

When an LLM connects to your MCP server, it trusts whatever you expose. Bad tool schemas mean bad tool calls. Missing descriptions mean the model guesses. 50+ tools flood the context window. Leaked environment variables expose secrets. mcp-quality-gate catches all of this in one command.

npx mcp-quality-gate validate "npx -y @modelcontextprotocol/server-filesystem /tmp"

One command. Four dimensions. 0-100 score.

What It Catches

mcp-quality-gate scores every MCP server across four dimensions:

Install

npm install -g mcp-quality-gate

Requires Node.js >= 22.

Usage

# Test any stdio MCP server
mcp-quality-gate validate "npx -y @modelcontextprotocol/server-filesystem /tmp"

# Test with environment variables
mcp-quality-gate validate "npx -y @supabase/mcp-server-supabase@latest --read-only --project-ref REF" \
  --env "SUPABASE_ACCESS_TOKEN=your-token"

# JSON output for CI/CD pipelines
mcp-quality-gate validate "./my-server" --reporter json --output report.json

# Fail CI if score is below threshold
mcp-quality-gate validate "./my-server" --threshold 80

# Test HTTP/SSE servers
mcp-quality-gate validate "http://localhost:3000/mcp" --transport http

Real-World Benchmarks

Tested against official MCP reference servers (April 2026). These are real results from live server connections, not synthetic data:

Servers tested: @modelcontextprotocol/server-memory, @modelcontextprotocol/server-sequential-thinking, @modelcontextprotocol/server-everything, @modelcontextprotocol/server-filesystem, @anthropic/mcp-server-playwright.

Example Output

mcp-quality-gate v0.1.0
Server: npx -y @modelcontextprotocol/server-filesystem /tmp

  lifecycle
    PASS Server reports name and version (0ms)
    PASS Server reports capabilities (0ms)
    PASS Server responds to ping (1ms)

  tools
    PASS Server lists tools without error (5ms)
    PASS Tool definitions have required fields (6ms)
    PASS Tool names follow naming convention (8ms)
    PASS Tool inputSchema has type object (4ms)
    PASS Can call a listed tool (10ms)
    PASS Calling nonexistent tool returns error (1ms)
    PASS Tool descriptions are present (8ms)

  resources
    SKIP Server lists resources without error
    SKIP Resource definitions have required fields
    SKIP Resource descriptions are present
    SKIP Can read a listed resource

  prompts
    SKIP Server lists prompts without error
    SKIP Prompt definitions have required fields
    SKIP Can get a listed prompt

  efficiency
    14 tools, ~3057 schema tokens

  quality
    Param description coverage: 28%
    Deprecated: read_file
    Duplicates: read_file, read_text_file
    CRIT 18 of 25 parameters lack descriptions (72%)
    CRIT 1 deprecated tool(s) still listed: read_file
    WARN Tools with identical schemas: read_file, read_text_file

  security
    WARN "write_file" performs destructive operations — description warns of risk

Results: 10 passed, 7 skipped (45ms)
Score: 81/100
  compliance 40/40 | quality 11/25 | efficiency 15/15 | security 15/20

Compliance Tests (17)

mcp-quality-gate connects to your server, makes real protocol calls, and verifies behavior. This is not static analysis — it's a live test suite that actually calls your tools with generated arguments.

Tests are skipped (not failed) when a server doesn't advertise a capability. A tools-only server won't lose points for missing resources.

Full Test Reference

Quality Analysis

Checks how well your tool schemas help LLMs understand and use your tools:

Security Analysis

Static analysis on tool definitions to detect common security antipatterns:

Efficiency Analysis

Catches tool proliferation and schema bloat — the top causes of poor LLM performance with MCP servers:

Token estimation uses chars/4 heuristic (~15% accuracy vs tiktoken for JSON schemas).

Scoring

Composite 0-100 score. Each dimension starts at its maximum and deducts for findings:

Skipping a dimension with --skip-* flags means those points are not awarded. A server with --skip-security can score at most 80.

CLI Reference

CI/CD Integration

Add to your GitHub Actions workflow:

- name: Test MCP Server
  run: npx mcp-quality-gate validate "./my-server" --threshold 80 --reporter json --output mcp-quality-gate-report.json

mcp-quality-gate exits with code 1 when the score falls below --threshold, failing the CI step.

Programmatic API

import {
  createMCPClient,
  listAllTools,
  runTests,
  complianceTests,
  analyzeEfficiency,
  analyzeQuality,
  analyzeSecurity,
  ConsoleReporter,
} from "mcp-quality-gate";

const client = await createMCPClient({
  command: "node",
  args: ["./my-server.js"],
  transport: "stdio",
});

const tools = await listAllTools(client);
const efficiency = analyzeEfficiency(tools);
const quality = analyzeQuality(tools);
const security = analyzeSecurity(tools);

const result = await runTests(
  complianceTests,
  { client, timeout: 10000 },
  undefined,
  "my-server",
  efficiency,
  quality,
  security,
);

console.log(new ConsoleReporter().format(result));
await client.close();

Architecture

mcp-quality-gate
├── CLI (Commander)         → parse args, orchestrate
├── MCP Client Wrapper      → connect via stdio or HTTP, manage lifecycle
├── Compliance Tests (17)   → live protocol verification
│   ├── Lifecycle (3)       → init, capabilities, ping
│   ├── Tools (7)           → list, fields, naming, schema, call, errors, descriptions
│   ├── Resources (4)       → list, fields, descriptions, read
│   └── Prompts (3)         → list, fields, get
├── Quality Analyzer        → param descriptions, description length, deprecated, duplicates
├── Security Analyzer       → env exposure, code execution, dangerous defaults
├── Efficiency Analyzer     → tool count, schema token estimation
├── Score Calculator        → 4-dimension weighted composite (40+25+15+20=100)
└── Reporters               → console (colored), JSON (CI/CD)

Releases

mcp-quality-gate follows Semantic Versioning:

• 0.x.y — pre-1.0, API may change between minor versions
• PATCH (0.x.Y) — bug fixes, new compliance tests, doc updates
• MINOR (0.X.0) — new analyzer dimensions, new reporters, CLI flags
• MAJOR (X.0.0) — breaking API changes, scoring formula changes

How Releases Work

1. Bump version in package.json
2. Update CHANGELOG.md with the new version entry
3. Merge to main
4. CI automatically: runs lint + test + build, publishes to npm with provenance, creates a GitHub Release with tag v{version}

The prepublishOnly script runs lint && build && test as a safety gate. See CONTRIBUTING.md for full release instructions.

Roadmap

• v0.1 — Compliance tests (lifecycle, tools, resources, prompts), quality + security + efficiency analysis, 4-dimension scoring, CI/CD workflows
• v0.2 — Transport compliance tests (HTTP/SSE edge cases), response schema validation, capability refusal tests
• v0.3 — mcp-quality-gate init scaffolding, GitHub Action for CI, performance benchmarking
• v1.0 — Dynamic security testing, MCP server registry scanner, stable API

Contributing

See CONTRIBUTING.md for development setup, code standards, and how to add tests.

Security

See SECURITY.md for reporting vulnerabilities.

Code of Conduct

See CODE_OF_CONDUCT.md.

License

MIT