Mcp Eval Runner

Overview

Mcp Eval Runner is a standardized testing harness for MCP servers and agent workflows. It enables any MCP-compatible AI assistant to run evaluations and tests directly, with the model deciding when and how to invoke testing capabilities during conversations or agent runs.

The server runs locally via the npm package mcp-eval-runner, communicating over stdio on your machine. Setup requires YOUR_API_KEY for authentication through environment variables. The latest release is v1.0.0, distributed under the MIT license with source code available at dbsectrainer/mcp-eval-runner on GitHub.

Mcp Eval Runner is categorized as AI & Agents on mcp.site and integrates with MCP clients including Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, Zed, and others. If you maintain this project, claim your listing to verify ownership, earn the Verified badge, and ensure information stays current.

Installation

// claude_desktop_config.json
{
  "mcpServers": {
    "mcp_eval_runner": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-eval-runner"
      ],
      "env": {
        "YOUR_API_KEY": "<your-api-key>"
      }
    }
  }
}

Requires YOUR_API_KEY