Agent Eval

Overview

Agent Eval is an MCP server that brings statistical regression testing to LLM agents. It measures behavior change through p-values, effect sizes, and confidence intervals, giving you rigorous insights into how agent modifications affect performance. By exposing these capabilities as MCP tools, it lets Claude, Cursor, VS Code, and other MCP clients run statistical tests directly without switching apps.

The server runs locally on your machine using the Python package agent-regress-cli, installed via uvx. No authentication is required—it works immediately after installation. Agent Eval is released as v0.1.8 under the Apache-2.0 license, with its source code available at RudrenduPaul/agent-eval on GitHub.

Agent Eval belongs in the AI & Agents category on mcp.site and works with MCP clients including Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, and Zed. If you maintain this project, claim your listing to earn a Verified badge and ensure the information stays accurate.

Installation

// claude_desktop_config.json
{
  "mcpServers": {
    "agent_eval": {
      "command": "uvx",
      "args": [
        "agent-regress-cli"
      ]
    }
  }
}