The real catalog of tools for AI agents

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

A standardized testing harness for MCP servers and agent workflows

LLM evals as MCP tools: score outputs for faithfulness, relevancy, and hallucination.

MCP server for Google Lighthouse performance metrics

Multi-LLM dev harness, MCP-operable: bugs, cycles, gates. Verdicts are exit codes, never opinions.

MCP server for Harness API integration

Query system performance metrics via MCP - CPU, memory, disk I/O, network, processes

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.

MCP server for FCA Handbook compliance analysis with verbatim citations.

Observability MCP server: query LLM cost/errors/latency & operate alerts/evals from Claude/Cursor

Cross-harness communication mesh for LLM agents — rooms, DMs, presence, and visibility over TCP

MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.

AI Security Harness: SAST, secrets, deps, containers, config & OWASP LLM. Scans locally.

MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.

MCP tools for Sema — eval, compile, build, format, and docs for a Lisp with LLM primitives.

LLM provider intelligence: recommendations, pricing, and status

MCP compliance proxy between agent and tools: deterministic governance, fail-closed, audit trail.

AI/LLM agent output audit MCP: policy eval, tamper-evident chain, AI safety, x402 USDC on Base.

35-probe LLM/agent security red-team scan (injection, jailbreak, MCP abuse) with report.