how are you planning to stay compliant with the exploding number of clients users will have in practice? Or do you not think the number of clients is going to balloon?
yeah the # of clients will definitely increase, we'll start by looking to stay up to date with major AI clients as best we can, see caniuse.dev for diffs in client capabilities
We also think giving people completely configurable clients is extremely valuable, we keep these client "templates" up to date with the latest capabilities- but folks can certainly change up protocol/extension capabilities beyond that e.g. if Copilot is planning to support more apps capabilities in the coming weeks, use the Copilot template and toggle on more MCP apps endpoints to see if you're able to make use of them
love the direction, but the problem for me has been about creating stronger evals and knowing what I should be checking for. does this help me understand that?
yeah, in my blog on Effective MCP (https://www.mcpjam.com/blog/effective-mcp-part-1) I introduce this framework called the "User-Value Chain" that essentially breaks down the full client-server request flow into distinct stages.
You want to test the full request flow between client and server: connection, tool discovery, tool selection, calls, responses, and whether the user’s GOAL was actually achieved.
What's hard: an external agent sits between your user and your MCP server, in a client you don't control (ChatGPT, claude etc.). Your tests need to cover how that client and agent find and uses your tools.
Recommend checking for :
- (deterministic assertions + non-deterministic judge checks) at essentially every stage of the request flow from client->server (User Value Chain)
- across many clients where YOUR target users are at (unfort. these clients change behavior every other day) and across harness + models
- our default eval assertions cover things like input schemas, valid arguments, repeated calls, errors, latency, and response size; pair those deterministic checks with non-deterministic evals for tool choice, response interpretation, and task completion.
how are you planning to stay compliant with the exploding number of clients users will have in practice? Or do you not think the number of clients is going to balloon?
yeah the # of clients will definitely increase, we'll start by looking to stay up to date with major AI clients as best we can, see caniuse.dev for diffs in client capabilities
We also think giving people completely configurable clients is extremely valuable, we keep these client "templates" up to date with the latest capabilities- but folks can certainly change up protocol/extension capabilities beyond that e.g. if Copilot is planning to support more apps capabilities in the coming weeks, use the Copilot template and toggle on more MCP apps endpoints to see if you're able to make use of them
love the direction, but the problem for me has been about creating stronger evals and knowing what I should be checking for. does this help me understand that?
yeah, in my blog on Effective MCP (https://www.mcpjam.com/blog/effective-mcp-part-1) I introduce this framework called the "User-Value Chain" that essentially breaks down the full client-server request flow into distinct stages.
You want to test the full request flow between client and server: connection, tool discovery, tool selection, calls, responses, and whether the user’s GOAL was actually achieved.
What's hard: an external agent sits between your user and your MCP server, in a client you don't control (ChatGPT, claude etc.). Your tests need to cover how that client and agent find and uses your tools.
Recommend checking for :
- (deterministic assertions + non-deterministic judge checks) at essentially every stage of the request flow from client->server (User Value Chain)
- across many clients where YOUR target users are at (unfort. these clients change behavior every other day) and across harness + models
- our default eval assertions cover things like input schemas, valid arguments, repeated calls, errors, latency, and response size; pair those deterministic checks with non-deterministic evals for tool choice, response interpretation, and task completion.
[dead]
[dead]