How to Evaluate an MCP Server Before You Connect It

Most people connect an MCP server the way they install an app: find it in a directory, click connect, start using it. That works fine for low-stakes personal tools. For anything touching your company’s customer data, sales conversations, CRM records, or administrative permissions, it is the wrong approach.

You would not eat at a restaurant that refused to show you the menu. You would not hire a contractor whose only credential was that he owned a truck. And you should not connect an MCP server to a system that handles your pipeline or your customer conversations without actually checking what the vendor has built.

I recently wrote the technical pieces in this series, using the McDonald’s kitchen as an analogy, discussing how MCP servers actually work mechanically, and the three failure modes that separate a good server from a bad one. This piece is the practical buyer’s-side version of all of it. As an admin of our Enterprise Claude instance at Attention, this evaluation work lands on my desk as often on the buyer side as on the vendor side. The checklist below is what I actually use when a new connector lands on my desk.

TL;DR

• Read the vendor’s MCP documentation first. If there is no public tool reference, the integration is unfinished or the vendor does not expect scrutiny.
• Count the read tools versus the write tools. Read-only servers are viewing windows, not integrations.
• Ask how many tool calls it takes to answer a real question. If a representative analytical query takes more than a handful of round trips, the architecture is wrong.
• Ask whether there is an autonomous agent layer. The best MCP servers expose more than individual tools—they expose orchestrated AI workflows.
• Verify permission inheritance. If your account cannot see a record in the product, the AI must not be able to see it through the connector either.
• Look at the version history. Active maintenance is the signal that the vendor is serving real customers, not running a press release.

Read the Documentation Before You Connect Anything

A vendor with a serious MCP integration has public documentation that lists every tool by name, describes what it does, specifies what scope it requires, and clearly distinguishes read operations from write operations. If you cannot find that document, or if it lists tools without descriptions, the integration is either unfinished or the vendor does not expect you to scrutinize it. Both are problems.

Attention’s full tool reference lives at docs.attention.com/mcp/overview. All 68 tools are listed with scope requirements and one-line descriptions. That is the baseline you should expect from any serious vendor.

If the only “documentation” is a blog post announcing the integration, you do not have documentation. You have marketing.

Count the Read Tools Versus the Write Tools

A server that only exposes read tools can retrieve information but cannot act. That is fine for some use cases and a meaningful limitation for others. More importantly, it often signals that the vendor shipped MCP as a checkbox rather than a genuine capability extension.

Ask specifically: what can Claude do through this server that a logged-in user cannot already see in the product UI? If the honest answer is “see the same things you can already see, but in a different window,” you have a viewing window, not a kitchen.

The test I use: pull up the vendor’s tool reference and group the tools into three buckets. Read tools (search, list, get, view). Write tools (create, update, delete, archive). Action tools (anything that triggers a workflow, runs analysis, or invokes the vendor’s own AI engine on your behalf). A server that is 90% read tools is a brochure. A server that has all three buckets meaningfully populated is an actual integration.

Attention’s MCP server has read, write, and action tools across all 15 functional groups, plus a dedicated Super Agent group for orchestrated work. When I look at competing servers in this space, that balance is rare. Most are heavy on retrieval and thin on everything else.

Ask How Many Tool Calls It Takes to Answer a Real Question

Not a toy example. A question your team actually needs answered.

Walk the vendor through it. “If I ask the AI to analyze the objections that came up in our enterprise deals last quarter, how does that work? Which tools get called? In what sequence?”

If the answer involves the AI calling search_calls 25 times, then get_call_ details 25 more times, then synthesizing the results in its own context window -- you are looking at the third failure mode from the previous piece. Lots of stations, lots of trips, lots of tokens, slow answers.

The better answer is: “It calls one tool, the analytical engine processes everything server-side, and the AI gets a finished answer back.” That is the Robot Coupe pattern, and it is the single most important architecture decision an MCP server vendor can make. Anthropic’s engineering team has documented a representative case in which this pattern reduced a 150,000-token workflow to roughly 2,000 tokens, a 98.7% reduction with no loss of accuracy.

At Attention, that tool is called ask_attention. If you are evaluating any vendor in the conversation intelligence or CRM-adjacent space, ask them directly whether they have something equivalent. If they do not, every analytical question you ask is going to be expensive.

Ask Whether There Is an Autonomous Agent Layer

The most capable MCP servers do not just expose individual tools. They expose an AI agent that can orchestrate complex multi-step work on its own—reasoning about which tools to use, in what order, with which inputs, without requiring the connecting AI client to chain everything manually.

Attention’s MCP server includes a Super Agent group: six tools for managing autonomous chat sessions that handle compound workflows. If the question is too complex to be answered by a single bundled tool, the Super Agent can run a multi-step session and return the synthesized result.

Not every product needs this. But when you are evaluating a vendor in a category where workflows span retrieval, analysis, configuration, and follow-up actions, the presence of an agent layer separates the genuine agentic integrations from the ones built only for simple retrieval.

Verify That Permission Inheritance Carries Through

This is the one most buyers forget to check, and it is the most important from a security standpoint.

Ask the vendor directly: if my account cannot see a specific record in your product, can the AI see it through the MCP server?

The answer should be no, unambiguously. The MCP server should run every tool under your authenticated identity and enforce the same access controls as the underlying product. If you cannot see something in the app, the AI cannot see it through the connector.

Attention enforces this server-side. Every tool runs under the authenticated user’s identity, and Attention’s permission boundaries apply through the MCP server exactly as they apply in the product UI.

A vendor who cannot answer this question clearly, or who waves it off as “we are working on it,” has a security gap you should not accept for a system handling customer conversations and sales data. Research scanning nearly 2,000 publicly-exposed MCP servers in mid-2025 found that all verified servers lacked authentication entirely. The market is still maturing. Asking directly is not overcautious. It is responsible.

Look at the Version History

A version history tells you whether the kitchen is staffed and serving, or whether it opened with a press release and quietly closed afterwards.

Attention’s MCP server is on version 1.7.0 as of May 2026, with continuous additions across seven named releases since v1.0. Each release has added functional depth—new tool groups, expanded write capabilities, transcript search, reporting endpoints. The fact that the server has a public changelog at all is a quiet signal that the vendor is treating this as a real product surface, not a one-time integration.

If the most recent update is from six months ago and the changelog only has one entry, the integration is probably not being used seriously by anyone, including the vendor. You should not be the first.

The 30-Minute Version of This Checklist

If you only have time for a five-question fast pass before connecting a new MCP server, here is the short version:

1. Does the vendor have public, complete documentation listing every tool with descriptions?
2. Are there meaningful write tools and action tools, not just read tools?
3. Is there a bundled, server-side analysis tool that collapses long call chains into a single request?
4. Does the vendor enforce the same permission model in the MCP server as in the product UI?
5. Has the server been actively maintained, with a public version history?

Five yeses means the vendor has done the work. Anything less, and you have something to dig into before you connect.

What Comes Next

The magic that the first piece in this series talked about does not disappear when you understand the mechanics. It just becomes something you can evaluate, compare, and hold vendors accountable for.

If you are running an AI-native sales motion in 2026, you are going to connect a lot of MCP servers over the next year. Some will be excellent. Most will not. The checklist above is what separates the two before you spend your team’s time and your company’s tokens finding out the hard way.

If you want to see what “good” looks like in practice, Attention’s MCP server documentation is at docs.attention.com/mcp/overview. Use it as a reference point when you are evaluating others.

References

1. Attention Docs — “Attention MCP Server” — https://docs.attention.com/mcp/overview — 2026
2. Maxim AI / Bifrost — “Cutting MCP Token Costs by 92% at 500+ Tools” — https://www.getmaxim.ai/articles/cutting-mcp-token-costs-by-92-at-500-tools/ — 2026
3. Attention Docs — “AI Analysis Tools” — https://docs.attention.com/mcp/tools/ai-analysis — 2026
4. Descope — “What Is the Model Context Protocol (MCP) and How It Works” — https://www.descope.com/learn/post/mcp — 2026
5. MCP Playground — “MCP Token Counter: Why Your Tools Are Silently Eating Your Context Window” — https://mcpplaygroundonline.com/blog/mcp-token-counter-optimize-context-window — 2026