SmartlyDocs
Open workspace

Enterprise quickstart

Connect your agent to Smartly

Connect your agent. Trace real execution. Evaluate failures. Turn corrections into measurable improvements.

About 15 minutesNo production changes neededSmartly never edits your code
Step 1Register the endpointWhere Smartly sends test conversations.Step 2Connect Smartly MCPSo every run reports what happened.Step 3Verify & publishOne full trace, then go live for evaluators.Step 4Review & improveAct on evidence-backed evaluations.

Your Smartly MCP endpoint

https://infra.smartlylabs.ai/api/mcp

How Smartly works

Smartly captures what your agent actually does on every run, routes the cases that need judgment to qualified human evaluators, and helps you turn accepted findings into a measurably better next version.

  1. 01

    Trace

    Capture inputs, model and tool calls, errors, outcomes.

  2. 02

    Evaluate

    Checks and automated judgment first.

  3. 03

    Human Evaluation

    Qualified people for the hard cases.

  4. 04

    Improve

    Your team ships a new version.

  5. 05

    Measure again

    Compare versions before and after.

It's a loop, not a one-time audit: every new version re-enters at Trace, so results can be compared over time.

The two connections

One lets people talk to your agent; the other lets Smartly see what it did.

Agent API endpoint

Smartly → your agent

Lets you and approved evaluators test the agent in multi-turn sessions.

Smartly MCP server

Your runtime → Smartly

Sends execution telemetry so every run becomes a structured trace.

Good to know

You need both for the full loop. You can start with the endpoint and add MCP right after.

Before you begin

  • One agent or workflow to connect
  • A staging or sandbox environment
  • A technical owner with a clear evaluation goal
  • Approved test data or scenarios
  • An MCP-compatible runtime
  • Approval for what data it may report

Tip

Start with staging and no live customer data. Move to production once the integration and data scope are reviewed.
1

Register the agent endpoint

Tell Smartly where your agent lives and how to call it.

  1. 1In the workspace, go to Agents → Create agent and give it a name.
  2. 2On the Configuration tab, add the public HTTPS endpoint. Requests are POST.
  3. 3Pick the API format and authentication, then save.
  4. 4On the Test tab, press Test connection.
API formatUse when
Smartly / OpenAI CompatibleAccepts messages, returns choices[0].message.content.
OpenAI Responses CompatibleAccepts input, returns output_text or a Responses-style item.
Custom JSONAnything else: provide a request template and a response path.
  • Authentication: None, Bearer token, or an API-key header. Credentials are encrypted and never shown again.
  • Timeout: 1–120 seconds (30 by default).
  • Requirements: public HTTPS, no redirects, reachable from Smartly's hosting.
2

Connect Smartly MCP

Add Smartly to your agent's runtime so it can report each run.

MCP endpoint
https://infra.smartlylabs.ai/api/mcp
Example MCP client configjson
{
  "mcpServers": {
    "smartly": {
      "url": "https://infra.smartlylabs.ai/api/mcp"
    }
  }
}
  1. 1Your runtime opens Smartly in the browser.
  2. 2Sign in as the enterprise admin.
  3. 3Review the authorization request.
  4. 4Approve the connection. It now appears under Settings.

Good to know

The connection is organization-scoped: it can report for any of your agents, so every call must include the right agent ID. Copy it from the agent's header in the workspace.
3

Report runs with the MCP tools

Four tools cover the life of a run.

smartly_start_run

Start of a run

Agent, agent version, session, task and model.

smartly_record_events

As things happen

Prompts, model and tool calls, decisions, errors, retries, fallbacks, human actions, usage, cost.

smartly_request_human_review

Optional, any time

Creates an async task for human evaluators. Never pauses the run.

smartly_finish_run

End of a run

Final output, duration, outcome, and whether a human intervened.

Tip

Your MCP client lists each tool's exact input schema (via tools/list). Use those argument names.

Versions & sessions

Small habits that unlock before-and-after comparisons.

  • Send an agent version with every run. The Insights page compares success rate, latency, tokens and rating version by version, so you can show a change worked (“v2 cut this failure from 18% to 7%”).
  • Give each turn a unique run ID, and reuse one session ID across a conversation.
  • Report usage and cost so Insights can show spend per run and per version.

What makes a useful trace

Smartly only learns from what your agent reports. Report applicable events as they happen, with enough context to follow the sequence.

Every run shows a trace checklist: task, model calls, tool calls, decisions, errors & retries, usage & cost, human intervention and final output, each marked Reported, N/A or Missing.

Good to know

Smartly never infers unreported activity. Runs with missing telemetry are labelled incomplete, never assumed successful.
4

Verify & publish

  1. 1On the agent's Test tab, run Test connection.
  2. 2Run MCP verification: one traced call must report start, events and finish within 10 minutes.
  3. 3Press Publish agent. Approved evaluators can now test it in multi-turn sessions.
5

Review evaluations

What the agent did → what went wrong → what should happen instead.

Each Human Evaluation comes with evidence from the run: test input and response, errors and latency, verdict, issue category, suggested correction and reasoning. Review them under Evaluations.

StatusMeaning
Pending reviewWaiting for your decision.
Evaluation reviewedAccepted and archived for your team (integrated once the change is live).
Not neededClosed: you decided not to act on it.

Tip

Your team makes every change in its own codebase. Smartly provides the evidence; you own every decision.

Data & security

Smartly sees only what your agent explicitly reports through MCP, plus its replies to test sessions. There is no discovery or crawling.

Important

Never send access tokens, passwords, session cookies, encryption keys or unnecessary personal data in telemetry. When in doubt, leave it out or mask it.

You choose the agent, staging vs. production, what gets reported, and which suggestions ship. Revoke MCP connections any time in Settings.

Troubleshooting

Paste any error from the Test tab, a run page or your logs. Fixes appear as you type.

Tip

Using an AI assistant or coding agent? Point it at /docs.md (or /llms.txt), or use Ask AI at the top of this page to open ChatGPT or Claude with this guide loaded.

Quick reference

The workspace Overview tracks these stages for you.

  1. 1JoinAccept the invitation and sign in.
  2. 2Connect agentAdd the endpoint and authentication.
  3. 3Test connectionConfirm Smartly can reach your agent.
  4. 4Connect MCPAuthorize Smartly in your runtime.
  5. 5VerifyCapture start → events → finish.
  6. 6PublishOpen the agent to Human Evaluation.
  7. 7Human EvaluationEvaluators test the agent.
  8. 8ReviewReview evidence and suggestions.
  9. 9ImproveImplement accepted changes.
  10. 10ConfirmMark the change integrated once live.

Glossary

Run
One agent turn: one input, one output and everything between. Has a unique run ID.
Session
Related runs, such as a multi-turn conversation, sharing a session ID.
Event
One recorded step inside a run: model call, tool call, error, retry, decision…
Trace
The ordered record of every event in a run.
Evaluator
A qualified, approved person who tests and evaluates your agent.
Evaluation
An evaluator's feedback, tied to evidence from a specific run.
Instrumentation gap
A run with missing telemetry. Shown openly, never assumed successful.
Version
An identifiable state of your agent, for before-and-after comparison.

Questions before, during or after onboarding?

We're happy to help you connect your first agent.

ojas@smartlylabs.ai