# Connect your agent to Smartly

> Smartly Enterprise is the composable intelligence layer for enterprise AI. It traces what your AI agent actually does, routes hard cases to qualified human evaluators, and helps you turn accepted findings into a measurably better next version. This guide explains how to connect an agent (about 15 minutes, no production changes needed; Smartly never edits your code).

Human docs: https://connect.smartlylabs.ai/docs
Workspace: https://connect.smartlylabs.ai/enterprise
Smartly MCP endpoint: https://infra.smartlylabs.ai/api/mcp

## How Smartly works

1. **Trace**: capture inputs, model and tool calls, errors, outcomes.
2. **Evaluate**: checks and automated judgment first.
3. **Human Evaluation**: qualified people for the hard cases.
4. **Improve**: your team ships a new version.
5. **Measure again**: compare versions before and after.

## The two connections

- **Agent API endpoint** (Smartly → your agent): lets admins and approved evaluators test the agent in multi-turn sessions.
- **Smartly MCP server** (your runtime → Smartly): sends execution telemetry so every run becomes a structured trace.

You need both for the full loop. You can start with the endpoint and add MCP right after.

## Before you begin

- One agent or workflow to connect
- A staging or sandbox environment (no live customer data at first)
- A technical owner with a clear evaluation goal
- Approved test data or scenarios
- An MCP-compatible runtime
- Approval for what data it may report

## Step 1: Register the agent endpoint

1. In the workspace, go to **Agents → Create agent** and name it.
2. On the **Configuration** tab, add the public HTTPS endpoint. Requests are `POST`.
3. Pick the API format and authentication, then save.
4. On the **Test** tab, press **Test connection**.

| API format | Use when |
| --- | --- |
| Smartly / OpenAI Compatible | Accepts `messages`, returns `choices[0].message.content`. |
| OpenAI Responses Compatible | Accepts `input`, returns `output_text` or a Responses-style item. |
| Custom JSON | Anything else: provide a request template (`{{input}}`, `{{messages}}`, `{{session_id}}`, `{{run_id}}`) and a response path such as `data.answer`. |

- **Authentication:** None, Bearer token, or an API-key header. Credentials are encrypted and never shown again.
- **Timeout:** 1–120 seconds (30 by default).
- **Requirements:** public HTTPS, no redirects, reachable from the internet (localhost and private networks are blocked; use a tunnel for local development).

## Step 2: Connect Smartly MCP

MCP endpoint: `https://infra.smartlylabs.ai/api/mcp`

Example MCP client config:

```json
{
  "mcpServers": {
    "smartly": {
      "url": "https://infra.smartlylabs.ai/api/mcp"
    }
  }
}
```

1. Your runtime opens Smartly in the browser.
2. Sign in as the enterprise admin.
3. Review and approve the authorization request. It then appears under Settings.

The connection is **organization-scoped**: it can report for any of your agents, so every call must include the right **agent ID** (copy it from the agent's header in the workspace).

## Step 3: Report runs with the MCP tools

| Tool | When | Records |
| --- | --- | --- |
| `smartly_start_run` | Start of a run | Agent ID, agent version, session, task, model |
| `smartly_record_events` | As things happen | Prompts, model and tool calls, decisions, errors, retries, fallbacks, human actions, usage, cost |
| `smartly_request_human_review` | Optional, any time | Creates an async task for human evaluators; never pauses the run |
| `smartly_finish_run` | End of a run | Final output, duration, outcome, whether a human intervened |

Your MCP client lists each tool's exact input schema via `tools/list`; use those argument names. Call `smartly_finish_run` in a `finally` block so it runs even when the agent fails. When Smartly calls your endpoint it sends a run ID and verification ID in request headers; pass them through to `smartly_start_run` so the trace matches the call.

## Versions & sessions

- Send an **agent version** with every run, so Insights can compare success rate, latency, tokens and rating version by version.
- Give each turn a unique **run ID**, and reuse one **session ID** across a conversation (the workspace shows the whole conversation as one thread).
- Report **usage and cost** so Insights can show tokens and spend per run and per version.

## What makes a useful trace

Report applicable events as they happen, with enough context to follow the sequence. Every run shows a trace checklist: task, model calls, tool calls, decisions, errors & retries, usage & cost, human intervention and final output, each marked Reported, N/A or Missing. Smartly never infers unreported activity: runs with missing telemetry are labelled incomplete, never assumed successful.

## Step 4: Verify & publish

1. On the agent's **Test** tab, run **Test connection**.
2. Run **MCP verification**: one traced call must report start, events and finish within 10 minutes.
3. Press **Publish agent**. Approved evaluators can now test it.

Each agent also has a **Setup doctor** (Doctor link in the agent header) that checks the endpoint, authentication, connection test, MCP, verification, telemetry freshness, trace completeness, versions, token usage and leaked secrets, with a fix for each.

## Step 5: Review evaluations

Each Human Evaluation includes evidence from the run: input and response, errors and latency, verdict, issue category, suggested correction and reasoning.

| Status | Meaning |
| --- | --- |
| Pending review | Waiting for your decision |
| Evaluation reviewed | Accepted and archived for your team (integrated once the change is live) |
| Not needed | Closed: you decided not to act on it |

Your team makes every change in its own codebase; Smartly provides the evidence.

## Data & security

Smartly sees only what your agent explicitly reports through MCP, plus its replies to test sessions. Never send access tokens, passwords, session cookies, encryption keys or unnecessary personal data in telemetry; mask them first. The workspace flags values that look like credentials and the Setup doctor generates a `redact()` helper. Revoke MCP connections any time in Settings.

## Troubleshooting

### Only https:// endpoints are allowed
_Area: Endpoint_

**Why:** The endpoint URL isn't a full public HTTPS address.

**Fix:**
- Use the full URL including https://, e.g. https://agent.example.com/v1/chat.
- For local development, expose the agent with a tunnel (ngrok, Cloudflare Tunnel) and use its https URL.

See: https://connect.smartlylabs.ai/docs#register-endpoint

### Endpoint resolves to a private, loopback or reserved address
_Area: Endpoint_

**Why:** Smartly only calls public internet addresses, so localhost and internal networks are blocked.

**Fix:**
- Deploy to a public URL or use a tunnel (ngrok, Cloudflare Tunnel).
- Make sure DNS for the hostname points to a public IP.

See: https://connect.smartlylabs.ai/docs#register-endpoint

### Hostname could not be resolved
_Area: Endpoint_

**Why:** The hostname has no DNS record (typo, expired tunnel, or a DNS change that hasn't spread yet).

**Fix:**
- Check the spelling of the hostname.
- Free tunnel URLs change on restart: copy the current one into Configuration.
- Wait a few minutes after creating a DNS record, then test again.

### Credentials embedded in the URL are not allowed
_Area: Security_

**Why:** The URL contains user:password@. Secrets must never be part of the endpoint URL.

**Fix:**
- Remove user:password@ from the URL.
- Set Authentication to Bearer token or API key header instead; Smartly stores it encrypted.

### Could not reach the endpoint
_Area: Endpoint_

**Why:** Smartly couldn't open a connection: the server is down, the tunnel is closed, or a firewall blocks it.

**Fix:**
- Open the URL from another network (or curl it) to confirm it's up.
- Restart the tunnel and update the URL if it changed.
- Allow inbound HTTPS from the internet in your firewall or WAF.

### Request timed out
_Area: Endpoint_

**Why:** The agent didn't answer within the configured timeout.

**Fix:**
- Raise Timeout in Configuration (up to 120 s).
- Check for slow tools or retries in the agent; a fast first token isn't enough, the full reply must arrive.
- Test with a short prompt to separate slowness from a hang.

See: https://connect.smartlylabs.ai/docs#register-endpoint

### HTTP 401 / 403: not authorized
_Area: Endpoint_

**Why:** The agent rejected Smartly's credentials.

**Fix:**
- Re-enter the key in Configuration (saved keys are never shown again, so paste the current one).
- Match the auth type your agent expects: Bearer token vs. a custom header like x-api-key.
- Check the header name for API-key auth is spelled exactly as your agent reads it.

### HTTP 404 / 405: wrong path or method
_Area: Endpoint_

**Why:** The URL points to a path that doesn't exist, or the route doesn't accept POST.

**Fix:**
- Use the exact chat route, e.g. /v1/chat/completions for OpenAI-compatible servers.
- Smartly always sends POST: make sure the route accepts it.

### HTTP 429: rate limited
_Area: Endpoint_

**Why:** The agent (or the model provider behind it) is throttling requests.

**Fix:**
- Wait and retry, or raise the rate limit for Smartly's traffic.
- Check your model provider's quota.

### HTTP 5xx: the agent crashed
_Area: Endpoint_

**Why:** The request reached your agent but it failed while handling it.

**Fix:**
- Check the agent's server logs at the time of the test.
- Compare the request Smartly sent (Test tab → expand a message) with what your agent expects.

### Endpoint attempted a redirect
_Area: Endpoint_

**Why:** Redirects are disabled for safety, so the endpoint must answer directly.

**Fix:**
- Use the final URL (often a trailing slash or http→https difference).
- Turn off redirects on that route.

### Response was not valid JSON
_Area: Endpoint_

**Why:** The agent returned HTML, plain text or a stream instead of a JSON body.

**Fix:**
- Return application/json with the full reply (no server-sent events).
- Make sure an error page or login page isn't being returned instead.

### Could not extract a reply using the configured format
_Area: Endpoint_

**Why:** The JSON came back but not in the shape the selected API format expects.

**Fix:**
- Smartly / OpenAI Compatible expects choices[0].message.content.
- OpenAI Responses Compatible expects output_text (or a Responses output item).
- For anything else choose Custom JSON and set the response path, e.g. data.answer.

See: https://connect.smartlylabs.ai/docs#register-endpoint

### Body exceeds the size limit
_Area: Endpoint_

**Why:** The request or reply is larger than Smartly accepts.

**Fix:**
- Return only the reply text and essential metadata.
- Don't echo the full conversation or large tool payloads back in the response.

### MCP not authorized
_Area: MCP & tracing_

**Why:** The runtime isn't connected to Smartly MCP, or its connection was revoked or expired.

**Fix:**
- Re-run the MCP connection from your runtime and approve it while signed in as the enterprise admin.
- Check Settings → Authorized MCP connections: it should be listed and not revoked.

See: https://connect.smartlylabs.ai/docs#connect-mcp

### Agent not found / wrong agent ID
_Area: MCP & tracing_

**Why:** The MCP call used an agent ID that isn't in your organization.

**Fix:**
- Copy the agent ID from the agent's header in the workspace (copy button next to it).
- Use the same ID in every smartly_* call for that agent.

See: https://connect.smartlylabs.ai/docs#connect-mcp

### Run start not reported
_Area: MCP & tracing_

**Why:** Smartly called the agent, but the adapter never called smartly_start_run for that call.

**Fix:**
- Call smartly_start_run at the beginning of every request.
- Pass through the run ID and verification ID Smartly sends in the request headers so the trace matches the call.

See: https://connect.smartlylabs.ai/docs#tools

### Trace events missing
_Area: MCP & tracing_

**Why:** The run started and finished but few or no events were recorded in between.

**Fix:**
- Call smartly_record_events for model calls, tool calls, decisions and errors as they happen.
- Check the trace checklist on the run page to see which categories are missing.

See: https://connect.smartlylabs.ai/docs#useful-trace

### Run finish not reported
_Area: MCP & tracing_

**Why:** The adapter didn't call smartly_finish_run, often because an exception skipped it.

**Fix:**
- Call smartly_finish_run in a finally block so it runs even when the agent fails.
- Report the outcome (failed) and error instead of skipping the call.

See: https://connect.smartlylabs.ai/docs#tools

### Verification expired
_Area: Publishing_

**Why:** Start, events and finish must all arrive within 10 minutes of the verification call.

**Fix:**
- Start a new verification from the Test tab and run it right away.
- Make sure the adapter reports synchronously, not from a delayed batch job.

See: https://connect.smartlylabs.ai/docs#verify-publish

### Publish is blocked
_Area: Publishing_

**Why:** An agent can only be published after a passing connection test and a verified trace.

**Fix:**
- Run Test connection, then MCP verification, on the Test tab.
- Open the agent's Setup doctor to see exactly which check is failing.

See: https://connect.smartlylabs.ai/docs#verify-publish

### Recursion limit reached (LangGraph)
_Area: Agent runtime_

**Why:** The agent's graph kept looping (often a tool that keeps failing or a missing stop condition).

**Fix:**
- Open the run's trace and look for the step that repeats (the compare view groups repeated tool calls).
- Add a stop condition or a max-retries guard for that tool.
- Only raise recursion_limit after the loop is fixed.

### Possible secrets in telemetry
_Area: Security_

**Why:** Values that look like keys, tokens or passwords were sent to Smartly in a trace.

**Fix:**
- Rotate any real credential.
- Wrap payloads with a redact() helper: the agent's Setup doctor generates one for you.

See: https://connect.smartlylabs.ai/docs#security

## Glossary

- **Run**: one agent turn: one input, one output and everything between. Has a unique run ID.
- **Session**: related runs, such as a multi-turn conversation, sharing a session ID.
- **Event**: one recorded step inside a run.
- **Trace**: the ordered record of every event in a run.
- **Evaluator**: a qualified, approved person who tests and evaluates your agent.
- **Evaluation**: an evaluator's feedback, tied to evidence from a specific run.
- **Version**: an identifiable state of your agent, for before-and-after comparison.

## Contact

Questions? Email ojas@smartlylabs.ai.