Skip to content
AssemblyAI Agent SDK

Evals

A test settles what your code does. An eval settles what the agent did: whether the model reached for the right tool, with the arguments the caller actually said, and answered with what came back.

Terminal window
aai eval          # agent.eval.test.ts

An eval is an ordinary vitest file. Everything in it is real — your prompt, your tools, the session’s own event stream — except the microphone and the speaker.

Your first eval

Section titled “Your first eval”
import { describeEval } from "@alexkroman1/aai-runtime/testing/vitest";
import agentDef from "virtual:aai/agent";
import { expect } from "vitest";


describeEval(agentDef, (test) => {
  test(
    "answers in its own voice",
    async ({ session }) => {
      const turn = await session.say("What is the capital of France?");
      expect(turn.text).toMatch(/paris/i);
    },
    { stubReply: "Paris is the capital of France." },
  );
});

Three things are going on:

  • describeEval opens a session for each case and closes it afterwards, so the body is its assertions and nothing else.
  • session.say() hands back that turn — the reply, its tool calls, its events.
  • stubReply is what a scripted model answers with when the suite runs without a provider key. With a key, the real model answers instead. See Live model, or scripted.

Import the agent from virtual:aai/agent, not from ./agent.ts. That is the agent as aai build lowers it, with tools/ discovered and system-prompt.md applied — the same rule as in Testing.

Asserting the agent used a tool

Section titled “Asserting the agent used a tool”

The reply is half of it. The other half is what the agent did before it spoke:

import { describeEval } from "@alexkroman1/aai-runtime/testing/vitest";
import agentDef from "virtual:aai/agent";
import { expect } from "vitest";


describeEval(agentDef, (test) => {
  test(
    "looks the order up before answering",
    async ({ session }) => {
      const turn = await session.say("Where is order W1234?");


      expect(turn.toolCalls.map((call) => call.name)).toContain("look_up_order");
      expect(turn.text).toMatch(/shipped/i);
    },
    {
      stubReply: [
        { tool: "look_up_order", args: { order_id: "W1234" } },
        "Order W1234 shipped yesterday.",
      ],
    },
  );
});

A stubReply naming a tool the agent does not declare fails at declaration, listing the tools it does declare.

What a turn hands back

Section titled “What a turn hands back”
turn.text The committed reply, joined — what the caller was told
turn.toolCalls This turn’s calls, in call order, each with its result
turn.completed The reply ended on its own terms rather than being cancelled
turn.events This turn’s events, from the utterance to the terminator
turn.errors The error.reported events this turn carried

The session answers the same questions over the whole conversation: session.events(), session.toolCalls(), and session.said() — which includes the greeting, since the agent’s opening line is a real turn. session.id is what a tool reads as ctx.sessionId.

Prefer the turn. A claim about one reply should not be able to pass because of a different one.

Live model, or scripted

Section titled “Live model, or scripted”

describeEval picks the model for you, and prints which it picked on every run.

Live Scripted
You get it when a provider key is in the environment there is no key
Replies come from the real model the case’s stubReply
A case costs tokens, and a few seconds nothing
A green run proves the agent chose the right tool and said the right thing, this once the agent boots, tools/ resolves, and a tool the script names really runs

A scripted run is a wiring check. The agent is real, so a green one proves it still starts and still has its tools — and it says nothing about what the agent chose, because you wrote the choice.

A live run does speak to the choice, but it is a noisy instrument. One failure is a question, not a verdict.

Writing a stubReply

Section titled “Writing a stubReply”

A bare string is a line the agent says. An array is a sequence — one entry per model call, { tool, args } for a call, the last line repeating.

Choose a stubReply the case’s own assertions still hold against. The point of a scripted run is that the case really executes, and a stub the case then fails against measures nothing.

Cases that only make sense in one mode

Section titled “Cases that only make sense in one mode”
Marker Skipped when Reach for it when
{ live: true } scripted no script can satisfy the claim: a tool the model has to choose for itself, a refusal, a judgement
{ scripted: true } live a competent model will not take the path — usually watching a guard refuse, since something has to call the gated tool before you can see it say no 
import { describeEval } from "@alexkroman1/aai-runtime/testing/vitest";
import agentDef from "virtual:aai/agent";
import { expect } from "vitest";


describeEval(agentDef, (test) => {
  test(
    "refuses a size the kitchen does not make",
    async ({ session }) => {
      const turn = await session.say("Can I get a five foot pizza?");
      expect(turn.text).not.toMatch(/\bfive foot\b/i);
    },
    // No script can honestly stand in for a judgement, so skip this one
    // rather than let it pass against a reply you wrote yourself.
    { live: true },
  );
});