Testing

A test settles what your code does. An eval settles what the agent did. This page is the first one.

aai test

aai test runs every spec in the project with vitest — everything but the evals, which have their own command. A tool is a plain function, so a spec calls it and asserts on what comes back. No model, no session, no server.

A spec imports its testing names from two doors: @alexkroman1/aai-runtime/testing for everything that installs nothing, and @alexkroman1/aai-runtime/testing/vitest for everything that installs or restores (the install* fakes) and the eval suites.

Your first test

toolRunner runs a tool by its def — the tool file’s default export — gated exactly as the agent gates the model’s own call, and typed by what the tool’s execute returns:

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "The weather in a city",
  inputSchema: z.object({ city: z.string() }),
  execute: ({ city }) => ({ city, tempF: 61, conditions: "clear" }),
});

// file: agent.test.ts
import agentDef from "virtual:aai/agent";
import { toolRunner } from "@alexkroman1/aai-runtime/testing";
import { expect, test } from "vitest";
import getWeather from "./tools/get_weather.ts";

const run = toolRunner(agentDef);

test("get_weather answers for the city it was given", async () => {
  const out = await run(getWeather, { city: "Denver" });
  expect(out.city).toBe("Denver");
});

That is a complete spec. Put it in agent.test.ts and run aai test. A tool is always reached by its def, never by its name string: a name is a string, and nothing can type what it looks up.

Assert on what the tool returns, not on what the API it called returned. The scaffold’s get_weather narrows wttr.in’s response to { city, tempF, conditions } before handing it back, so those are the fields a spec names — see Tools.

Import the agent from `virtual:aai/agent`

A tool is a file, so agent.ts’s own default export carries no tools at all. tools/get_weather.ts becomes the tool get_weather when aai build lowers the project, and virtual:aai/agent is that lowered agent — your tools/ directory discovered, your system-prompt.md applied.

Hand a runner the authored def instead and it says so on the spot: a runner over zero tools can only ever be this mistake.

Checking the agent is shippable

expectDeployable checks that an agent is actually deployable, and names what is missing when it is not:

import agentDef from "virtual:aai/agent";
import { expectDeployable } from "@alexkroman1/aai-runtime/testing";
import { expect, test } from "vitest";

test("is deployable", () => {
  expect(() => expectDeployable(agentDef)).not.toThrow();
});

Giving a tool a context

Each call above got a fresh ctx of its own. Pass one explicitly when a tool needs something from it, or when two calls are supposed to share a session:

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Add an item to the cart",
  inputSchema: z.object({ sku: z.string(), qty: z.number() }),
  execute: ({ sku, qty }) => ({ sku, qty }),
});

// file: tools/list_cart.ts
import { tool } from "@alexkroman1/aai";

export default tool({
  description: "List the cart",
  execute: () => [] as { sku: string; qty: number }[],
});

// file: agent.test.ts
import agentDef from "virtual:aai/agent";
import { createToolContext, toolRunner } from "@alexkroman1/aai-runtime/testing";
import { expect, test } from "vitest";
import addToCart from "./tools/add_to_cart.ts";
import listCart from "./tools/list_cart.ts";

const run = toolRunner(agentDef);

test("the cart belongs to one caller", async () => {
  const alice = createToolContext();
  const bob = createToolContext();

  await run(addToCart, { sku: "A1", qty: 1 }, alice);
  expect(await run(listCart, bob)).toEqual([]);
});

createToolContext() is a fake ctx, and nothing it does escapes the test. It gets its own detached slot store, so two contexts are two callers. Pass overrides for whatever the tool actually uses. ctx.generate and ctx.delegate reject until you do, naming themselves, so a tool that makes a model call tells you rather than silently passing.

Stubbing the model

Tools that call ctx.generate take a stub rather than a live key:

import { createToolContext } from "@alexkroman1/aai-runtime/testing";

const ctx = createToolContext({ generate: { reply: "A short summary." } });
// `ctx.model.calls` records what the model was asked.
// `ctx.desk` is the same thing for `ctx.delegate`.

A script names its shape — one reply for every call, or routes keyed by system prompt:

Pass	For
`{ reply: "…" }`	a text answer
`{ reply: { object: … } }`	structured output. Add `text` when the tool reads both
`{ reply: (call) => … }`	an answer computed from the call — a queue, say
`{ routes: { [system]: … } }`	a tool that plays more than one model role

delegate takes the same two shapes, with routes keyed by subagent name.

createToolContext hands the two fakes back on ctx.model and ctx.desk.

The full set — stubGateway, guardrails, workflow contexts, upload fixtures, run snapshots — is in the SDK reference under @alexkroman1/aai-runtime/testing.

Running the suite

aai test          # every spec in the project
aai test --only   # agent.test.ts alone, for the fast inner loop

--only names the spec files it skipped rather than reporting a green run over them, so a pass never hides an untested file.

aai build runs your whole suite before it bundles. aai publish only type-checks, so run aai build first if you want the tests to gate a ship.

Behaviour evals are a separate command, aai eval, and neither run above reaches them.

What a test cannot settle

Everything here runs your code without a model. That settles what a tool does with the arguments it is given. It says nothing about whether the agent reached for that tool, with the arguments the caller actually said.

For that there is a second command:

aai eval          # agent.eval.test.ts

An eval drives a real session — your prompt, your tools, the real event stream, with only the microphone and the speaker faked — and asserts on what the agent did. See Evals.

Evals — driving a real session, and what a green run means
Run it locally — aai dev, then a plain Node process
Publish — ship it, and where your secrets go