testing

@alexkroman1/aai/testing — framework-agnostic test helpers for agent code.

A FACADE. The subpath resolves here rather than at testing.ts, which buys two things the direct form could not. That module can be SPLIT as it grows without moving the published entry point — the path an implementation file happens to have is not a thing to promise anyone — and a name it gains next reaches the public surface only when a line is added below, rather than the moment it is written.

Named re-exports rather than export * for the second half of that: the wildcard form re-exports whatever arrives, and needs a noReExportAll suppression the escape-hatch ratchet only lets move down.

An agent project’s test files import these through @alexkroman1/aai-runtime/testing, which re-exports every name here as the same declaration alongside the runtime’s own drivers — one door per kind of helper, whatever package declares it. This subpath keeps working; it is where the names are declared.

Functions

createRecordingWorkflows()

createRecordingWorkflows(options?): RecordingWorkflows

Build a RecordingWorkflows.

Parameters

options?

RecordingWorkflowsOptions

Returns

RecordingWorkflows

Example

import { createRecordingWorkflows, createRunSnapshot } from "@alexkroman1/aai/testing";

const workflows = createRecordingWorkflows({
  runs: [createRunSnapshot({ workflow: "remind", key: "kitchen", runId: "wrun_pending" })],
});
await workflows.start("remind", { text: "flip the laundry" }, { key: "kitchen" });
console.log(workflows.started("remind").length); // 1
console.log((await workflows.find("remind", "kitchen")).length); // 2

createRouteContext()

createRouteContext(overrides?): RouteContext

Build a RouteContext for calling a route handler in isolation.

Defaults are inert: an empty env, a workflows client whose every method rejects naming itself (createStubWorkflows), a clientTranscript that answers no sessions, a sessionTranscript that answers no lines, a speech that finds no live session, and a signal that never aborts. Override any of them — most specs pass env and a workflows fake of their own.

A hand-built literal has to restate every field, and it is the field the runtime gained LAST that it leaves out: five shipped specs each carried one, and sessionTranscript was missing from all five.

Parameters

overrides?

RouteContextOverrides

Returns

RouteContext

Example

import { type RouteHandler, type RouteRequest, routeError } from "@alexkroman1/aai";
import { createRouteContext, createStubWorkflows } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";

const getCall: RouteHandler = async (req, ctx) => {
  const [run] = await ctx.workflows.find("call", req.params.id ?? "", { limit: 1 });
  if (run === undefined) throw routeError(404, `No call ${req.params.id}.`);
  return run;
};

const request: RouteRequest = {
  method: "GET",
  path: "/calls/c_1",
  params: { id: "c_1" },
  query: {},
  headers: {},
  body: undefined,
};

test("GET /calls/:id is a 404 for a call nobody placed", async () => {
  const workflows = createStubWorkflows({ find: () => Promise.resolve([]) });
  const ctx = createRouteContext({ env: { CALLS_API_KEY: "k" }, workflows });
  await expect(getCall(request, ctx)).rejects.toMatchObject({ status: 404 });
});

createRunSnapshot()

Call Signature

createRunSnapshot<R>(overrides?): WorkflowRunSnapshot<R>

Build a WorkflowRunSnapshot — the right arm of the union, without a cast.

Defaults to a running run, which is the state a tool that has just started one reads back.

Type Parameters

R

R = unknown

Parameters

overrides?

RunSnapshotOverrides<R>

Returns

WorkflowRunSnapshot<R>

Example

import { createRunSnapshot, createStubWorkflows } from "@alexkroman1/aai/testing";

const workflows = createStubWorkflows({
  find: () => Promise.resolve([createRunSnapshot({ status: "failed", error: "gateway down" })]),
});

Call Signature

createRunSnapshot<R>(workflow, overrides?): WorkflowRunSnapshot<R>

Build a WorkflowRunSnapshot of a run of workflow, with output TYPED by the def.

The def carries R — what its body returns, or what its output schema declares — so createRunSnapshot(recap, { status: "completed", output }) checks output against it, where the bare form takes whatever R the caller names. A fixture whose output stopped matching the body is then a compile error in the spec rather than a page that renders undefined.

The def names the run too, when it can: workflow is the key the def was declared under (workflowNameOf, recorded when the agent is lowered — a virtual:aai/agent import, withWorkflows, workflowRegistry), the overrides’ workflow winning, and "workflow" for a def no agent declares.

Type Parameters

R

R

Parameters

Example

import { workflow } from "@alexkroman1/aai";
import { z } from "zod";

export const recap = workflow({
  description: "Write up a recording",
  input: z.object({ recording: z.string() }),
  output: z.object({ headline: z.string(), points: z.array(z.string()) }),
  run: async () => ({ headline: "…", points: [] }),
});

// file: recap.test.ts
import { createRunSnapshot } from "@alexkroman1/aai/testing";
import { recap } from "./shared.ts";

const done = createRunSnapshot(recap, {
  status: "completed",
  output: { headline: "…", points: [] }, // typed by `recap`
}); // `workflow` is "recap" once the agent declaring it has been imported

createSlotStore()

createSlotStore(): SlotStore

A fresh, empty, REAL slot store — one session’s, with no backend: a durable write is checked and frozen exactly as a deployed session’s is.

What createToolContext() puts on ctx.slots by default; named so a spec can hand ONE store to two contexts (a tool and an event handler sharing a session), or build a SlotHolder without a whole context.

Returns

SlotStore

Example

import { createSlotStore, createToolContext } from "@alexkroman1/aai/testing";

const slots = createSlotStore();
const first = createToolContext({ slots });
const second = createToolContext({ slots, sessionId: first.sessionId });

createStubWorkflows()

createStubWorkflows(overrides?): WorkflowClient

A ctx.workflows for testing a tool that starts or reads durable runs: every method rejects by default, and overrides replaces the ones the test drives.

The alternative is a cast, and the cast is what goes wrong. A complete WorkflowClient is eight methods, of which a tool’s test usually drives one or two, so the hand-rolled version is a literal with as WorkflowClient — which keeps compiling when the client GAINS a method and leaves that method undefined. Two shipped templates had exactly that, and adding wakeUp and stream to the client is what surfaced it: the casts still compiled.

Rejecting rather than no-op defaults, for the reason createUnusedDb rejected before it went away with ctx.db — a tool that reaches for a method the test did not stub should say so, not silently receive undefined. listing is the exception and returns [], because it is synchronous and an empty list is a truthful answer.

import { createStubWorkflows, createToolContext } from "@alexkroman1/aai/testing";

const workflows = createStubWorkflows({ start: async () => "wrun_1" });
const ctx = createToolContext({ workflows });

Parameters

overrides?

Partial<WorkflowClient>

Returns

WorkflowClient

createToolContext()

createToolContext(overrides?): TestToolContext

Build a ToolContext for testing a tool’s execute in isolation.

Defaults are chosen so the context is inert: empty env, an empty slot store, workflows, generate and delegate that reject with a message naming themselves, a signal that never aborts, and a send that records. Override any of them.

generate and delegate also take a SCRIPT, which is the way in for a tool that calls a model: pass stubGenerate’s own argument and the fake is built here, installed, and handed back on ctx.model (ctx.desk for delegate).

Each call is a distinct session. sessionId auto-increments, which is what makes the two-context isolation test — the same tool run against two contexts must not share state — read the way it does. Pass sessionId explicitly when a test needs two contexts to be the SAME session (a reconnect, a keyed lock).

An override may be undefined, which means “I do not have one” and leaves the default in place — see ToolContextOverrides for why that is not Partial<ToolContext>.

There is no state type parameter, because there is no ctx.state bag to type: a slot types its own value in the module that declares it, and reading the slot back is how a spec asserts what a tool wrote.

Parameters

overrides?

ToolContextOverrides

Returns

TestToolContext

Examples

import { sessionSlot } from "@alexkroman1/aai";

export const cartSlot = sessionSlot("cart", () => ({ items: [] as string[] }));

// file: tools/add_item.ts
import { z } from "zod";
import { cartSlot } from "../shared.ts";

export default cartSlot.updateTool({
  description: "Add an item to the cart",
  inputSchema: z.object({ item: z.string() }),
  execute: ({ item }, cart) => {
    cart.items.push(item);
    return { count: cart.items.length };
  },
});

// file: agent.test.ts
import { createToolContext } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import { cartSlot } from "./shared.ts";
import addItem from "./tools/add_item.ts";

test("add_item appends to this session's cart", async () => {
  const ctx = createToolContext();
  await addItem.execute({ item: "apple" }, ctx);
  expect(cartSlot.get(ctx).items).toEqual(["apple"]);
});

Asserting on what a tool sent

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Recommend something for a mood",
  inputSchema: z.object({ mood: z.string() }),
  execute: ({ mood }, ctx) => {
    ctx.send("recommendations", { mood, picks: ["a walk by the river"] });
    return { picks: 1 };
  },
});

// file: agent.test.ts
import { createToolContext } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import recommend from "./tools/recommend.ts";

test("recommend pushes its picks to the client", async () => {
  const ctx = createToolContext();
  await recommend.execute({ mood: "chill" }, ctx);
  expect(ctx.sent).toEqual([{ event: "recommendations", data: expect.anything() }]);
});

Scripting the model in the same call

import { createToolContext } from "@alexkroman1/aai/testing";

// `{ reply }` answers every call; `{ routes }`, keyed by system prompt,
// answers a tool that plays more than one model role.
const ctx = createToolContext({ generate: { reply: "A short summary." } });
// … run the tool, then assert on what it asked:
// expect(ctx.model.calls.map((call) => call.prompt)).toEqual([…]);

createWorkflowContext()

createWorkflowContext(options?): WorkflowContextRecorder

Build a WorkflowContext that runs a body and records what it asked for.

Parameters

options?

WorkflowContextOptions

Returns

WorkflowContextRecorder

Example

import type { WorkflowContext } from "@alexkroman1/aai";

export async function digestFlow(input: { url: string }, ctx: WorkflowContext) {
  const article = await ctx.step("fetchArticle", () => `the article at ${input.url}`);
  const headline = await ctx.step("summarize", () => article.slice(0, 11));
  await ctx.sleep("settle", 10_000);
  await ctx.step("file", () => undefined);
  return { headline };
}

// file: digest.test.ts
import { createWorkflowContext } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import { digestFlow } from "./workflows/digest.ts";

test("digest runs its steps in order and sleeps once", async () => {
  const ctx = createWorkflowContext();
  const output = await digestFlow({ url: "https://example.com/a" }, ctx);

  expect(output.headline).toBe("the article");
  expect(ctx.steps.map((s) => s.name)).toEqual(["fetchArticle", "summarize", "file"]);
  expect(ctx.slept).toEqual([{ label: "settle", until: 10_000 }]);
});

deployedAgent()

deployedAgent<D>(authored, project): D

The def a DEPLOYED agent runs: the one agent.ts exports, plus the tools its tools/ directory declares, plus what its system-prompt.md says.

This is one call because forgetting HALF of it is the failure it exists to prevent, and that failure is silent. Neither lowering is applied by agent() — both are applied by the BUILD (aai build enumerates tools/ and resolves the prompt file) — so a spec or an eval driving the raw default export measures an agent with NO TOOLS and the FRAMEWORK-DEFAULT system prompt. Nothing fails: the model answers plausibly out of its own knowledge, every case that asserts a sentence still passes, and the suite reports green on a different agent than the one anybody deploys. It produced four bogus green eval results in one day, and the two nested wrappers it replaces — a tools lowering inside a prompt lowering, written out in seventeen template evals — are exactly the shape where one of the two goes missing under an edit.

Under vitest, prefer import agentDef from "virtual:aai/agent", which is this call made for you against the importing spec’s own directory (see the module doc). Reach for this one when the runner is not vitest, or when the lowering itself is the subject of the spec.

An EMPTY tools glob throws. That is the same bug wearing its other face: import.meta.glob("./tool/*.ts") (or a tools/ directory that moved) matches nothing, and lowering nothing onto the def is indistinguishable from not lowering at all. A project with no tools omits the field instead, which is a statement rather than an accident.

import { agent } from "@alexkroman1/aai";

export default agent({ name: "Notes" });

// file: tools/add_note.ts
import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Save a note",
  inputSchema: z.object({ text: z.string() }),
  execute: ({ text }) => ({ saved: text.length }),
});

// file: agent.test.ts
import { deployedAgent } from "@alexkroman1/aai/testing";
import authored from "./agent.ts";
import systemPrompt from "./system-prompt.md?raw";

const agentDef = deployedAgent(authored, {
  tools: import.meta.glob("./tools/*.ts", { eager: true }),
  systemPrompt,
});

Every rule the build applies applies here too, and each is an error naming the file: the tool-name grammar, the default-export requirement, no nested files, a name declared twice, an empty prompt file, and a system-prompt.md that exists while the def carries a different prompt STRING (a raw config’s, or a host’s — agent() refuses the key) — the “I edited the prompt and nothing changed” failure.

Bounded by the two fields it lowers ONTO, not by AgentDef. An agent() def satisfies it and comes back as its own type, so a template keeps its exported workflow types; the bound says what the function reads, and keeps AgentDef and everything behind it off @alexkroman1/aai/testing’s contract.

Type Parameters

D

D extends ToolBearingAgent & object

Parameters

authored

D

project

ProjectFiles

Returns

D

dialogRefusalPattern()

dialogRefusalPattern(state?): RegExp

A pattern matching the sentence a dialog() gate refuses with — optionally pinned to the state it names.

For a SPEC. A gated tool called out of state answers a ToolFailure whose error is this sentence, and every template spec that asserts a gate held used to spell a regex for it by hand. Two kinds of spec read it, and the pattern serves both: a unit test holds the ToolFailure itself (prefer expectDialogRefused there, which also throws on a success), while an eval reads a tool result off the event stream as a SERIALIZED string, where the state’s quotes arrive escaped (\"identifying\"). The pattern admits the escaping, so one matcher reads both.

With no state, it matches any refusal — for a spec that pins the state a line later, or whose subject is that the body did not run rather than where the conversation was.

Parameters

state?

string

The state the refusal must name, as DialogPosition.state spells it ("identifying", "onCall.inbox"). Matched literally.

Returns

RegExp

Example

import { dialogRefusalPattern } from "@alexkroman1/aai/testing";

const refused = 'Not available yet: this conversation is at "identifying". Verify the caller first.';
dialogRefusalPattern("identifying").test(refused); // true
dialogRefusalPattern("transferred").test(refused); // false
dialogRefusalPattern().test(refused); // true

dialogResultSchema()

dialogResultSchema<T>(result): ZodObject<{ done: ZodBoolean; instruction: ZodOptional<ZodString>; result: T; state: ZodString; }, $strip>

The envelope a gated tool answers with, as a schema around the tool’s own.

expectDialogOk unwraps a value a spec HOLDS. A harness that holds the SERIALIZED copy the model was handed — a transcript, a recording — reads it back through a schema, so it needs the same envelope as a schema rather than as a function, and three shipped evals had each written it out: z.object({ result, state: z.string(), done: z.boolean() }), under a comment saying the shape was the SDK’s. It is, and this is where it lives (an eval reading a live turn takes the DEF instead: toolResultIn(turn.toolCalls, setStay) is typed by it). result is whatever the author’s execute returned, state is where the call landed, done whether that state is final, and instruction is the state’s own brief when it declares one — the fields of DialogToolResult, which a dialog.tool writes and no tool file does.

Parsing rather than casting is what makes a template that stopped carrying its position fail naming the field, instead of a later expect reading undefined.state.

Type Parameters

T

T extends ZodType<unknown, unknown, $ZodTypeInternals<unknown, unknown>>

The schema of the tool’s OWN result, under result.

Parameters

result

T

What the tool’s execute answers with.

Returns

ZodObject<{ done: ZodBoolean; instruction: ZodOptional<ZodString>; result: T; state: ZodString; }, $strip>

Example

import { dialogResultSchema } from "@alexkroman1/aai/testing";
import { z } from "zod";

// Holding the serialized result yourself, it is one parse:
const Stay = dialogResultSchema(z.object({ options: z.string() }));
const stay = Stay.parse(
  JSON.parse('{"result":{"options":"garden view"},"state":"booking.room","done":false}'),
);
stay.state; // "booking.room"
stay.result.options; // "garden view"

endSessionCalls()

endSessionCalls(ctx): readonly object[]

Every endSession(ctx, …) a tool made on a createToolContext context’s session, in call order, with its options resolved (afterReply defaults to true). Empty when the tool never ended the session.

A function of the context rather than a field on TestToolContext, for the reason endSession is one: it reads the session, which ctx.sessionId names.

import { endSession, tool } from "@alexkroman1/aai";
import { createToolContext, endSessionCalls } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import { z } from "zod";

const endCall = tool({
  description: "Hang up.",
  inputSchema: z.object({}),
  execute: (_args, ctx) => ({ ended: endSession(ctx) }),
});

test("end_call hangs up after the goodbye", async () => {
  const ctx = createToolContext();
  await endCall.execute({}, ctx);
  expect(endSessionCalls(ctx)).toEqual([{ afterReply: true }]);
});

Parameters

ctx

Pick<ToolContext, "sessionId">

Returns

readonly object[]

eventsOf()

eventsOf<E, K>(events, type): Extract<E, { type: K; }>[]

Every event in events named type, in order, typed as that member.

import type { SessionEvent } from "@alexkroman1/aai";
import { eventsOf } from "@alexkroman1/aai/testing";

declare const recorded: SessionEvent[];
const calls = eventsOf(recorded, "tool.called");
console.log(calls.map((e) => e.toolName));

Type Parameters

E

E extends object

K

K extends string

Parameters

events

Iterable<E>

type

K

Returns

Extract<E, { type: K; }>[]

expectDeployable()

expectDeployable<D>(def): DeployedConfig

Run the invariants a deployable agent owes, and hand back the RESOLVED config so a spec can go on to assert its own specifics — a chosen model, a declared builtin — without converting twice.

Three invariants, each thrown by name:

The config passes manifest validation — the same toAgentConfig that aai build and aai deploy run, so an invalid provider combination or tuning fails here rather than at the first live session.
The platform can name it — there IS a name, and the conversion carries it through. Not the literal: renaming the agent is the first edit a starter invites, and the studio lists a deployed agent by exactly this string.
Every stage its mode needs is filled, declared or defaulted — asserted per MODE so it survives a swap. A pipeline agent has an stt, llm and tts kind, each declared stage surviving as declared and each unset one filled by defaultProviders; a text agent has no audio stage (its llm may be absent — createTextAgent defaults the one stage it has); an s2s agent has an s2s kind and NO cascade, since speech-to-speech replaces the pipeline rather than joining it — the one thing that must never happen by fallthrough.

toAgentConfig already refuses most of the states the second and third invariants describe (a blank name, s2s beside a pipeline stage). They are checked here anyway, and BEFORE or AFTER the conversion as the message needs, because the value of this helper is the sentence: a spec that failed on “expected function not to throw” has to re-run the conversion by hand to learn which invariant went.

import { agent } from "@alexkroman1/aai";
import { expectDeployable } from "@alexkroman1/aai/testing";

const config = expectDeployable(agent({ name: "Desk", builtinTools: ["run_code"] }));
// The invariants held; now the template's own claim.
console.log(config.builtinTools); // ["run_code"]

Type Parameters

D

D extends object

Parameters

def

D

The agent under test — an agent() definition, or the raw default export of an agent.ts. Structural: only name is required of the TYPE, because validating the rest is this helper’s job at run time — the same toAgentConfig a deploy runs. Generic only so a spread literal carrying a field this type does not name ({ ...def, maxSteps: 0 }) is not an excess-property error.

Returns

DeployedConfig

The config a deploy carries, mode derived and defaults injected — see DeployedConfig for the fields it names.

Throws

Naming the invariant that failed, and — for the validation one — the sentence toAgentConfig wrote about the field.

expectDialogOk()

expectDialogOk<R>(result): [Exclude<R, ToolFailure>] extends [DialogToolResult<V>] ? DialogToolResult<V> : DialogToolResult<unknown>

The dialog envelope a gated tool answered, keeping WHERE the dialog landed.

The half a spec needs when the assertion is about the conversation rather than about the tool’s own value — that a call advanced the machine into quote.pending, that a final state reports done. Unlike expectToolOk, which passes a plain tool’s value through, this THROWS on anything that is not a dialog envelope: keeping a position claims there is one.

Typed by INFERENCE, as expectToolOk is: handed the envelope a run answered (runTool(theTool, …), a runner’s run(theTool, …), theTool.execute(…)) it keeps the type under result. There is no claimed form.

Type Parameters

R

R

What was handed in, inferred — never written.

Parameters

result

R

Returns

[Exclude<R, ToolFailure>] extends [DialogToolResult<V>] ? DialogToolResult<V> : DialogToolResult<unknown>

Throws

When the tool refused, quoting the refusal, as expectToolOk does.

Throws

When the value is not a dialog envelope — a plain tool() has no position to keep; use expectToolOk for it.

Example

import { tool } from "@alexkroman1/aai";

// In a project this is a `dialog.tool(...)`, whose envelope the dialog writes.
export default tool({
  description: "Quote the job",
  execute: () => ({ result: { quoted: 42 }, state: "quote.pending", done: false }),
});

// file: agent.test.ts
import agentDef from "virtual:aai/agent";
import { createToolContext, expectDialogOk, toolRunner } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import quote from "./tools/quote.ts";

const run = toolRunner(agentDef);

test("quote moves the dialog on", async () => {
  const ctx = createToolContext();
  const answered = expectDialogOk(await run(quote, ctx));
  expect(answered.state).toBe("quote.pending");
  expect(answered.result.quoted).toBe(42);
});

expectDialogRefused()

expectDialogRefused(result, state?): ToolFailure

The refusal a gated tool answered with, or a throw saying the gate did NOT hold.

The mirror of expectDialogOk, for the spec whose subject is that a tool was REFUSED: called before the dialog reached its state, or after it left. Six template specs had written the other half by hand — an isToolFailure check, a toBe(true), and a regex for the sentence the gate writes — and a success slipped through that shape as three assertions that never ran, because each sat inside the if the guard opened.

With a state, the refusal must also NAME it: that the tool was refused is half the claim, and that the conversation was where the spec thinks it was is the half a gate on the wrong state hides in. Matched with dialogRefusalPattern, so a spec never spells the sentence.

Parameters

result

unknown

What runTool / toolOf(...).execute(...) answered.

state?

string

The position the refusal must name, as DialogPosition.state spells it. Omit to accept a refusal at any state.

Returns

ToolFailure

Throws

When the tool was NOT refused — a dialog envelope is reported with the state it landed in, since that is the fact the spec got wrong.

Throws

When it was refused for some other reason, or at some other state, quoting the refusal.

Example

import { expectDialogRefused } from "@alexkroman1/aai/testing";

const refused = expectDialogRefused(
  { error: 'Not available yet: this conversation is at "idle". Call start_plan first.' },
  "idle",
);
refused.error.includes("start_plan"); // true — the instruction the model recovers from

expectPromptBuiltinsDeclared()

expectPromptBuiltinsDeclared(def): BuiltinTool[]

Every builtin the prompt commands is one builtinTools declares — or a throw naming the ones that are not.

The pairing a prompt-driven template is made of: the prose holds the rules (“you MUST use run_code for arithmetic”, “look rates up with fetch_json”), and agent.ts holds the array that makes those tools exist. The failure is silent in both directions and shows up in a diff of neither file — a prompt commanding fetch_json at an agent that never declared it produces a model apologizing for a tool it cannot see, and a builtin dropped from agent.ts alone leaves an endpoint list addressed to nothing.

A prompt commanding NO builtin is a failure, not a pass. Non-vacuity earns its keep twice: a loop over nothing asserts nothing, and it is also the state a template lands in when system-prompt.md was not applied — the framework default names no builtin, so “I edited the prompt and nothing changed” fails here instead of passing quietly with the template’s rules nowhere in its context. A spec whose prompt legitimately describes its tools rather than naming them does not want this helper; it asserts on builtinTools directly.

The converse is deliberately NOT asserted: declaring a builtin the prompt never mentions is an ordinary edit, and the model learns about it from its own tool schema rather than from the prose.

A custom tool of the same NAME declares it too. An agent may replace a builtin with its own tools/text_me.ts — a different channel, a different recipient rule — and the prompt’s “text it with text_me” is then addressed to that tool, which the model sees under exactly that name. The claim is “the model has a tool called this”, and def.tools answers it as well as builtinTools does. That is why tools is read, and why a def lowered with deployedAgent (or imported from virtual:aai/agent) is the one to pass: the authored ./agent.ts carries no tools.

A RESOLVER is CALLED, and refused when it cannot be. systemPrompt may be a function, and toAgentConfig drops one rather than putting it on the wire — so scanning the converted config would read the FRAMEWORK’s default prompt and report on a prompt this agent never sends. That is the one outcome a check may not have: the default names no builtin, so scanning it fails for the wrong reason — pointing at an unapplied system-prompt.md that is not the problem — and PASSES the day the default happens to name one. So the resolver is called with a bare createToolContext — a fresh session id, no env, an empty slot store — and its answer is what gets scanned. That is enough for the prose half, which is a ?raw import closed over by the function and does not vary with session state. A resolver that cannot answer from a bare context (it reads an env var, or a slot it expects seeded) THROWS, and this refuses by name rather than falling back to the default: seed a context, call the resolver yourself and pass its text as the systemPrompt of the def this takes, or assert on builtinTools directly.

import { agent } from "@alexkroman1/aai";
import { deployedAgent, expectPromptBuiltinsDeclared } from "@alexkroman1/aai/testing";

// Under vitest, `import agentDef from "virtual:aai/agent"` is this lowering made for you.
const commanded = expectPromptBuiltinsDeclared(
  deployedAgent(agent({ name: "Coda", builtinTools: ["run_code"] }), {
    systemPrompt: "Answer every sum by calling run_code.",
  }),
);
console.log(commanded); // ["run_code"]

Parameters

def

The agent under test — only its systemPrompt, builtinTools and the KEYS of tools are read, so an agent() def passes as it is. Whether the WHOLE def converts is expectDeployable’s claim, not this one’s.

builtinTools?

readonly BuiltinTool[]

systemPrompt?

AgentSystemPrompt

tools?

Readonly<Record<string, unknown>>

Returns

BuiltinTool[]

The commanded builtins, for a spec that wants to say more about them.

Throws

When the prompt names no builtin, when it names one builtinTools lacks, or when a systemPrompt resolver cannot answer from a bare context.

expectSlotContract()

expectSlotContract<K, T, V>(slot, options?): void

Assert the three invariants of slot’s declaration — see the module doc.

Type Parameters

K

K extends string

T

T

V

V

Parameters

slot

SessionSlot<K, T, V>

options?

SlotContractOptions<T>

Returns

void

Throws

Naming the invariant that failed, and for a cap the key.

Example

import { sessionSlot } from "@alexkroman1/aai";
import { expectSlotContract } from "@alexkroman1/aai/testing";

const log = sessionSlot("log", () => ({ lines: [] as string[], total: 0 }), {
  caps: { lines: 50 },
  view: (value) => ({ count: value.lines.length }),
});

expectSlotContract(log, {
  write: (draft) => {
    draft.lines.push("hello");
  },
  caps: { lines: 50 },
});

expectToolOk()

expectToolOk<R>(result): R extends DialogToolResult<V> ? V : Exclude<R, ToolFailure>

What a tool answered, minus the refusal — or a throw quoting the refusal.

Takes ANY tool’s result. A dialog() tool’s envelope (DialogToolResult) is unwrapped to the author’s own value under result; a plain tool()’s value comes back as it is. Either way a ToolFailure throws HERE, naming it, rather than as an undefined read off the failure several assertions later.

Typed by INFERENCE: handed a result — runTool(theTool, …), a runner’s run(theTool, …), or theTool.execute(…) directly — it answers that type minus ToolFailure (for a dialog tool, the type under result), so no type argument is written. There is no claimed form: a result is typed by the def that produced it, and expectToolOk<Order>(x) compiles only when x already IS an Order.

Use expectDialogOk to keep WHERE a dialog landed, and expectDialogRefused when the refusal is the subject.

Type Parameters

R

R

What was handed in, inferred — never written.

Parameters

result

R

What a tool’s execute, runTool or a toolRunner answered.

Returns

R extends DialogToolResult<V> ? V : Exclude<R, ToolFailure>

Throws

When the tool refused (ToolFailure), quoting the refusal — which for a dialog() tool is the sentence naming the state the conversation is actually in and what has to happen first.

Example

import { tool } from "@alexkroman1/aai";
import { expectToolOk, runTool } from "@alexkroman1/aai/testing";
import { toolFailure } from "@alexkroman1/aai/utils";
import { z } from "zod";

// In a spec this is `import placeOrder from "./tools/place_order.ts"`.
const placeOrder = tool({
  description: "Place the order",
  inputSchema: z.object({ item: z.string() }),
  execute: async ({ item }) => (item ? { id: "ord_1" } : toolFailure("Name an item.")),
});
const order = expectToolOk(await runTool(placeOrder, { item: "pizza" }));
console.log(order.id); // typed: the failure arm is subtracted

fireEvent()

fireEvent(agent, event, ctx?): Promise<FiredEvent>

Drive the agent’s dialogs and events handlers with one event, the way the session dispatches it: every declared dialog is offered the event FIRST (dialog.receive, so a "@session.timedOut" transition fires), then the handler declared for event.type, then "*", each handed the same context. Dialogs first because a dialog is part of the session’s state and moves before any observer sees the effect — the order aai-runtime’s session keeps (session-controls.ts).

Two deliberate differences from the runtime’s dispatch, both because a SPEC is reading the outcome rather than a live call surviving it:

A throw PROPAGATES. The session catches a handler’s throw per handler and logs it, so a broken audit hook does not end a phone call; a spec wants the throw, because a handler that threw is the finding.
A promise is AWAITED. The session does not wait on an async handler (the caller is mid-turn); a spec asserting on what the handler wrote has to.

ctx is a SessionEventContext — the four fields a handler reads (sessionId, env, slots, speech) — and a createToolContext() satisfies it, which is the point: a slot the handler wrote is read back with slot.get(ctx) through the same context, and what it said is on ctx.said. Defaults to a fresh one, i.e. a DISTINCT session; pass one explicitly wherever a second event is meant to see the first’s writes.

Parameters

agent

dialogs?

readonly AnyDialog[]

events?

SessionEventHandlers

name?

string

event

SessionEvent

ctx?

SessionEventContext

Returns

Promise<FiredEvent>

Throws

When the agent declares no events handler and no dialogs — an absent events or an empty {} alike — since there is nothing to drive, and a spec that fired into nothing would pass against nothing. A TypeError when event is not a session event (built by sessionEvent).

Example

import { agent, sessionSlot } from "@alexkroman1/aai";
import { createToolContext, fireEvent, sessionEvent } from "@alexkroman1/aai/testing";

const heard = sessionSlot("heard", () => ({ lines: [] as string[] }));
const def = agent({
  name: "Listener",
  events: {
    "userTranscript.committed": (event, ctx) => {
      heard.update(ctx, (draft) => {
        draft.lines.push(event.text);
      });
    },
  },
});

const ctx = createToolContext();
await fireEvent(def, sessionEvent("userTranscript.committed", { text: "hello" }), ctx);
heard.get(ctx).lines; // ["hello"]

isEvent()

isEvent<E, K>(event, type): event is Extract<E, { type: K }>

Whether event is the one named type — a type guard, so the branch it guards reads that member’s fields without a cast.

import type { SessionEvent } from "@alexkroman1/aai";
import { isEvent } from "@alexkroman1/aai/testing";

declare const recorded: SessionEvent[];
const last = recorded.at(-1);
if (last && isEvent(last, "tool.called")) console.log(last.toolName);

Type Parameters

E

E extends object

K

K extends string

Parameters

event

E

type

K

Returns

event is Extract<E, { type: K }>

parseSchemaInput()

parseSchemaInput<T>(schema, value, what?): Promise<T>

Validate value against schema, or throw naming every issue.

Type Parameters

T

T = Record<string, unknown>

What the schema produces. Defaults to Record<string, unknown>, which is what a tool input schema is declared as.

Parameters

schema

StandardSchemaV1<unknown, unknown> | undefined

A Standard Schema, or undefined — the shape tool.inputSchema and workflow.input both have. undefined is an ERROR rather than a pass, because “this declares no schema” is a different fact from “the schema accepted it” and a spec asserting the second must not be satisfied by the first.

value

unknown

what?

string

How the schema is named in a failure. Defaults to "the schema"; pass the tool or workflow name where one is at hand.

Returns

Promise<T>

Throws

When the schema refuses value, with the issues rendered as one line (quantity: too small; size: invalid enum value) — which is what makes the failure readable at all, since a raw issue array prints as [Object].

Example

import { workflow } from "@alexkroman1/aai";
import { parseSchemaInput } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import { z } from "zod";

const myWorkflow = workflow({
  description: "Read a recording aloud",
  input: z.object({ recording: z.string(), voice: z.string().default("jane") }),
  run: async (input) => input,
});

test("the voice survives parsing", async () => {
  const parsed = await parseSchemaInput<{ voice: string }>(myWorkflow.input, {
    recording: "upl_1",
    voice: "jane",
  });
  expect(parsed.voice).toBe("jane");
});

parseToolInput()

parseToolInput<D>(tool, value): Promise<InferToolInput<D>>

Validate value against tool’s input schema, typed by the def.

parseSchemaInput with the schema read off the def — the same def runTool takes and the eval readers match by: the tool file’s default export, a speaker’s tools entry, toolOf(agentDef, name) for a minted tool. The parsed value is InferToolInput<typeof tool>, what execute receives, so a schema that stopped defaulting a field fails at compile time in the spec that reads it, and a name string in the tool position does not compile. No agent is needed: validation is the schema’s alone, and the toolset’s gate (a roster not on the line, a dialog out of state) runs only on a CALL.

A failure names the tool by the name its agent registered it under when one did (Tool add_pizza refused that input: …), else by its description — a def built inline in a spec is as valid a subject here as a lowered one.

Type Parameters

D

D extends ToolDef<ToolInputSchema>

Parameters

tool

D

value

unknown

Returns

Promise<InferToolInput<D>>

Throws

When the tool declares no inputSchema, or when the schema refuses value.

Example

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Add a pizza to the order",
  inputSchema: z.object({ size: z.enum(["small", "large"]), quantity: z.number().default(1) }),
  execute: ({ size, quantity }) => ({ size, quantity }),
});

// file: agent.test.ts
import { parseToolInput } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import addPizza from "./tools/add_pizza.ts";

test("add_pizza defaults the quantity", async () => {
  const parsed = await parseToolInput(addPizza, { size: "small" });
  // The schema's own default, which is the thing worth asserting here — and
  // `parsed` is typed by the def, so a dropped field is a compile error.
  expect(parsed.quantity).toBe(1);
});

routeHandlerOf()

routeHandlerOf(value): FetchRouteHandler

A table’s value as a handler: a handler is itself; a fixed answer is given to every request it is asked, a Response CLONED per request — a body is read once, and the second request would otherwise find it consumed.

Parameters

value

FetchRouteHandler | StubStepAnswer

Returns

FetchRouteHandler

routeOf()

routeOf(routes): FetchRouteHandler

routes as ONE handler — a table matched by specificity, a list tried in order — that DECLINES (undefined) a request no route answers, so it can be one leg of a list or the otherwise of another fake. The same matching stubFetchRoutes does, without installing anything.

Parameters

routes

Readonly<Record<string, FetchRouteHandler | StubStepAnswer>> | readonly FetchRouteHandler[]

Returns

FetchRouteHandler

runGuardrail()

runGuardrail(def, text, answer?): GuardrailVerdict

Run def’s guardrail over one answer and return its verdict.

The answer is text with a ZERO cost report — one step, no tool calls — because that is what most guardrails read; a guardrail that judges the cost (toolCalls.length === 0, say) is handed it through answer, which is spread over the defaults.

Throws when the def declares no guardrail, rather than returning true: a spec calling this is asserting that a check exists, and a def that lost its guardrail should fail here, not pass by default. Throws when the guardrail returns a promise: this helper is for the SYNCHRONOUS guardrail, which is the ordinary one, and an async guardrail’s spec awaits def.guardrail(answer) itself — the verdict is then a promise a test can await, and nothing here would add to that.

Parameters

def

SpeakerDef

text

string

answer?

Partial<DelegateAnswer>

Returns

GuardrailVerdict

Example

import { speaker } from "@alexkroman1/aai";
import { runGuardrail } from "@alexkroman1/aai/testing";

const checker = speaker({
  name: "fact-checker",
  systemPrompt: "Open with Confirmed:, Contradicted: or Unclear:.",
  guardrail: ({ text }) => /^(Confirmed|Contradicted|Unclear):/.test(text) || "Open with a verdict word.",
});

runGuardrail(checker, "Confirmed: the figure is 12%."); // true
runGuardrail(checker, "It seems prices fell."); // "Open with a verdict word."

runTool()

runTool<T>(tool, argsOrCtx?, ctx?): Promise<Awaited<ReturnType<T["execute"]>>>

Run a tool — the tool DEF itself, the way the runtime calls it.

TYPED end to end: the arguments are checked against what execute takes and the result is what it returns, so await runTool(addItem, { item: "apple" }, ctx) needs no cast. A tool FILE’s default export is the very object a deployed agent registers under its name, so importing it runs the same code the model’s call reaches. Matched on execute alone rather than on ToolDef, so any tool shape — tool(), slot.tool(), dialog.tool() — is accepted and keeps its own result type.

UNGATED: this calls the def’s own execute, as a unit test of the tool body wants. The run a spec cares about is usually the GATED one — a dialog out of state refuses it, a roster entry not on the line refuses it — which is what toolRunner binds to the agent, and what describeAgent hands a suite.

args is unvalidated on purpose: the runtime parses a model’s arguments against inputSchema BEFORE execute sees them, so a spec that pre-validated would be testing a path the tool never runs on. Pass the arguments the tool body expects to receive. (To test the SCHEMA itself, which is a different question, use parseToolInput / toolInputIssues.)

A tool that takes no arguments may say so by leaving them out, passing the context in their place: runTool(viewOrder, ctx). A no-argument tool is common — one shipped template has thirteen — and the {} those calls were obliged to pass appeared 66 times across seven template specs, always between the two values a reader actually cares about. The two are told apart by SHAPE, and the probe is narrow enough to be safe: a ToolContext is a record carrying a string sessionId, a slots store and a send function, and tool arguments arrive as JSON from a model, which cannot contain a function. A context is never a plausible argument object.

Type Parameters

T

T extends object

Parameters

tool

T

The tool def: a tools/ file’s default export, a speaker’s tools entry, or a def built in the spec.

argsOrCtx?

ToolContext | Parameters<T["execute"]>[0]

The tool’s arguments, or the ToolContext when it takes none. Defaults to {}.

ctx?

ToolContext

The context. Defaults to a fresh createToolContext — so an omitted context is a DISTINCT SESSION with empty slots, which is what a stateless tool wants and never what two calls sharing state want. Pass one explicitly wherever the second call is supposed to see the first call’s work.

Returns

Promise<Awaited<ReturnType<T["execute"]>>>

Example

import { tool } from "@alexkroman1/aai";
import { createToolContext, runTool } from "@alexkroman1/aai/testing";
import { z } from "zod";

// In a spec this is `import addItem from "./tools/add_item.ts"`.
const addItem = tool({
  description: "Add an item",
  inputSchema: z.object({ item: z.string() }),
  execute: async ({ item }) => ({ added: item }),
});
const { added } = await runTool(addItem, { item: "apple" }, createToolContext());
console.log(added.toUpperCase()); // typed: `added` is a string

// No arguments, one session shared across the two calls.
const viewOrder = tool({ description: "View the order", execute: () => ({ items: [] }) });
const ctx = createToolContext();
await runTool(addItem, { item: "apple" }, ctx);
const { items } = await runTool(viewOrder, ctx);
console.log(items.length);

schemaInputIssues()

schemaInputIssues(schema, value, what?): Promise<readonly StandardSchemaIssue[] | undefined>

The issues schema found in value, or undefined when it accepted it.

The negative half of parseSchemaInput, and undefined-on-success is deliberate: expect(await schemaInputIssues(…)).toBeUndefined() is the accepting case and …toBeDefined() the refusing one, which is the pair every hand-rolled site was already writing against .issues.

Parameters

schema

StandardSchemaV1<unknown, unknown> | undefined

As parseSchemaInput: undefined throws rather than reporting “no issues”, which would make a negative test pass for a schema that does not exist.

value

unknown

what?

string

How the schema is named in that error.

Returns

Promise<readonly StandardSchemaIssue[] | undefined>

Example

import { workflow } from "@alexkroman1/aai";
import { schemaInputIssues } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import { z } from "zod";

const myWorkflow = workflow({
  description: "Read a recording aloud",
  input: z.object({ recording: z.string(), voice: z.enum(["jane", "paul"]) }),
  run: async (input) => input,
});

test("an unknown voice is refused", async () => {
  expect(await schemaInputIssues(myWorkflow.input, { voice: "not-a-voice" })).toBeDefined();
});

sessionEvent()

sessionEvent<K>(type, …rest): SessionEvent<K>

Build a well-formed session event of type, envelope included.

meta.id is minted as evt_test_<n> (the evt_ prefix is what the wire schema requires; the counter is per process and only has to be distinct) and meta.at is a FIXED instant, for the reason createRunSnapshot fixes its createdAt: a fixture that read the clock would not replay. Pass meta to pin either.

The fields are TYPED by the name: sessionEvent("tool.called", { … }) needs toolName, toolCallId and args, and a misspelled field is a compile error here rather than a handler reading undefined. An event that is only its envelope takes no fields.

Type Parameters

K

K extends SessionEventType

Parameters

type

K

rest

…SessionEventRest<K>

Returns

SessionEvent<K>

Example

import { sessionEvent } from "@alexkroman1/aai/testing";

const committed = sessionEvent("userTranscript.committed", { text: "two pizzas" });
committed.type; // "userTranscript.committed"
committed.meta.id.startsWith("evt_"); // true
// `sessionEvent("reply.completed").text` is a compile error: the event has no `text`.

stubClientInbox()

stubClientInbox(options?): StubClientInbox

Publish an inbox whose device records every notice and answers it.

Parameters

options?

StubClientInboxOptions

Returns

StubClientInbox

stubClientTranscript()

stubClientTranscript(answer?): StubClientTranscript

Publish a reader that answers every stepClientTranscript with answer — a fixed transcript, or one computed per call (e.g. honouring the cursor). Omitted, every client has said nothing.

Parameters

answer?

StubClientTranscriptAnswer

Returns

StubClientTranscript

stubDelegate()

stubDelegate(script): StubDelegate

Build a fake ctx.delegate from a script: one reply, or routes keyed by subagent name.

Pass { reply } to answer every delegation the same way, which is what a one-subagent tool wants.

Parameters

script

StubDelegateScript

Returns

StubDelegate

Example

Two subagents, one queue

import { createToolContext, stubDelegate } from "@alexkroman1/aai/testing";

const findings = ["Rain on Tuesday.", "Clear on Wednesday."];
const desk = stubDelegate({
  routes: {
    researcher: () => ({ text: findings.shift() ?? "Nothing found.", steps: 3 }),
    "fact-checker": "Both claims check out.",
  },
});
const ctx = createToolContext({ delegate: desk.delegate });
// … run the tool, then assert on who was asked what:
// expect(desk.calls.map((call) => call.subagent.name)).toEqual([…]);

stubFetchRoutes()

stubFetchRoutes(routes, options?): StubFetchRoutes

Install one router as the global fetch (and, by default, the published step fetch), and return its log. Call restore when the test ends — in a vitest spec, onTestFinished(net.restore) right after the call.

Parameters

routes

Readonly<Record<string, FetchRouteHandler | StubStepAnswer>> | readonly FetchRouteHandler[]

A FetchRouteTable, or a list of handlers tried in order (the first that answers wins, and later ones are not consulted; a handler that throws is left alone). A catch-all leg goes LAST in a list.

options?

FetchRoutesOptions

Returns

StubFetchRoutes

Example

import { stubFetchRoutes } from "@alexkroman1/aai/testing";

const net = stubFetchRoutes({
  "GET supabase.test": (req) => ({ body: req.searchParams.get("id") ? [{ id: 1 }] : [] }),
  "POST supabase.test": { status: 201 },
  "https://api.mem0.ai/v3/memories/": { body: { results: [] } },
});
await fetch("https://supabase.test/rest/v1/calls", { method: "POST", body: "{}" });
console.log(net.to("POST supabase.test").length); // 1
net.restore();

stubGateway()

stubGateway(replies, options?): StubGateway

Build a fake LLM gateway answering replies in order, as a fetch for the caller to install on the GLOBAL fetch.

Three names, one fake, picked by SEAM:

You need	Use
the global `fetch`, installed and undone for you (vitest)	`installStubGateway` (`@alexkroman1/aai/testing/vitest`)
the global `fetch`, installed by you (any runner)	`stubGateway` — this
a published `stepFetch` that other fakes share (a page, a transcription)	stubGatewayRoute, composed into `installStubStepFetch`

The LAST reply repeats once the list runs out, so a spec names only the turns it cares about — which is what makes this usable for a step whose model call sits in a LOOP: a stub that says the same thing every turn can only ever drive such a loop into its budget, and one that runs out mid-loop fails on the stub rather than on the code.

Parameters

replies

string | readonly string[]

Completion contents, in order. A bare string is one reply.

options?

StubGatewayOptions

Returns

StubGateway

Example

import { stepGenerateJson } from "@alexkroman1/aai/step";
import { z } from "zod";

export const summarize = (article: string) =>
  stepGenerateJson(`One headline for: ${article}`, { schema: z.object({ headline: z.string() }) });

// file: digest.test.ts
import { stubGateway } from "@alexkroman1/aai/testing";
import { expect, test, vi } from "vitest";
import { summarize } from "./workflows/digest.ts";

test("summarize sends the article and returns the headline", async () => {
  const gateway = stubGateway(['{"headline":"Otters use tools"}']);
  vi.stubGlobal("fetch", gateway.fetch);
  vi.stubEnv("ASSEMBLYAI_API_KEY", "sk-test");

  expect(await summarize("Otters use tools.")).toEqual({ headline: "Otters use tools" });
  expect(gateway.calls[0]?.prompt).toContain("Otters use tools.");
});

stubGatewayRoute()

stubGatewayRoute(replies, options?): StubGatewayRoute

A gateway reply for a step that goes through the PUBLISHED stepFetch slot rather than the global fetch — a ROUTE to compose, not a fake to install. stubGateway says which of the three gateway fakes fits which seam.

stubGateway answers over globalThis.fetch, which is the wrong seam whenever anything has published a stepFetch: publishing REPLACES, so a flow that transcribes AND calls a model — or fetches a page and calls a model — can install only one fake and has to route by URL inside it. Seven eval files did exactly that, and each hand-typed the same two things:

The envelope. { body: { choices: [{ message: { content } }] } }, written out six times in six spellings. It is a WIRE shape, so a typo in it does not fail — stepGenerate reads no content and reports an empty completion, i.e. the fake and the code under test disagree and the case blames the code.
The cursor. contents.at(Math.min(next, contents.length - 1)), re-derived twice, because a model call inside a LOOP cannot know how many calls it will make: a script that repeats one line can only drive the loop into its budget, and one that runs out mid-loop fails on the script. The last reply repeats, which is stubGateway’s convention and now literally the same code.

And it hands back DECODED calls — prompt, system, body, headers — which is the half no hand-rolled version had. Reading what the model was ASKED off a StubStepRequest means String(call.body), i.e. the raw JSON of the whole request; one eval asserted its prompts that way and was really asserting against the serialized model and temperature too.

import { stubGatewayRoute } from "@alexkroman1/aai/testing";
import { installStubStepFetch } from "@alexkroman1/aai/testing/vitest";
import { expect, test } from "vitest";

const PAGE_HTML = "<html><body>the brief</body></html>";

test("the reviewer is asked about the brief", async () => {
  const model = stubGatewayRoute(['{"verdict":"ship"}']);
  installStubStepFetch((request) => model.route(request) ?? { body: PAGE_HTML });
  // … run the workflow …
  expect(model.calls[0]?.prompt).toContain("the brief");
});

Parameters

replies

string | readonly string[]

Completion contents, in order; the last repeats. A bare string is one reply.

options?

StubGatewayOptions

Returns

StubGatewayRoute

stubGenerate()

stubGenerate(script): StubGenerate

Build a fake ctx.generate from a script: one reply, or routes keyed by system prompt.

A call whose system prompt names no route throws, naming it — an unscripted model call is a spec that has drifted from the tool, not a case to paper over. Pass { reply } to answer every call the same way, which is what a one-model tool wants.

Parameters

script

StubGenerateScript

Returns

StubGenerate

Examples

Two model roles, one queue

import { createToolContext, stubGenerate } from "@alexkroman1/aai/testing";

const verdicts = ["yes", "no"];
const model = stubGenerate({
  routes: {
    "You grade documents.": () => ({ object: { score: verdicts.shift() ?? "yes" } }),
    "You answer questions.": "The documented answer.",
  },
});
const ctx = createToolContext({ generate: model.generate });
// … run the tool, then assert on the roles it played:
// expect(model.calls.map((call) => call.system)).toEqual([…]);

One model role

import { stubGenerate } from "@alexkroman1/aai/testing";

const model = stubGenerate({ reply: { object: { steps: ["Only step"] } } });
const answerer = stubGenerate({ reply: "The documented answer." });

stubPlaceCall()

stubPlaceCall(options?): StubPlaceCall

Publish a Twilio whose Calls API records every dial and answers it.

Parameters

options?

StubPlaceCallOptions

Returns

StubPlaceCall

Example

import { stubPlaceCall } from "@alexkroman1/aai/testing";

const twilio = stubPlaceCall({ status: (call) => (call.polls < 2 ? "ringing" : "completed") });
// … run the workflow, then read what it dialled, as the answering session sees it:
console.log(twilio.calls[0]?.parameters.call);
twilio.restore(); // in an `afterEach`

stubReporter()

stubReporter(): StubReporter

Capture what a step narrates and emits.

stepReport() and stepEmit() both go through a published slot, and with nothing published they fall back to the console — which is right for a step under test that nobody is asserting on, and useless the moment the narration IS the subject. It is for a step whose partial results are part of its contract: a fan-out that emits each segment as it lands has a page depending on the shape of those chunks, and nothing else in a spec can see them.

The two are separated the way the streams are, so a spec asserting a chunk never has to filter the sentences out of it.

import { stepEmit } from "@alexkroman1/aai/step";
import { stubReporter } from "@alexkroman1/aai/testing";
import { afterEach, expect, test } from "vitest";

const reported = stubReporter();
afterEach(reported.restore);

test("each segment is emitted as it lands", async () => {
  await stepEmit("transcript", { index: 0, text: "hello there" });
  expect(reported.emitted).toEqual([
    { namespace: "transcript", chunk: { index: 0, text: "hello there" } },
  ]);
});

Publishing REPLACES, so a spec that forgets to restore leaves this one answering the next file’s steps — the same rule stubStepFetch follows, and the same remedy.

Returns

StubReporter

stubSpeech()

stubSpeech(options?): StubSpeech

Publish a synthesizer that records what it was asked to say and answers with silence.

Silence rather than a tone, because nothing downstream of a step listens: a spec asserts on the TEXT that was spoken, the duration, and where the bytes went. Generating audible audio would only make the fixtures bigger.

Parameters

options?

StubSpeechOptions

Returns

StubSpeech

stubStepDelegate()

stubStepDelegate(script): StubStepDelegate

PUBLISH a fake runner, so an exported step that calls stepDelegate can be driven without a host.

stubDelegate with the slot filled in, and deliberately nothing more: the step-side and tool-side capabilities have the same signature because they are the same runner bound differently, so a spec routes them the same way and a template that moves a subagent from a tool into a step rewrites no fake.

An unpublished slot THROWS rather than degrading (see sdk/step-delegate.ts), which is what makes this the ONE way to test such a step — and why the failure an author meets first names this function.

Parameters

script

StubDelegateScript

Returns

StubStepDelegate

Example

import { stubStepDelegate } from "@alexkroman1/aai/testing";

const desk = stubStepDelegate({ routes: { researcher: "Prices fell 12% in 2025." } });
try {
  // … call the exported step, then assert on `desk.calls`
} finally {
  desk.restore();
}

stubStepFetch()

stubStepFetch(answer?): StubStepFetch

Publish a fake stepFetch, so a step’s HTTP can be asserted without a server and without stubbing a global.

A step’s outbound call goes through a process-wide slot rather than globalThis.fetch (see stepFetch on @alexkroman1/aai/step for why — HTTP/1.1 pinning, and a fan-out that breaks on HTTP/2 stream resets), so this is the honest way to intercept it. vi.stubGlobal("fetch", …) still works, because an unpublished slot falls back to the global; it just tests a path production does not take, and it cannot see the request BODY as bytes.

answer may return a Response, or a { status, body, headers } shorthand, or throw — a throw is what a connection failure looks like, and stepFetch wraps it in a StepTransportError exactly as it would in production.

Returns restore, and calling it in an afterEach is not optional — a fetch left published makes the next file’s steps answer to this one’s handler. installStubStepFetch (@alexkroman1/aai/testing/vitest) is the same fake with that registration already done.

Parameters

answer?

(request) => StubStepAnswer | Promise<StubStepAnswer>

Called per request with the recorded request. Defaults to an empty 200.

Returns

StubStepFetch

Example

import { stepFetch } from "@alexkroman1/aai/step";
import { stubStepFetch } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";

test("the step sends its credential", async () => {
  const sync = stubStepFetch(() => ({ body: { text: "hello there" } }));
  await stepFetch("https://sync.example.com/transcribe", {
    headers: { Authorization: "sk-test" },
  });
  expect(sync.calls[0]?.headers.Authorization).toBe("sk-test");
  sync.restore();
});

stubStepInfo()

stubStepInfo(step): object

Answer stepInfo() for the step under test, so a body’s RETRY branch is reachable from a spec.

A step that degrades on its last attempt has two paths and a spec could only ever take one: outside a run stepInfo() answers undefined, which a body reads as “not retrying”. So the branch that exists precisely for the case that goes wrong was the branch no test could enter — and it is the one whose failure is quiet, since a body that mis-reads the ceiling degrades early on every run and still returns an answer.

import { stubStepInfo } from "@alexkroman1/aai/testing";
import { onTestFinished, expect, test } from "vitest";

declare function summarizeChapter(text: string): Promise<string>;

test("falls back to the cheap model on the last attempt", async () => {
  const stub = stubStepInfo({ attempt: 3, maxAttempts: 3 });
  onTestFinished(stub.restore);
  expect(await summarizeChapter("…")).toContain("…");
});

isLastAttempt is DERIVED from the two numbers rather than accepted, for the reason the real reader derives it: a fake that let a spec set attempt: 1 and isLastAttempt: true would let a body pass against a state no run can be in.

Publishing REPLACES, so a spec that forgets to restore leaves this answering the next file’s steps — the same rule stubReporter follows, and the same remedy.

Parameters

step

attempt?

number

1-based. Defaults to 1.

maxAttempts?

number

Defaults to whichever is larger of 3 (the SDK’s own default) and attempt.

name?

string

Defaults to "step".

Returns

object

restore

() => void

stubTranscribe()

stubTranscribe(options?): StubTranscribe

Answer AssemblyAI’s transcription endpoints in memory, and record what was sent.

Covers all four calls — the async trio (stepTranscribeUpload, stepTranscribeSubmit, stepTranscribePoll) and stepTranscribeSync — so a workflow that uploads, submits, polls and reads is testable end to end without naming upload_url, audio_duration or status: "completed" anywhere in the spec.

What it does NOT do is stand in for the upload STORE: stepTranscribeUpload streams the recording out of the app’s own store, so a spec still publishes one with stubUploads. The two fakes fill different slots and compose.

Parameters

options?

StubTranscribeOptions

Returns

StubTranscribe

Example

A whole async job, in one line of setup

import {
  stepReadUpload,
  stepTranscribePoll,
  stepTranscribeSubmit,
  stepTranscribeSync,
  stepTranscribeUpload,
} from "@alexkroman1/aai/step";
import { orFail } from "@alexkroman1/aai/step-errors";

export async function transcribeRecording(uploadId: string) {
  const { audioUrl } = await stepTranscribeUpload(uploadId);
  const { id } = await stepTranscribeSubmit(audioUrl);
  let progress = await stepTranscribePoll(id);
  while (!progress.done) progress = await stepTranscribePoll(id);
  return progress.transcript.text;
}

export async function transcribeSegment(uploadId: string, segment: { start: number; end: number }) {
  const { bytes } = await stepReadUpload(uploadId, segment);
  return orFail(stepTranscribeSync)(bytes);
}

// file: transcribe.test.ts
import { RetryableError } from "@alexkroman1/aai/step-errors";
import { stubTranscribe, stubUploads } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import { transcribeRecording, transcribeSegment } from "./workflows/transcribe.ts";

test("a whole async job, in one line of setup", async () => {
  const uploads = stubUploads({ upl_1: new Uint8Array(5000) });
  const provider = stubTranscribe({ text: "we ship tuesday", durationSec: 42 });

  expect(await transcribeRecording("upl_1")).toBe("we ship tuesday");
  // The file really streamed: `stubStepFetch` drains the body into bytes.
  expect(provider.calls.find((call) => call.leg === "upload")?.body).toBeInstanceOf(Uint8Array);

  provider.restore();
  uploads.restore();
});

test("a rate limit, classified by the SDK rather than by the fake", async () => {
  const uploads = stubUploads({ upl_1: new Uint8Array(5000) });
  const provider = stubTranscribe({
    failure: { leg: "sync", status: 429, retryAfterSeconds: 30 },
  });
  // `toStepError` reads `retryable` and `retryAfter` off the real TranscribeError.
  await expect(transcribeSegment("upl_1", { start: 0, end: 1000 })).rejects.toBeInstanceOf(
    RetryableError,
  );
  provider.restore();
  uploads.restore();
});

stubUploads()

stubUploads(files, options?): StubUploads

Publish an in-memory upload store, so a step that calls stepReadUpload can be tested without a server.

A step reads uploads through a process-wide slot rather than dialling anything, which is what makes this possible at all: a spec supplies its own bytes and the step under test is unchanged.

Returns a StubUploads — restore, plus what a step WROTE. Calling restore in an afterEach is not optional; a store left published makes the next file’s steps read this one’s bytes, which is the kind of cross-file leak that presents as a passing test somewhere else.

Parameters

files

Readonly<Record<string, StubUpload>>

Keyed by upload id — the same string a run input would carry.

options?

StubUploadsOptions

Returns

StubUploads

Examples

import { stubUploads } from "@alexkroman1/aai/testing";

const uploads = stubUploads({ upl_1: new Uint8Array([1, 2, 3]) });
// … call the step …
uploads.restore();

// A streamed upload mid-flight: `stepReadUpload` comes back short and
// `stepUploadInfo(...).complete` is false, which is what a polling body sees.
const firstHalf = new Uint8Array([1, 2]);
stubUploads({ upl_2: { bytes: firstHalf, complete: false } }).restore();

What a step wrote, without reading it back through the slot

import { stepWriteUpload } from "@alexkroman1/aai/step";
import { stubUploads } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";

test("the step files its summary", async () => {
  const uploads = stubUploads({}, { writable: true });
  await stepWriteUpload(new Uint8Array(16), { name: "summary.wav", type: "audio/wav" });
  expect(uploads.writes.map((one) => one.name)).toEqual(["summary.wav"]);
  uploads.restore();
});

toolInputIssues()

toolInputIssues(tool, value): Promise<readonly StandardSchemaIssue[] | undefined>

The issues tool’s input schema found in value, or undefined.

The negative half of parseToolInput — the assertion behind “a mood outside the enum is refused by the schema”, which is the one thing standing between an LLM’s untyped tool call and the tool body. Takes the def, as parseToolInput does.

Parameters

tool

ToolDef<ToolInputSchema>

value

unknown

Returns

Promise<readonly StandardSchemaIssue[] | undefined>

Throws

When the tool declares no inputSchema. A tool that takes no arguments accepts anything, and saying so out loud beats answering undefined — which reads as “accepted”.

Example

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Recommend something for tonight",
  inputSchema: z.object({ mood: z.enum(["cozy", "tense", "silly"]) }),
  execute: ({ mood }) => ({ picks: [mood] }),
});

// file: agent.test.ts
import { toolInputIssues } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import recommend from "./tools/recommend.ts";

test("recommend refuses a mood off its list", async () => {
  expect(await toolInputIssues(recommend, { mood: "melancholy" })).toBeDefined();
});

toolNameOf()

toolNameOf(def): string

The name the model calls def by — the tools/<name>.ts file it was discovered from, or the key it sits under on a speaker’s tools map.

Only a def an agent LOWERED has one. A file tool is named when the agent is imported as DEPLOYED (virtual:aai/agent under vitest, deployedAgent under any other runner); a roster tool is named when agent({ roster }) runs, i.e. when agent.ts is imported at all. A def built inline in a spec has never been lowered, so it has no name, and the throw says so.

Parameters

def

ToolDef<ToolInputSchema>

Returns

string

Throws

When def was never lowered onto an agent.

Example

import { agent } from "@alexkroman1/aai";

export default agent({ name: "Orders" });

// file: tools/place_order.ts
import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Place the order",
  inputSchema: z.object({ item: z.string() }),
  execute: ({ item }) => ({ placed: item }),
});

// file: agent.test.ts
import { deployedAgent, toolNameOf } from "@alexkroman1/aai/testing";
import { expect, test } from "vitest";
import authored from "./agent.ts";
import placeOrder from "./tools/place_order.ts";

// Lowering the agent is what records the names; under vitest, importing it
// as `virtual:aai/agent` does the same.
deployedAgent(authored, { tools: import.meta.glob("./tools/*.ts", { eager: true }) });

test("the model calls the tool by its file name", () => {
  expect(toolNameOf(placeOrder)).toBe("place_order");
});

toolOf()

toolOf(agent, name): ToolDef

The tool name is declared under, or a throw naming the ones that are.

The LOOKUP, for a spec asserting on what the agent declares — a description, a schema, a tool the roster minted and no file exports (HANDOFF_TOOL_NAME). To RUN a tool, hand its def to runTool or to the runner toolRunner binds; the def a tools/ file default-exports is the object the deployed agent registered, so no name is needed to find it.

A tool is a FILE, so agent.ts’s default export declares no tools at all — import the agent as DEPLOYED, exactly as this example does and as every shipped template’s spec does: virtual:aai/agent under vitest, or deployedAgent (@alexkroman1/aai/testing) under any other runner. Handing this the authored def directly is the common mistake, and it fails with “(none)”.

Parameters

agent

ToolBearingAgent

name

string

Returns

ToolDef

Example

import agentDef from "virtual:aai/agent";
import { toolOf } from "@alexkroman1/aai/testing";
import { expect } from "vitest";

expect(toolOf(agentDef, "add_item").description).toContain("cart");

toolRunner()

toolRunner(agent): ToolRunner

runTool bound to one agent and GATED by it — the run(...) a spec actually calls, and what describeAgent hands its body.

A spec drives one agent, so agentDef is the same in every call and the tool is the thing that varies. Binding the agent is what lets the run go through the agent’s own gates: the def is found in the table by identity, its toolset’s gate is asked first, and a refusal comes back as the result — a dialog out of state, a roster entry whose speaker is not on the line. The bare runTool calls the def’s execute directly, gate and all skipped, which is the unit test of a tool body and not the test of the agent.

The runner is stateless and holds only the agent, so one per spec file at the top level is the shape: each call still defaults to a FRESH context, i.e. a distinct session with empty slots. Pass a context explicitly wherever the second call is meant to see the first call’s work — see runTool.

Parameters

agent

ToolBearingAgent

Returns

ToolRunner

Example

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Add an item to the order",
  inputSchema: z.object({ item: z.string() }),
  execute: ({ item }) => ({ added: item }),
});

// file: tools/view_order.ts
import { tool } from "@alexkroman1/aai";

export default tool({
  description: "View the order",
  execute: () => ({ items: ["apple"] }),
});

// file: agent.test.ts
import agentDef from "virtual:aai/agent";
import { createToolContext, toolRunner } from "@alexkroman1/aai/testing";
import { expect } from "vitest";
import addItem from "./tools/add_item.ts";
import viewOrder from "./tools/view_order.ts";

const run = toolRunner(agentDef);

const { added } = await run(addItem, { item: "apple" }); // typed: `added` is a string
expect(added).toBe("apple");

// No arguments, one session shared across the two calls.
const ctx = createToolContext();
await run(addItem, { item: "apple" }, ctx);
expect(await run(viewOrder, ctx)).toEqual({ items: ["apple"] });

A runner over an agent with NO tools is refused HERE, rather than at the first run(...). A tool is a file, so agent.ts’s default export declares an empty table and every call through such a runner fails identically — the mistake is the argument on this line, and reporting it at a call site several dozen lines away names the symptom instead. It is the one shape that cannot be a legitimate runner: a runner exists to reach the agent’s tools, and there are none to reach. Reach for toolOf or runTool directly if a spec really means to assert on an empty table.

Interfaces

DeployedConfig

Sealed

What expectDeployable hands back: the RESOLVED config a deploy carries, narrowed to the fields a starter spec asserts on.

Not the whole AgentConfig, on purpose. That type is inferred from the canonical config SCHEMA, so returning it put the schema — every serializable agent field, each with its own validation shape — into this subpath’s contract, and a new agent field moved a TEST helper’s hash. These are the fields the shipped specs read; the object returned is the real config, so a spec that needs one more can read it off toAgentConfig (@alexkroman1/aai/manifest) directly.

mode is always present: expectDeployable refuses a conversion that derived none.

Properties

builtinTools?

readonly optional builtinTools?: readonly BuiltinTool[]

The builtins the agent declares (absent: the default surface).

llm?

readonly optional llm?: DeployedStage

The LLM stage — declared, or the injected default in pipeline mode.

mcpServers?

readonly optional mcpServers?: Readonly<Record<string, Readonly<Record<string, unknown>>>>

The MCP servers whose tools join the agent’s own, by key.

mode

readonly mode: AgentMode

The agent’s mode, as the deploy carries it.

name

readonly name: string

The name the platform lists the agent under.

requiredEnv?

readonly optional requiredEnv?: readonly string[]

The env var names a deploy preflights.

s2s?

readonly optional s2s?: DeployedStage

The speech-to-speech descriptor, for an s2s agent.

stt?

readonly optional stt?: DeployedStage

The STT stage — declared, or the injected default in pipeline mode.

systemPrompt

readonly systemPrompt: string

The system prompt a deploy carries — the author’s string, or the framework default when there is none. A RESOLVER is not carried (it cannot be serialized), so an agent with one reads the default here.

tts?

readonly optional tts?: DeployedStage

The TTS stage — declared, or the injected default in pipeline mode.

turnTaking?

readonly optional turnTaking?: object

Pipeline turn-taking — detection: "manual" is push-to-talk.

detection?

readonly optional detection?: string

usageLimits?

readonly optional usageLimits?: object

The session’s token budget, when it declares one.

totalTokens?

readonly optional totalTokens?: number

DeployedStage

Sealed

One provider stage of a DeployedConfig — the descriptor as it will be deployed: its kind, and its options exactly as serialized.

Properties

kind

readonly kind: string

The provider the stage resolves through, e.g. "assemblyai".

options?

readonly optional options?: Readonly<Record<string, unknown>>

The descriptor’s options, as they cross the wire.

SaidLine

Sealed

One ctx.speech.say a createToolContext context recorded.

Properties

interrupt

readonly interrupt: boolean

Whether it asked to cut the agent off first ({ interrupt: true }).

interruptible

readonly interruptible: boolean

false when it asked not to be cut off by the caller ({ interruptible: false }).

record

readonly record: boolean

false when it asked to stay out of history ({ record: false }).

text

readonly text: string

The text, exactly as passed.

SentEvent

Sealed

One ctx.send(event, data) call that would REACH the client, as recorded by createToolContext — see the send default for what is left out.

Properties

data

data: unknown

event

event: string

StubDelegate

Sealed

A fake ctx.delegate: the function to pass, and what it was asked.

Properties

calls

calls: StubDelegateCall[]

Every call, in order.

delegate

delegate: DelegateFn

Pass as delegate to createToolContext.

StubDelegateCall

Sealed

One ctx.delegate call, as recorded by stubDelegate.

Properties

options

options: DelegateOptions

The whole options object, for asserting context and maxSteps.

subagent

subagent: SpeakerDef

The subagent that was asked.

task

task: string

The task it was given.

StubGateway

Sealed

A fake gateway: the fetch to install, and what it was asked.

Properties

calls

calls: StubGatewayCall[]

Every request, in call order.

fetch

fetch: (url, init?) => Promise<Response>

Install with vi.stubGlobal("fetch", gateway.fetch).

Parameters

url

string | Request | URL

init?

RequestInit

Returns

Promise<Response>

StubGatewayCall

Sealed

One request a StubGateway answered.

Properties

body

body: Record<string, unknown>

The whole decoded request body, for asserting model, temperature, …

headers

headers: Record<string, string>

The request headers, lower-cased.

Worth asserting rather than assuming: the gateway is OpenAI-compatible and takes the key as a Bearer, where AssemblyAI’s streaming sockets take it raw — and getting that backwards is a 401 that reads like a wrong key.

prompt

prompt: string

The user message — what the step actually asked.

system

system: string | undefined

The system instruction, or undefined when the step sent none.

url

url: string

The endpoint the call went to, so a spec can assert the gateway URL.

StubGatewayOptions

Input

Options for stubGateway.

Properties

headers?

optional headers?: Record<string, string>

Extra response headers — Retry-After is the one specs reach for.

status?

optional status?: number

HTTP status to answer with. Defaults to 200. A non-2xx answers with an error body, which is what stepGenerate (@alexkroman1/aai/step) quotes back in its StepGenerateError.

StubGatewayRoute

Sealed

A gateway answer for a stepFetch-published slot, plus what it was asked.

Properties

calls

calls: StubGatewayCall[]

Every completion request this route answered, DECODED, in call order.

route

route: (request) => StubStepAnswer | undefined

Answers a completion request and undefined for anything else, so the caller composes it: as the first leg of a stubFetchRoutes list (where an unexpected request is a finding by default), ?? { body: html } for a flow that also fetches a page, or straight into stubTranscribe’s otherwise.

Parameters

request

StubStepRequest

Returns

StubStepAnswer | undefined

StubGenerate

Sealed

A fake ctx.generate: the function to pass, and what it was asked.

Properties

calls

calls: StubGenerateCall[]

Every call, in order.

generate

generate: GenerateFn

Pass as generate to createToolContext.

StubGenerateCall

Sealed

One ctx.generate call, as recorded by stubGenerate.

Properties

options

options: GenerateOptions

The whole options object, for asserting llm, temperature, schema, …

prompt

prompt: string

The user prompt — what the tool actually asked.

system

system: string | undefined

The system instruction, or undefined when the call carried none.

StubStepDelegate

Sealed

A fake stepDelegate: the calls it recorded, and the slot to give back.

Methods

restore()

restore(): void

Unpublish the runner.

Calling it in an afterEach is not optional — a stub left published makes the next file’s steps delegate into this one’s log, which is the kind of cross-file leak that presents as a passing test somewhere else.

Returns

void

Properties

calls

calls: StubDelegateCall[]

Every call, in order — the same log stubDelegate keeps.

Type Aliases

FetchRouteHandler

FetchRouteHandler = (request) => StubStepAnswer | undefined | Promise<StubStepAnswer | undefined>

A route: answers a request with a Response or the { status, body, headers } shorthand stubStepFetch takes — or undefined to DECLINE, leaving it to the next route (in a list) or to FetchRoutesOptions.unmatched.

stubGatewayRoute().route is one, so a model leg sits in a list beside a spec’s own.

Parameters

request

FetchRouteRequest

Returns

StubStepAnswer | undefined | Promise<StubStepAnswer | undefined>

FetchRouteHit

FetchRouteHit = FetchRouteRequest & object

Sealed

One request the router saw, whatever became of it.

Type Declaration

outcome

readonly outcome: "routed" | "passthrough" | "unmatched" | string & object

Answered by a route, sent to the real network, or answered by nothing.

route?

readonly optional route?: string

The table key that answered it (a list’s routes have none).

status?

readonly optional status?: number

The response’s status, for a request that got one.

via

readonly via: "fetch" | "stepFetch" | string & object

Which fetch it arrived through.

FetchRouteRequest

FetchRouteRequest = StubStepRequest & object

Sealed

One request as a route sees it: the recorded request (url, method, headers, body — the same fields stubStepFetch records) plus the parts a route branches on, already parsed.

Type Declaration

host

readonly host: string

new URL(url).hostname.

json

readonly json: unknown

The body parsed as JSON when it parses; undefined otherwise (and for none).

pathname

readonly pathname: string

new URL(url).pathname.

searchParams

readonly searchParams: URLSearchParams

new URL(url).searchParams — PostgREST filters, query strings.

FetchRoutesOptions

FetchRoutesOptions = object

Input

What stubFetchRoutes may be told.

Properties

globalFetch?

optional globalFetch?: boolean

Install the router as the global fetch too (the default). Pass false to route ONLY the step fetch and leave the global untouched — for an eval whose live mode sends the voice model’s own traffic over the global while its steps stay scripted.

passThrough?

optional passThrough?: RegExp

URLs that reach the REAL network before any route is consulted, tested against the full URL — e.g. /^https://[^/]*assemblyai\.com// for a live model’s own traffic.

stepFetch?

optional stepFetch?: boolean

Publish the router as the step fetch too (the default), so a step’s stepFetch lands in the same routes and the same log as a tool’s fetch. Pass false for a spec that installs its own step fetch.

unmatched?

optional unmatched?: "throw" | "notFound" | "passthrough"

What a request no route answers means.

"throw" (the default) — a finding: the fetch rejects naming the method and URL, the way an unreachable host does, and the request is logged with outcome: "unmatched". An invented 200 {} reads to a tool as success, so the spec would pass having tested the wrong branch.
"notFound" — a real 404, for a spec whose subject is one.
"passthrough" — the REAL network. Rarely right in a unit test.

FetchRouteTable

FetchRouteTable = Readonly<Record<string, FetchRouteHandler | StubStepAnswer>>

Routes by where they answer. A key is an optional METHOD, then one of:

a HOST — "api.mem0.ai" — matching that hostname exactly;
a WILDCARD host — "*.example" — matching any subdomain of it;
a URL PREFIX — "https://api.mem0.ai/v3/memories/" — matching any URL that starts with it.

So "POST textbelt.com" answers only a POST. The most specific key answers: a URL prefix (longest first), then an exact host, then a wildcard (longest first); a METHOD-qualified key beats the same key without one. A value is a FetchRouteHandler, or a fixed answer given to every request it matches.

FiredEvent

FiredEvent = object

Sealed

What fireEvent reports: how many handlers the event reached.

Properties

dialogs

readonly dialogs: readonly string[]

The keys of the declared dialogs the event MOVED — one whose on map names it ("@session.timedOut": "abandoned") and whose active state handles it. Every declared dialog is offered the event, before the handlers, as the session offers it.

typed

readonly typed: boolean

The per-type handler ran.

wildcard

readonly wildcard: boolean

The "*" handler ran, after the typed one.

ProjectFiles

ProjectFiles = object

Input

What the BUILD lowers onto an agent.ts default export — the files beside it that a deployed agent runs with and a spec has to apply itself.

Both fields are optional and at least one must be present: an empty object is a call that does nothing, which is the shape of a forgotten argument rather than of a project with no files.

Properties

systemPrompt?

readonly optional systemPrompt?: string

import prompt from "./system-prompt.md?raw".

Omit it for a project with no system-prompt.md. Passing it to a def the file was already applied to leaves that def exactly as it is, so a spec never has to know whether its def is the raw agent.ts export or a lowered one.

tools?

readonly optional tools?: Readonly<Record<string, unknown>>

import.meta.glob("./tools/*.ts", { eager: true }), written at the CALL SITE — see the module doc for why it cannot be a directory string.

Omit it for a project with no tools/ directory. Passing an EMPTY glob is an error, not a no-op: see deployedAgent.

Spelled out rather than named as ToolModules (the same type, on /manifest): the value is a glob result, not something a spec names, so /testing exports no alias for it and a named one would be a declaration no capability owns.

workflows?

readonly optional workflows?: Readonly<Record<string, unknown>>

import.meta.glob("./workflows/*.ts", { eager: true }), written at the call site like tools. A module default-exporting a workflow() def is registered under its file name, an explicit agent({ workflows }) entry winning a clash; the rest of the directory (a body’s helpers) is skipped, so unlike tools an empty or helper-only result is not an error.

RecordedSignal

RecordedSignal = object

Sealed

One signalRun as a tool made it.

Properties

key

key: string

label

label: string

options

options: SignalOptions | undefined

payload

payload: unknown

workflow

workflow: string

The workflow’s declared name.

RecordedSleep

RecordedSleep = object

Sealed

One wait the body asked for — and did NOT take.

Properties

correlationId?

optional correlationId?: string

label

label: string

The wait’s label — its identity in a real run’s journal, and the field a case asserting a SCHEDULE actually wants: a body with two waits is telling you WHICH one it reached, which a duration cannot.

until

until: number | Date

Exactly what the body passed: milliseconds, or a Date.

RecordedStart

RecordedStart = object

Sealed

One start the client recorded.

Properties

def

readonly def: AnyWorkflowDef | undefined

The def passed, when one was (undefined for a start by name).

input

readonly input: unknown

The input, exactly as the tool passed it — not validated, since nothing runs.

options

readonly options: StartOptions | undefined

The start options (key, label, notify, …), when any were passed.

runId

readonly runId: string

The run id start resolved with.

workflow

readonly workflow: string

The declared name — the key in agent({ workflows }) — or the string passed.

RecordedStep

RecordedStep = object

Sealed

One step the body reached, as the recorder saw it.

Properties

maxAttempts?

optional maxAttempts?: number

What the body asked for, or undefined when it passed no options.

maxWaits?

optional maxWaits?: number

The step’s wait budget (StepOptions.maxWaits), or undefined when none was asked for.

name

name: string

RecordingWorkflows

RecordingWorkflows = WorkflowClient & object

Sealed

A WorkflowClient that records, plus its log.

start records and resolves a fresh run id, and the run it “started” is visible to get/find/recent as running — so a tool that checks for a pending run before starting a second one sees its own first start. cancel marks a known, unfinished run cancelled and resolves true (else false); wakeUp resolves 0; lastLine resolves undefined. The progress-channel reads (stream, streamTail, signal, publicWebhookUrl) REJECT, as createStubWorkflows’ do — spread over it to answer one.

Type Declaration

cancelled

readonly cancelled: string[]

Every run id cancel was called with, in order, whatever it resolved.

signals

readonly signals: RecordedSignal[]

Every signalRun, in order — delivered (true) without a run to deliver to.

starts

readonly starts: RecordedStart[]

Every start, in order.

seed()

seed(…runs): void

Add runs for the reads to answer from, e.g. inside a case before say().

Parameters

runs

…WorkflowRunSnapshot[]

Returns

void

started()

started(workflow?): RecordedStart[]

The starts of one workflow — by declared name or by def — or all of them.

Parameters

workflow?

string | AnyWorkflowDef

Returns

RecordedStart[]

RecordingWorkflowsOptions

RecordingWorkflowsOptions = object

Input

What createRecordingWorkflows takes.

Properties

runIdPrefix?

optional runIdPrefix?: string

Prefix of the minted run ids, numbered from 1. Defaults to "wrun_rec_".

runs?

optional runs?: readonly WorkflowRunSnapshot[]

Runs the reads answer from before anything starts — a reminder already pending, a job that failed. Build them with createRunSnapshot; more can be added later with seed.

workflows?

optional workflows?: Readonly<Record<string, AnyWorkflowDef>>

The agent’s declared workflows — pass agentDef.workflows — so a start by DEF is recorded under its declared name, and listing() reports them. A def not in it is refused, as the real client refuses it. Without it, a def is recorded under its description and matched by identity.

RouteContextOverrides

RouteContextOverrides = { [K in keyof RouteContext]?: RouteContext[K] }

Input

What createRouteContext accepts: a field per RouteContext field, each also taking undefined for one the caller does not have.

RunSnapshotOverrides

RunSnapshotOverrides<R> = Partial<WorkflowRunBase> & { status?: undefined; } | WorkflowRunSnapshot<R> extends infer S ? S extends unknown ? Omit<S, keyof WorkflowRunBase> : never : never

What createRunSnapshot accepts: the shared fields, plus whatever the chosen status requires.

The status-bearing half is DERIVED from WorkflowRunSnapshot’s own union, arm by arm, so asking for status: "completed" without an output is a compile error rather than a fixture that lies — and an arm the snapshot grows is one this accepts without a second edit.

Type Parameters

R

R = unknown

The workflow’s return type, when the caller names it.

SessionEventFields

SessionEventFields<K> = { [T in K]: Omit<SessionEventMap[T], “type” | “meta”> }[K]

The fields of one event beyond its envelope — what sessionEvent takes after the type: the member of SessionEventMap minus type and meta. Distributes over K, so each member keeps its own fields.

Type Parameters

K

K extends SessionEventType

SessionEventRest

SessionEventRest<K> = keyof SessionEventFields<K> extends never ? [SessionEventFields<K>, Partial<SessionEventMeta>] : [SessionEventFields<K>, Partial<SessionEventMeta>]

The trailing parameters of sessionEvent: [meta?] for an envelope-only event, [fields, meta?] for one with a payload. An event that is only its envelope (reply.completed) takes no fields argument at all, so sessionEvent("reply.completed") is the whole call rather than sessionEvent("reply.completed", {}).

Type Parameters

K

K extends SessionEventType

SlotContractOptions

SlotContractOptions<T> = object

Input

What expectSlotContract may be told.

Type Parameters

T

T

Properties

caps?

readonly optional caps?: Readonly<Record<string, number>>

The caps to check — each top-level array key and the most entries it keeps. Defaults to what the slot DECLARED (slot.caps); given, each is driven past its bound and the surviving entries must be the NEWEST ones, and a bound that differs from the declared one fails naming both.

write?

readonly optional write?: (draft) => void

A mutation that CHANGES the value — what the isolation check writes through one session to prove the other does not see it. Optional: without it the check is that two sessions hold distinct values of the default, which catches a shared default object and nothing subtler. A write that leaves the value equal to the default is refused, because it proves nothing.

Parameters

draft

T

Returns

void

StubClientInbox

StubClientInbox = object

Sealed

What stubClientInbox returns: the call log, and how to put the slot back.

Methods

restore()

restore(): void

Unpublish the inbox. Call it in an afterEach, like stubSpeech’s.

Returns

void

Properties

calls

calls: StubClientInboxCall[]

Every notice pushed, in order — including those the device did not take.

StubClientInboxCall

StubClientInboxCall = object

Sealed

One pushed notice, as stubClientInbox records it.

Properties

clientId

clientId: string

notice

notice: ClientNotice

StubClientInboxOptions

StubClientInboxOptions = object

Input

What stubClientInbox may be told.

Properties

answer?

optional answer?: "acked" | ClientUnreachableReason | ((call) => "acked" | ClientUnreachableReason)

How the device answers each notice: "acked" (the default), or a reason it did not take it — "offline", "busy", "no-ack", "disconnected". A function answers per call, e.g. busy once and then acked.

StubClientTranscript

StubClientTranscript = object

Sealed

What stubClientTranscript returns: the call log, and how to put the slot back.

Methods

restore()

restore(): void

Unpublish the reader. Call it in an afterEach, like stubClientInbox’s.

Returns

void

Properties

calls

calls: StubClientTranscriptCall[]

Every read, in order.

StubClientTranscriptAnswer

StubClientTranscriptAnswer = ClientTranscript | ((call) => ClientTranscript)

What stubClientTranscript answers each read with.

StubClientTranscriptCall

StubClientTranscriptCall = object

Sealed

One read, as stubClientTranscript records it.

Properties

clientId

clientId: string

options

options: StepClientTranscriptOptions

StubDelegateReply

StubDelegateReply = string | { complaint?: string; revisions?: number; steps?: number; text: string; toolCalls?: readonly DelegateToolCall[]; }

What one route answers with.

A bare string is the subagent’s final text with an empty cost report, which is what a tool that only reads text wants. The object form fills in steps and toolCalls for a tool that narrates the wait.

Union Members

string

Type Literal

{ complaint?: string; revisions?: number; steps?: number; text: string; toolCalls?: readonly DelegateToolCall[]; }

complaint?

optional complaint?: string

Stage a run the subagent’s GUARDRAIL never accepted: the complaint the real runtime returns beside the last rejected attempt.

Its presence is what makes the result’s accepted false — the two cannot be staged apart, because in the runtime they cannot occur apart. A spec cannot describe an unaccepted answer with no reason, and a caller reading complaint on an accepted one would be reading a field that is never set.

revisions?

optional revisions?: number

How many times a guardrail sent an answer back. Defaults to 0.

steps?

optional steps?: number

text

text: string

toolCalls?

optional toolCalls?: readonly DelegateToolCall[]

StubDelegateRoute

StubDelegateRoute = StubDelegateReply | ((call) => StubDelegateReply)

How a route answers: a fixed reply, or a function of the call — the function form being what a route asked more than once (a subagent run per document) needs in order to shift its own script.

StubDelegateScript

StubDelegateScript = { reply: StubDelegateRoute; routes?: never; } | { reply?: never; routes: Readonly<Record<string, StubDelegateRoute>>; }

Everything stubDelegate and stubStepDelegate accept: ONE route answering every delegation, or a table of routes keyed by subagent name — each under a key that says which.

The same two shapes StubGenerateScript takes, for the same reason: a bare “a table, or a reply” union is told apart at runtime by the reply’s shape, so a subagent named text could never be routed and a reply object could be read as a table. Named, there is nothing to guess.

Union Members

Type Literal

{ reply: StubDelegateRoute; routes?: never; }

reply

readonly reply: StubDelegateRoute

Answers EVERY delegation, whichever subagent it names.

routes?

readonly optional routes?: never

Type Literal

{ reply?: never; routes: Readonly<Record<string, StubDelegateRoute>>; }

reply?

readonly optional reply?: never

routes

readonly routes: Readonly<Record<string, StubDelegateRoute>>

One route per subagent, keyed by its name. A delegation naming no route rejects, naming the subagent.

StubEmitted

StubEmitted = object

Sealed

One chunk stepEmit() wrote, and the stream it went to.

Properties

chunk

chunk: unknown

The value, exactly as the step passed it.

namespace

namespace: string

The stream named at the call site.

StubFetchRoutes

StubFetchRoutes = object

Sealed

What stubFetchRoutes returns.

Methods

restore()

restore(): void

Put the global fetch back and unpublish the step fetch.

Returns

void

to()

to(filter): FetchRouteHit[]

The hits a filter selects: a string matched the way a route KEY is ("POST supabase.test", "*.example"), a RegExp tested against the URL.

Parameters

filter

string | RegExp

Returns

FetchRouteHit[]

Properties

fetch

readonly fetch: typeof globalThis.fetch

The router as a fetch, for code handed one explicitly.

hits

readonly hits: FetchRouteHit[]

Every request, in order, including passed-through and unmatched ones.

StubGenerateReply

StubGenerateReply = string | { object: unknown; text?: string; }

What one route answers with.

A bare string is text (the schemaless shape); an object is structured output, and its text defaults to the JSON the real host would have returned — a schema call’s text IS the stringified object, so a fake that left it empty would differ from production in the one place a caller might read it.

StubGenerateRoute

StubGenerateRoute = StubGenerateReply | ((call) => StubGenerateReply)

How a route answers: a fixed reply, or a function of the call.

The function form is what a route with a QUEUE needs — a grader asked once per document, an executor asked once per turn — since it can shift its own script.

StubGenerateScript

StubGenerateScript = { reply: StubGenerateRoute; routes?: never; } | { reply?: never; routes: Readonly<Record<string, StubGenerateRoute>>; }

Everything stubGenerate accepts: ONE route answering every call, or a table of routes keyed by system prompt — each under a key that says which.

import { stubGenerate } from "@alexkroman1/aai/testing";

stubGenerate({ reply: "The documented answer." });
stubGenerate({ routes: { "You grade documents.": { object: { score: 1 } } } });

Why two keys rather than “a record, or a route”. The bare form was a union of a route table and a single reply, told apart at runtime by whether the object had an object key — so stubGenerate({ text: "…" }) type-checked as a table with one route named text and rejected every call. A misuse arm in the type was meant to refuse it, and could not SPEAK: the reply arm’s optional text out-scored it, so tsc printed “Property ‘object’ is missing” — the wrong remedy. With the shape named, there is nothing to disambiguate: { reply: { text } } is a reply, and a function under reply is a computed route, never mistaken for the seam itself (see ToolContextOverrides.generate).

Named because it is written down in two places — that function and the generate field of createToolContext’s overrides — and a union restated at each of them is a union that drifts.

Union Members

Type Literal

{ reply: StubGenerateRoute; routes?: never; }

reply

readonly reply: StubGenerateRoute

Answers EVERY call, whatever its system prompt.

routes?

readonly optional routes?: never

Type Literal

{ reply?: never; routes: Readonly<Record<string, StubGenerateRoute>>; }

reply?

readonly optional reply?: never

routes

readonly routes: Readonly<Record<string, StubGenerateRoute>>

One route per model ROLE, keyed by the call’s system prompt. A call whose system prompt names no route rejects, naming it; "" is the route for a call that carries none.

StubPlaceCall

StubPlaceCall = object

Sealed

What stubPlaceCall returns: the call log, and how to put the slot back.

Methods

restore()

restore(): void

Unpublish the stepFetch and the public URL. Call it in an afterEach.

Returns

void

Properties

calls

calls: StubPlacedCall[]

Every dial, in order — including refused ones.

StubPlaceCallOptions

StubPlaceCallOptions = object

Input

What stubPlaceCall may be told.

Properties

dial?

optional dial?: "accept" | StubPlaceCallRefusal | ((call) => "accept" | StubPlaceCallRefusal)

How Twilio answers each dial: "accept" (the default), or a refusal — { status: 400, code: 21219 } is a trial account calling an unverified number. A function answers per call.

otherwise?

optional otherwise?: (request) => StubStepAnswer | Promise<StubStepAnswer>

Every request that is not to Twilio’s Calls API. Default: throw, naming it — a request nobody set up is a finding (see stubFetchRoutes).

Parameters

request

StubStepRequest

Returns

StubStepAnswer | Promise<StubStepAnswer>

publicUrl?

optional publicUrl?: string

This agent’s public URL while the stub is installed — what a deployment publishes and stepPlaceCall dials back to when no agentUrl is passed. Default "https://agent.example.test"; restore() puts back the previous one.

status?

optional status?: PlacedCallStatus | "initiated" | ((call) => PlacedCallStatus | "initiated")

What each status read answers: a status, or a function of the call (its polls already counts this read). Default "completed".

StubPlaceCallRefusal

StubPlaceCallRefusal = object

Input

A Twilio refusal to stage: the HTTP status and, optionally, Twilio’s error code and message.

Properties

code?

optional code?: number

message?

optional message?: string

status

status: number

StubPlacedCall

StubPlacedCall = object

Sealed

One call a step placed, as stubPlaceCall records it.

Properties

callId

callId: string | undefined

The call id the stub answered with (CA + a counter), or undefined for a refused dial.

from

from: string

parameters

parameters: Record<string, string>

The <Parameter>s, decoded — what the answering session reads as call.parameters.

polls

polls: number

How many times StubPlaceCallOptions.status has been asked about this call.

ringTimeoutS

ringTimeoutS: number

timeLimitS

timeLimitS: number

to

to: string

webhookUrl

webhookUrl: string | undefined

The webhook Twilio fetches once the call is answered, WITHOUT the parameters (they are StubPlacedCall.parameters): https://…/phone?carrier=twilio.

StubReporter

StubReporter = object

Sealed

What stubReporter returns.

Properties

emitted

emitted: StubEmitted[]

Every chunk stepEmit() wrote, oldest first.

lines

lines: string[]

Every line stepReport() wrote, oldest first.

restore

restore: () => void

Unpublish. Call it in an afterEach — see stubReporter.

Returns

void

StubSpeech

StubSpeech = object

Sealed

What stubSpeech returns: the call log, and how to put the slot back.

Methods

restore()

restore(): void

Unpublish the synthesizer.

Calling it in an afterEach is not optional — a stub left published makes the next file’s steps speak into this one’s log, which is the kind of cross-file leak that presents as a passing test somewhere else.

Returns

void

Properties

calls

calls: StubSpeechCall[]

Every call, in order.

StubSpeechCall

StubSpeechCall = object

Sealed

One stepSpeak call, as stubSpeech records it.

Properties

apiKey

apiKey: string

The credential stepSpeak resolved out of the step env — "" when the env holds none, which the stub accepts unless StubSpeechOptions.requireApiKey is set.

language

language: string | undefined

The language code, or undefined when the caller named none.

sampleRate

sampleRate: number

The rate the audio was asked for at.

text

text: string

The text handed to the synthesizer, trimmed the way stepSpeak trims it.

voice

voice: string

The voice, with stepSpeak’s default already filled in.

StubSpeechOptions

StubSpeechOptions = object

Input

What stubSpeech may be told.

Properties

error?

optional error?: Error

Fail instead of speaking, with this error.

The half a spec cannot write by leaving the slot empty: an unpublished slot is “no synthesizer here”, which is a different sentence and a different branch from a provider that answered and refused.

pcmBytes?

optional pcmBytes?: number

Bytes of PCM to answer with, per call.

Defaults to STUB_SPEECH_PCM_BYTES, which is enough that the WAV stepSpeak frames has a plausible duration and a spec asserting on one gets a number rather than zero. A caller that cares about the exact duration sets this: at the default 24 kHz mono 16-bit, one second is 48,000 bytes.

requireApiKey?

optional requireApiKey?: boolean

Refuse, as the real synthesizer does, when the step env holds no credential. Defaults to false: the stub presents the key to nobody, so a spec should not have to vi.stubEnv("ASSEMBLYAI_API_KEY", …) just to get past a check that guards a socket it never opens. Set it for the one spec whose subject IS the missing-key sentence.

StubStepAnswer

StubStepAnswer = Response | { body?: unknown; headers?: Record<string, string>; status?: number; }

What a stubStepFetch answer may be: a whole Response, or the { status, body, headers } shorthand that JSON-encodes body.

Named because the transcription fake (stubTranscribe) hands its otherwise handler the same vocabulary, and a spec routing by URL should not have to restate the union to write one.

StubStepFetch

StubStepFetch = object

Sealed

What stubStepFetch returns.

Properties

calls

calls: StubStepRequest[]

Every request the step made, in order.

restore

restore: () => void

Unpublish. Call it in an afterEach — see stubStepFetch.

Returns

void

StubStepRequest

StubStepRequest = object

Sealed

One request a stubStepFetch recorder captured.

Properties

body

body: Uint8Array | string | undefined

The body as sent.

A STREAMING body (an async iterable — see StepFetchInit.body) is DRAINED into a Uint8Array before it reaches a spec, so an assertion reads the bytes that went out rather than an iterator it would have to consume itself — and consuming it in the spec would be consuming the one the request was going to send.

headers

headers: Record<string, string>

method

method: string

url

url: string

StubTranscribe

StubTranscribe = object

Sealed

What stubTranscribe returns.

Methods

restore()

restore(): void

Unpublish.

Not optional — a stepFetch left published answers the next file’s steps. installStubTranscribe (@alexkroman1/aai/testing/vitest) is this with the registration already done.

Returns

void

Properties

calls

calls: StubTranscribeCall[]

Every request that reached the fake, in order, each tagged with its leg.

StubTranscribeCall

StubTranscribeCall = StubStepRequest & object

Sealed

One request stubTranscribe answered, with the leg it belonged to.

Type Declaration

leg

leg: StubTranscribeLeg

Which of the four calls this was, or "other".

StubTranscribeFailure

StubTranscribeFailure = object

Input

A refusal to stage, as an HTTP answer the SDK then classifies.

Deliberately not a TranscribeError: the verdict a spec cares about (retryable, retryAfter) is computed by transcribeFailure from the status and the headers, so staging the STATUS exercises that classification and staging the error would replace it. 429 and 5xx are the transient pair, 408 counts, and everything else is terminal — see isTransientStatus on @alexkroman1/aai/step.

Properties

leg?

optional leg?: StubTranscribeLeg | readonly StubTranscribeLeg[]

Which leg refuses. Defaults to ALL FOUR, which is what a spec asserting “this flow reports a 429 as retryable” wants — it does not care which call met the limit.

message?

optional message?: string

What the body says went wrong.

Sent as { error }, which both endpoints’ readers understand — the async API’s own spelling, and one of the three transcribeFailure accepts.

retryAfterSeconds?

optional retryAfterSeconds?: number

Seconds to put in Retry-After.

The field a fan-out’s behaviour turns on: four segments that hit a per-minute limit together re-collect their 429s on a backoff nobody chose unless the header is honoured, so a spec about batching needs to be able to send one.

status?

optional status?: number

The status to answer. Defaults to 500.

StubTranscribeLeg

StubTranscribeLeg = "upload" | "submit" | "poll" | "sync" | "other" | string & object

Which transcription call a request was.

"other" is anything that is not one of the four — a model call, a feed download — which reaches the otherwise handler rather than this fake.

StubTranscribeOptions

StubTranscribeOptions = object

Input

What stubTranscribe may be told.

Properties

audioUrl?

optional audioUrl?: string

What the upload leg answers with. Defaults to a fixed fake CDN URL.

durationSec?

optional durationSec?: number

The provider’s own duration measurement, in seconds. Defaults to 60.

failure?

optional failure?: StubTranscribeFailure

Refuse at the HTTP level. See StubTranscribeFailure.

jobError?

optional jobError?: string

Fail the JOB rather than the request: the poll answers 200 with status: "error" and this reason.

A different branch from StubTranscribeOptions.failure and the one most likely to be got wrong in production code — the provider succeeded at answering and the answer is “no”. It is TERMINAL, and a flow that retried it would poll a dead job until its budget ran out.

jobIdPrefix?

optional jobIdPrefix?: string

Prefix for the job ids the submit leg mints. Defaults to "stub_transcript_", with a 1-based counter after it.

Minted rather than random for the reason stubUploads’s ids are: a spec asserting that a run journaled the job it later polled needs the id to be a value it can write down.

otherwise?

optional otherwise?: (request) => StubStepAnswer | undefined | Promise<StubStepAnswer | undefined>

Answer everything that is not a transcription call.

Publishing a stepFetch REPLACES, so a flow that transcribes AND calls a model cannot have two fakes installed — this is the seam for the second one. Returning undefined (or passing no handler) answers 404 with a body naming the URL, which is a better failure than an empty 200 a step would try to parse.

Parameters

request

StubStepRequest

Returns

StubStepAnswer | undefined | Promise<StubStepAnswer | undefined>

pendingPolls?

optional pendingPolls?: number

How many polls answer “still working” before the job completes. Defaults to 0 — the first poll finds it done.

Counted PER JOB ID, so a flow that submits two jobs sees each of them take the same number of polls. Keep it small: a caller’s polling loop usually sleeps between polls, and outside a real run that wait is not one a spec should be taking.

text?

optional text?: string | readonly string[]

The words a completed job or a sync request comes back with.

A list is consumed one per COMPLETED answer and the last repeats, matching stubGateway’s convention and for the same reason: a fan-out over segments wants a different line per segment, and a stub that ran out mid-fan-out would fail on the stub rather than on the code.

An EMPTY string is meaningful rather than a lazy default: the async API’s poll refuses it ("There is no speech in that recording", terminal), and the sync endpoint accepts it — a silent segment in a fan-out is ordinary. That asymmetry is real, and this is how a spec drives it.

StubUpload

StubUpload = Uint8Array | { bytes: Uint8Array; complete?: boolean; name?: string; type?: string; }

One file a stubUploads store answers for.

A bare Uint8Array is the common case and means “these bytes, no name”.

Union Members

Uint8Array

Type Literal

{ bytes: Uint8Array; complete?: boolean; name?: string; type?: string; }

bytes

bytes: Uint8Array

complete?

optional complete?: boolean

Whether every byte is in. Defaults to true.

false stages a STREAMED upload that is still arriving, which is the state a step polling one has to handle and the only one where stepReadUpload legitimately comes back short. Being able to write that down is most of why this field exists: a body that treats a stalled size as the end returns a transcript of most of a recording and reports success, and a spec cannot catch that without an incomplete upload to hand it.

name?

optional name?: string

type?

optional type?: string

StubUploads

StubUploads = object

Sealed

What stubUploads returns.

An OBJECT, like every other fake here (stubSpeech, stubReporter, stubStepFetch) — this one used to be the bare restore function, which made it the only stub in the family a spec had to remember was different, and left a spec asserting on a WRITE to round-trip through stepUploadInfo/stepReadUpload: the published slot, read back through the same seam the step wrote it through, to answer “did it write anything at all”.

Methods

read()

read(id): StubUploadWrite | undefined

What is stored under id right now — a seeded file or one a step wrote.

Synchronous and outside the published slot, so a spec asserting on bytes does not have to await stepReadUpload through the very seam it is testing.

Parameters

id

string

Returns

StubUploadWrite | undefined

restore()

restore(): void

Unpublish.

Not optional — a store left published makes the next file’s steps read this one’s bytes, which is the kind of cross-file leak that presents as a passing test somewhere else. installStubUploads (@alexkroman1/aai/testing/vitest) is this store with the registration already done.

Returns

void

Properties

writes

writes: StubUploadWrite[]

Every file a step wrote, in write order.

Empty unless the store was opened { writable: true }, which is what makes the pair readable as an assertion: a read-only store cannot accept a write, so writes staying empty is the same fact as the step never having tried.

StubUploadsOptions

StubUploadsOptions = object

Input

What stubUploads may be told beyond the files themselves.

Properties

idPrefix?

optional idPrefix?: string

Prefix for the ids writes are given. Defaults to "upl_stub_", with a 1-based counter after it — upl_stub_1, upl_stub_2 — so the id a step returned is a value a spec can assert on rather than a fresh UUID.

writable?

optional writable?: boolean

Accept WRITES, so a step calling stepWriteUpload can be tested.

Off by default, and deliberately: a store that silently accepts writes it was not asked for cannot fail a spec whose step wrote a file nobody meant it to, and stepWriteUpload naming a read-only store is a better failure than an upload appearing from nowhere. What a step writes is readable through stepReadUpload/stepUploadInfo on the id it was given, like any other upload.

StubUploadWrite

StubUploadWrite = object

Sealed

One file a step WROTE into a stubUploads store.

Properties

bytes

bytes: Uint8Array

Every byte written, drained from the step’s stream.

id

id: string

The minted id the step was handed back — upl_stub_1, unless renamed.

name

name: string

The name the step declared, or "" when it named none.

type

type: string

The content type the step declared, or "".

TestToolContext

TestToolContext = ToolContext & object

Sealed

A ToolContext that records what its tools sent and said.

Assignable to ToolContext wherever one is required, so it passes straight to execute. Only createToolContext makes one, so a recorder it gains is a revision rather than a break.

Type Declaration

desk

readonly desk: StubDelegate

The ctx.delegate fake — desk.calls is every subagent run the tools asked for. Present and wired on the same terms as TestToolContext.model.

interrupts

readonly interrupts: number

How many times ctx.speech.interrupt() was called.

model

readonly model: StubGenerate

The ctx.generate fake — model.calls is every prompt the tools sent.

Present on every context, so an assertion needs no null check, and WIRED whenever generate arrived as a script or as a fake. Given a bare function (or nothing at all) it is a fake nothing reaches: model.calls stays empty for the same reason TestToolContext.sent does when a test brings its own send spy — the seam belongs to the caller, and so does the log.

said

readonly said: SaidLine[]

Every ctx.speech.say, in call order. Empty when a spec passes its own speech, on the rule sent follows for send.

sent

readonly sent: SentEvent[]

Events ctx.send would put on the wire, in call order. An event the runtime would drop (over the payload cap, an over-long name, no JSON form) is not here, for the same reason it is not in the browser.

ToolBearingAgent

ToolBearingAgent = object

Input

The slice of an agent these helpers read: its tool table — the tools/ files plus every toolset agent() attached (a roster’s handoff, delegate and gated tools), gated by its dialogs.

Structural rather than AgentDef, so a spec may pass the agent’s default export, a bare { tools } literal, or anything else carrying one.

Properties

dialogs?

readonly optional dialogs?: readonly AnyDialog[]

tools

readonly tools: Readonly<Record<string, ToolDef>>

ToolContextOverrides

ToolContextOverrides = object

Input

What createToolContext accepts: a field per ToolContext field, each also taking undefined for one the caller does not have.

Not Partial<ToolContext>, and the difference is the whole point. Under exactOptionalPropertyTypes — which this repo and the scaffold both set — Partial<T> means sessionId?: string, a property that may be ABSENT but whose value may never be undefined. So a spec holding a string | undefined could not pass it, and the workaround it reached for instead was a conditional spread:

import { createToolContext } from "@alexkroman1/aai/testing";

declare const sessionId: string | undefined;
createToolContext({ ...(sessionId ? { sessionId } : {}) });

Two shipped templates had that line byte-identical, and it is the exact shape this repo’s own guard-invariants rule 22 counts as debt — so the SDK’s signature was teaching the pattern its gates refuse. Adding | undefined to every field costs nothing (an explicit undefined and an absent key both fall through to the default, because createToolContext takes the overrides through omitUndefined before spreading them) and strictly widens what compiles.

Every field is NAMED rather than mapped over keyof ToolContext. A mapped type is one a reader cannot see the members of without expanding it, and one that silently grows a field when ToolContext does — which is the moment a test double should have to decide what its default is. testing.test-d.ts pins that the two key sets agree, so a new ToolContext field fails there rather than being unoverridable.

The two MODEL seams also accept the SCRIPT their fake is built from — see their own docs below.

Properties

call?

optional call?: SessionCall

The phone call this session is, as sessionCall(ctx) will read it — recorded under the context’s sessionId the way the runtime records a WS /phone stream’s start frame. Omitted, sessionCall answers undefined, which is what a browser tab gets.

clientId?

optional clientId?: string

The device this session belongs to, as sessionClientId(ctx) will read it — recorded under the context’s sessionId the way the runtime records a socket’s ?client=. Omitted, sessionClientId answers undefined, which is what a browser tab or a phone call gets.

clientLocation?

optional clientLocation?: string

Where this session’s client is, as sessionClientLocation(ctx) — and the google_places / open_meteo builtins — will read it, cleaned by the ?location= rule. Omitted, or refused by it, it answers undefined.

clientPhone?

optional clientPhone?: string

The phone number this session’s client reported, as sessionClientPhone(ctx) will read it — recorded the way the runtime records ?phone=, in E.164 ("+1 (503) 555-0123" reads back "+15035550123"). Omitted, or not an E.164 number at all, sessionClientPhone answers undefined.

deadlineAt?

optional deadlineAt?: ToolContext["deadlineAt"]

See ToolContext.deadlineAt. Defaults to the runtime’s tool deadline, from now.

delegate?

optional delegate?: ToolContext["delegate"] | StubDelegateScript

A real ctx.delegate, or stubDelegate’s own SCRIPT — { reply } or { routes } keyed by subagent name. A function is the seam, on the same rule as generate above; the fake comes back on TestToolContext.desk.

desk?

optional desk?: StubDelegate

The stubDelegate twin of ToolContextOverrides.model.

env?

optional env?: ToolContext["env"]

See ToolContext.env. Defaults to {}.

generate?

optional generate?: ToolContext["generate"] | StubGenerateScript

A real ctx.generate, or stubGenerate’s own SCRIPT — { reply } or { routes }.

A script is built into the fake here, so the two-step every spec wrote — stubGenerate(script), destructure, createToolContext({ generate }) — is one call, and the fake comes back on TestToolContext.model.

A FUNCTION in this position is always the seam itself, and nothing else can be one: a computed route is written { reply: (call) => … }, so it cannot be mistaken for a GenerateFn the way a bare function route used to be.

messages?

optional messages?: ToolContext["messages"]

See ToolContext.messages. Defaults to [].

model?

optional model?: StubGenerate

A fake this spec built itself, to be exposed as TestToolContext.model — and, unless generate also names a function, INSTALLED as the seam.

The escape hatch under the script sugar: a caller holding a stubGenerate it wants to share across two contexts names it here rather than leaving ctx.model pointing at a fake nothing reaches.

now?

optional now?: ToolContext["now"]

See ToolContext.now. Defaults to Date.now.

random?

optional random?: ToolContext["random"]

See ToolContext.random. Defaults to a SEEDED source.

send?

optional send?: ToolContext["send"]

See ToolContext.send. Defaults to the recorder behind TestToolContext.sent.

sessionId?

optional sessionId?: ToolContext["sessionId"]

See ToolContext.sessionId. Defaults to a fresh id per call.

signal?

optional signal?: ToolContext["signal"]

See ToolContext.signal. Defaults to a signal that never aborts.

slots?

optional slots?: ToolContext["slots"]

See ToolContext.slots. Defaults to a fresh, empty, REAL slot store.

speech?

optional speech?: ToolContext["speech"]

See ToolContext.speech. Defaults to the recorder behind TestToolContext.said.

turn?

optional turn?: ToolContext["turn"]

See ToolContext.turn. Defaults to 0 — a context that belongs to no turn.

workflows?

optional workflows?: ToolContext["workflows"]

See ToolContext.workflows. Defaults to a client whose every method rejects.

ToolRunner

ToolRunner = <T>(tool, argsOrCtx?, ctx?) => Promise<Awaited<ReturnType<T["execute"]>>>

Sealed

What toolRunner hands back: a GATED runTool with the agent already supplied.

run(addItem, { item: "apple" }) checks the arguments against what execute takes and answers what it returns, so a spec reads fields off the result with no cast. The def is resolved in the agent’s table by IDENTITY (a tools/<name>.ts default export IS the registered object, and so is a speaker’s tools entry), and the call goes through the toolset’s GATE — a dialog out of state refuses it, a roster entry not on the line refuses it — exactly as the model’s own call would. A def the agent does not declare throws naming the tools it does.

Takes the arguments-or-context union runTool documents: a tool that takes none may be handed the context in their place.

Named so a caller can annotate a helper that takes one, and so the union in the second position is written down once here rather than at every call site that binds it.

Type Parameters

T

T extends object

Parameters

tool

T

argsOrCtx?

Parameters<T["execute"]>[0] | ToolContext

ctx?

ToolContext

Returns

Promise<Awaited<ReturnType<T["execute"]>>>

WorkflowContextOptions

WorkflowContextOptions = object

Input

What createWorkflowContext takes.

Properties

hooks?

optional hooks?: Record<string, unknown>

Payloads for ctx.waitFor, by token.

A token that is absent THROWS rather than hanging, because a spec that hangs reports a timeout naming the runner instead of the missing payload.

key?

optional key?: string

The run’s StartOptions.key — what ctx.key answers and what ctx.signalToken/ctx.waitForSignal derive from. Defaults to none, so a body that signals is driven with the key it would be started with.

now?

optional now?: number | (() => number)

What ctx.now() answers — a fixed number, or a function called per reach.

Defaults to WORKFLOW_CONTEXT_NOW, a FIXED instant, so a body’s derived durations are constants a spec can write down. There is no journal here, so nothing is memoized: a function is called once per reach, which is what a spec asserting on two reads (a start and an end) wants.

random?

optional random?: number | (() => number)

What ctx.random() answers. Defaults to a fixed 0.5.

results?

optional results?: Record<string, unknown>

Results to answer particular steps with, by step NAME.

Takes precedence over running the step, so it works in both modes: with runSteps: true it stubs one expensive step and leaves the rest real, and with runSteps: false it is what makes a body whose control flow READS its steps drivable at all — planAngles returning undefined otherwise reaches the fan-out below it as a missing list.

Keyed by name rather than by occurrence: a step in a loop is one name, and a spec that needs the iterations to differ wants runSteps: true with the collaborator stubbed instead.

runId?

optional runId?: string

Defaults to "wrun_test".

runSteps?

optional runSteps?: boolean

Run each step’s fn, or only record that it was reached.

Defaults to true, which is what makes this drive a REAL body. Pass false when the subject is the policy or the order — a step that is not run needs no collaborator stubbed, so such a spec stays short.

Note a recorded-only step resolves undefined, so a body that reads its result will see one. That is the honest cost of not running it.

uuid?

optional uuid?: string | (() => string)

What ctx.uuid() answers.

Defaults to a DISTINCT value per reach — "uuid-0", "uuid-1", … — because a body that mints two ids and gets one is a body whose bug the spec would hide. Not a real UUID, deliberately: a spec asserting on a shape rather than on a value is asserting on the fake.

workflow?

optional workflow?: string

The declared key. Defaults to "test".

WorkflowContextRecorder

WorkflowContextRecorder = WorkflowContext & object

Sealed

What createWorkflowContext answers: a real WorkflowContext plus its log.

Type Declaration

slept

readonly slept: RecordedSleep[]

Every ctx.sleep, in order.

steps

readonly steps: RecordedStep[]

Every step reached, in the order the body reached them.

waited

readonly waited: string[]

Every token ctx.waitFor was called with, in order.

Variables

STUB_SPEECH_PCM_BYTES

const STUB_SPEECH_PCM_BYTES: number

PCM bytes stubSpeech answers with when no size is named — ~0.25s at 24 kHz.

WORKFLOW_CONTEXT_NOW

const WORKFLOW_CONTEXT_NOW: number

The instant createWorkflowContext freezes ctx.now() at.

2026-01-01T00:00:00.000Z. Exported so a spec computes an expected duration from it rather than copying the number.