eval/vitest

@alexkroman1/aai-runtime/eval/vitest — the eval harness’s vitest door.

An agent project reaches every name here through @alexkroman1/aai-runtime/testing/vitest, which re-exports them all beside the SDK’s installers, so a test file imports from the two testing doors (/testing, /testing/vitest) whether it is a unit spec or an eval. This subpath keeps working, and it is where a stub an eval needs is added first.

One *.eval.test.ts used to reach four subpaths of two packages: the suite from here, the readers and the session from /eval, the simulated caller from /eval/simulate, and the stubs a case composes with from @alexkroman1/aai/testing and /testing/vitest. Every one of those names is re-exported here — the SAME declarations, so a type from one door is the type from another — and an eval needs no other import from either package for its harness:

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Look an order up",
  inputSchema: z.object({ orderId: z.string() }),
  execute: ({ orderId }) => ({ orderId, status: "shipped" }),
});

// file: agent.eval.test.ts
import agentDef from "virtual:aai/agent";
import { describeEval, expectCalled } from "@alexkroman1/aai-runtime/testing/vitest";
import lookUp from "./tools/look_up.ts";

describeEval(agentDef, (test) => {
  test(
    "looks the order up before answering",
    async ({ session }) => {
      // The claim takes the tool's DEF, matched by the name the agent registered.
      expectCalled(await session.say("where is order W1234?"), lookUp);
    },
    { stubReply: "Order W1234 shipped yesterday." },
  );
});

Why HERE rather than on /eval, and why not in the SDK:

/eval must stay importable without vitest. vitest is an OPTIONAL peer dependency and this module imports it (describeEval registers a suite, opens a session per case and closes it afterwards — everything defined here either INSTALLS something or OWNS a lifetime, the repo’s rule for a runner-flavoured subpath). A harness that is not vitest — a load-test stub, a recording runner — imports /eval, the runner-free half this is built on. An eval FILE is always vitest (*.eval.test.ts is a vitest tier), so the one author-facing door is the vitest one.
The SDK cannot host it. @alexkroman1/aai never imports this package (the dependency runs one way), and the harness is host runtime. So the re-export runs the other way: this subpath re-exports the SDK’s unit-level stubs, which stay DECLARED on @alexkroman1/aai/testing for a tool’s or a step’s own spec.
Capabilities are unchanged. A re-export does not move ownership: the readers stay eval’s, evalNetwork eval-network’s, the claims eval-assert’s, the simulation eval-simulate’s, and the SDK stubs aai:testing’s, each on its own epoch.

/eval/simulate was removed for this reason (its names are here and on /eval); /eval stays — it is the runner-free door, not a second author-facing one.

Functions

createRecordingWorkflows()

createRecordingWorkflows(options?): RecordingWorkflows

Build a RecordingWorkflows.

Parameters

options?

RecordingWorkflowsOptions

Returns

RecordingWorkflows

Example

import { createRecordingWorkflows, createRunSnapshot } from "@alexkroman1/aai/testing";

const workflows = createRecordingWorkflows({
  runs: [createRunSnapshot({ workflow: "remind", key: "kitchen", runId: "wrun_pending" })],
});
await workflows.start("remind", { text: "flip the laundry" }, { key: "kitchen" });
console.log(workflows.started("remind").length); // 1
console.log((await workflows.find("remind", "kitchen")).length); // 2

describeEval()

describeEval<Network, Client>(agent, define, options?): void

Declare an eval suite for agent.

Generic over the suite’s network and workflows only so a case’s ctx.network and ctx.workflowClient are typed by them (see EvalTestContext.network); nobody writes the type arguments, they are read off options.

describeEval(agentDef, (test) => {
  test(
    "offers to take an order",
    async ({ session }) => {
      const turn = await session.say("hi, what can you do?");
      expect(turn.text).toMatch(/order/i);
    },
    { stubReply: "I can take an order for you." },
  );
});

Type Parameters

Network

Network extends EvalNetwork = never

Client

Client extends WorkflowClient = never

Parameters

agent

AgentDef

define

(test) => void

options?

Omit<DescribeEvalOptions, "workflows" | "network"> & object

Returns

void

describeTextEval()

describeTextEval(agent, define, options?): void

Declare an eval suite for a TEXT agent.

import { agent } from "@alexkroman1/aai";
import { toolNames } from "@alexkroman1/aai-runtime/eval";
import { describeTextEval } from "@alexkroman1/aai-runtime/eval/vitest";
import { expect } from "vitest";

const agentDef = agent({ name: "Coder", mode: "text" });

describeTextEval(agentDef, (test) => {
  test(
    "reads a file before it edits one",
    async ({ agent: coder }) => {
      const turn = await coder.send("rename `total` to `sum` in cart.ts");
      expect(toolNames(turn.toolCalls)).toContain("read_file");
    },
    { stubReply: [{ tool: "read_file", args: { path: "cart.ts" } }, "Renamed it."] },
  );
});

Parameters

agent

AgentDef

define

(test) => void

options?

DescribeTextEvalOptions

Returns

void

describeWorkflowEval()

describeWorkflowEval(agent, define, options?): void

Declare an eval suite for a workflow app.

The signature mirrors describeEval down to the two things a LINTER decides — the callback parameter is named test (noMisplacedAssertion matches the callee identifier) and a case body takes a DESTRUCTURED context (noDoneCallback reads the first positional parameter of an async test callback as jest’s done). Do not tidy either.

Parameters

agent

AgentDef

define

(test) => void

options?

Omit<EvalWorkflowsOptions, "agent"> & object

Returns

void

dialogRefusalPattern()

dialogRefusalPattern(state?): RegExp

A pattern matching the sentence a dialog() gate refuses with — optionally pinned to the state it names.

For a SPEC. A gated tool called out of state answers a ToolFailure whose error is this sentence, and every template spec that asserts a gate held used to spell a regex for it by hand. Two kinds of spec read it, and the pattern serves both: a unit test holds the ToolFailure itself (prefer expectDialogRefused there, which also throws on a success), while an eval reads a tool result off the event stream as a SERIALIZED string, where the state’s quotes arrive escaped (\"identifying\"). The pattern admits the escaping, so one matcher reads both.

With no state, it matches any refusal — for a spec that pins the state a line later, or whose subject is that the body did not run rather than where the conversation was.

Parameters

state?

string

The state the refusal must name, as DialogPosition.state spells it ("identifying", "onCall.inbox"). Matched literally.

Returns

RegExp

Example

import { dialogRefusalPattern } from "@alexkroman1/aai/testing";

const refused = 'Not available yet: this conversation is at "identifying". Verify the caller first.';
dialogRefusalPattern("identifying").test(refused); // true
dialogRefusalPattern("transferred").test(refused); // false
dialogRefusalPattern().test(refused); // true

dialogResultSchema()

dialogResultSchema<T>(result): ZodObject<{ done: ZodBoolean; instruction: ZodOptional<ZodString>; result: T; state: ZodString; }, $strip>

The envelope a gated tool answers with, as a schema around the tool’s own.

expectDialogOk unwraps a value a spec HOLDS. A harness that holds the SERIALIZED copy the model was handed — a transcript, a recording — reads it back through a schema, so it needs the same envelope as a schema rather than as a function, and three shipped evals had each written it out: z.object({ result, state: z.string(), done: z.boolean() }), under a comment saying the shape was the SDK’s. It is, and this is where it lives (an eval reading a live turn takes the DEF instead: toolResultIn(turn.toolCalls, setStay) is typed by it). result is whatever the author’s execute returned, state is where the call landed, done whether that state is final, and instruction is the state’s own brief when it declares one — the fields of DialogToolResult, which a dialog.tool writes and no tool file does.

Parsing rather than casting is what makes a template that stopped carrying its position fail naming the field, instead of a later expect reading undefined.state.

Type Parameters

T

T extends ZodType<unknown, unknown, $ZodTypeInternals<unknown, unknown>>

The schema of the tool’s OWN result, under result.

Parameters

result

T

What the tool’s execute answers with.

Returns

ZodObject<{ done: ZodBoolean; instruction: ZodOptional<ZodString>; result: T; state: ZodString; }, $strip>

Example

import { dialogResultSchema } from "@alexkroman1/aai/testing";
import { z } from "zod";

// Holding the serialized result yourself, it is one parse:
const Stay = dialogResultSchema(z.object({ options: z.string() }));
const stay = Stay.parse(
  JSON.parse('{"result":{"options":"garden view"},"state":"booking.room","done":false}'),
);
stay.state; // "booking.room"
stay.result.options; // "garden view"

eventsOf()

eventsOf<E, K>(events, type): Extract<E, { type: K; }>[]

Every event in events named type, in order, typed as that member.

import type { SessionEvent } from "@alexkroman1/aai";
import { eventsOf } from "@alexkroman1/aai/testing";

declare const recorded: SessionEvent[];
const calls = eventsOf(recorded, "tool.called");
console.log(calls.map((e) => e.toolName));

Type Parameters

E

E extends object

K

K extends string

Parameters

events

Iterable<E>

type

K

Returns

Extract<E, { type: K; }>[]

installStubSpeech()

installStubSpeech(options?): StubSpeech

Publish a synthesizer that records what it was asked to say, restored when this test finishes.

stubSpeech with the bookkeeping done — see it for the call log’s shape, the silence it answers with, and how to make it fail instead.

Parameters

options?

StubSpeechOptions

Returns

StubSpeech

installStubStepDelegate()

installStubStepDelegate(script): StubStepDelegate

Publish a fake subagent runner for stepDelegate, restored when this test finishes.

stubStepDelegate with the bookkeeping done. It is the only way to drive an exported step that delegates — the real slot THROWS when nothing has published, deliberately, because there is no degraded version of running a model loop.

Parameters

script

StubDelegateScript

Returns

StubStepDelegate

installStubStepFetch()

installStubStepFetch(answer?): StubStepFetch

Publish a fake stepFetch, restored when this test finishes.

stubStepFetch with the bookkeeping done — see it for why a step’s HTTP goes through a published slot rather than the global, and what the recorded request carries.

Parameters

answer?

(request) => StubStepAnswer | Promise<StubStepAnswer>

Called per request. Defaults to an empty 200.

Returns

StubStepFetch

installStubTranscribe()

installStubTranscribe(options?): StubTranscribe

Answer AssemblyAI’s transcription endpoints in memory, restored when this test finishes.

stubTranscribe with the bookkeeping done — see it for the four legs it routes, why a refusal is staged as an HTTP status rather than as a TranscribeError, and why it takes an otherwise handler.

Parameters

options?

StubTranscribeOptions

Returns

StubTranscribe

installStubUploads()

installStubUploads(files, options?): StubUploads

Publish an in-memory upload store, restored when this test finishes.

stubUploads with the bookkeeping done — see it for what the store serves, why writes are opt-in, and why the minted ids count up.

Parameters

files

Readonly<Record<string, StubUpload>>

options?

StubUploadsOptions

Returns

StubUploads

Example

import { stepReadUpload, stepWriteUpload } from "@alexkroman1/aai/step";

export async function ingest(uploadId: string) {
  const { bytes } = await stepReadUpload(uploadId);
  return stepWriteUpload(bytes);
}

// file: ingest.test.ts
import { installStubUploads } from "@alexkroman1/aai/testing/vitest";
import { expect, test } from "vitest";
import { ingest } from "./workflows/ingest.ts";

test("the step reads the recording it was given", async () => {
  const uploads = installStubUploads({ upl_1: new Uint8Array(5000) }, { writable: true });
  await ingest("upl_1");
  expect(uploads.writes).toHaveLength(1);
});

isEvent()

isEvent<E, K>(event, type): event is Extract<E, { type: K }>

Whether event is the one named type — a type guard, so the branch it guards reads that member’s fields without a cast.

import type { SessionEvent } from "@alexkroman1/aai";
import { isEvent } from "@alexkroman1/aai/testing";

declare const recorded: SessionEvent[];
const last = recorded.at(-1);
if (last && isEvent(last, "tool.called")) console.log(last.toolName);

Type Parameters

E

E extends object

K

K extends string

Parameters

event

E

type

K

Returns

event is Extract<E, { type: K }>

resolveEvalMode()

resolveEvalMode(agent, hostEnv?, overrides?): object

Live if this machine can be, stub if it cannot — unless a caller has said which it wants.

AAI_REQUIRE_EVAL is for a pipeline that means to MEASURE: with it set, a missing credential is a failure instead of a quiet downgrade to a wiring check. AAI_EVAL_STUB is the opposite instruction, and CI wants it — a required check must not start spending tokens the day a key reaches its environment, and must not become a flaky gate on a live model’s behaviour.

Parameters

agent

AgentDef

hostEnv?

Record<string, string | undefined>

overrides?

What the CASE overrides, which decides the credential question with it.

Without this the mode was read off the AGENT alone, so describeEval(def, define, { llm: llm({ provider: "assemblyai", model }) }) on an agent declaring anthropic() announced “SCRIPTED — ANTHROPIC_API_KEY is not set” while holding the key the run would actually have used. Measured on custom-pipeline-agent: the override was honoured by the session and ignored by the gate, so a case could not be run live at all.

llm?

LlmProvider

Returns

object

mode

mode: EvalMode

reason

reason: string

resolveWorkflowEvalMode()

resolveWorkflowEvalMode(agent, hostEnv?): object

resolveEvalMode for a WORKFLOW app, whose credentials are a different question.

Split rather than folded in because the two gates read different fields and the wrong one is silent: a mode: "workflow-app" agent needs no provider credential, so evalCredentials reports every workflow app ready and a keyless run goes LIVE — then every case fails on a 401 three layers down. evalWorkflowCredentials reads requiredEnv, which is the only thing a workflow app declares its credentials in.

Parameters

agent

AgentDef

hostEnv?

Record<string, string | undefined>

Returns

object

mode

mode: EvalMode

reason

reason: string

stubGatewayRoute()

stubGatewayRoute(replies, options?): StubGatewayRoute

A gateway reply for a step that goes through the PUBLISHED stepFetch slot rather than the global fetch — a ROUTE to compose, not a fake to install. stubGateway says which of the three gateway fakes fits which seam.

stubGateway answers over globalThis.fetch, which is the wrong seam whenever anything has published a stepFetch: publishing REPLACES, so a flow that transcribes AND calls a model — or fetches a page and calls a model — can install only one fake and has to route by URL inside it. Seven eval files did exactly that, and each hand-typed the same two things:

The envelope. { body: { choices: [{ message: { content } }] } }, written out six times in six spellings. It is a WIRE shape, so a typo in it does not fail — stepGenerate reads no content and reports an empty completion, i.e. the fake and the code under test disagree and the case blames the code.
The cursor. contents.at(Math.min(next, contents.length - 1)), re-derived twice, because a model call inside a LOOP cannot know how many calls it will make: a script that repeats one line can only drive the loop into its budget, and one that runs out mid-loop fails on the script. The last reply repeats, which is stubGateway’s convention and now literally the same code.

And it hands back DECODED calls — prompt, system, body, headers — which is the half no hand-rolled version had. Reading what the model was ASKED off a StubStepRequest means String(call.body), i.e. the raw JSON of the whole request; one eval asserted its prompts that way and was really asserting against the serialized model and temperature too.

import { stubGatewayRoute } from "@alexkroman1/aai/testing";
import { installStubStepFetch } from "@alexkroman1/aai/testing/vitest";
import { expect, test } from "vitest";

const PAGE_HTML = "<html><body>the brief</body></html>";

test("the reviewer is asked about the brief", async () => {
  const model = stubGatewayRoute(['{"verdict":"ship"}']);
  installStubStepFetch((request) => model.route(request) ?? { body: PAGE_HTML });
  // … run the workflow …
  expect(model.calls[0]?.prompt).toContain("the brief");
});

Parameters

replies

string | readonly string[]

Completion contents, in order; the last repeats. A bare string is one reply.

options?

StubGatewayOptions

Returns

StubGatewayRoute

Interfaces

StubGatewayRoute

Sealed

A gateway answer for a stepFetch-published slot, plus what it was asked.

Properties

calls

calls: StubGatewayCall[]

Every completion request this route answered, DECODED, in call order.

route

route: (request) => StubStepAnswer | undefined

Answers a completion request and undefined for anything else, so the caller composes it: as the first leg of a stubFetchRoutes list (where an unexpected request is a finding by default), ?? { body: html } for a flow that also fetches a page, or straight into stubTranscribe’s otherwise.

Parameters

request

StubStepRequest

Returns

StubStepAnswer | undefined

StubStepDelegate

Sealed

A fake stepDelegate: the calls it recorded, and the slot to give back.

Methods

restore()

restore(): void

Unpublish the runner.

Calling it in an afterEach is not optional — a stub left published makes the next file’s steps delegate into this one’s log, which is the kind of cross-file leak that presents as a passing test somewhere else.

Returns

void

Properties

calls

calls: StubDelegateCall[]

Every call, in order — the same log stubDelegate keeps.

Type Aliases

DescribeEvalOptions

DescribeEvalOptions = Omit<EvalSessionOptions, "agent"> & object

Input

What describeEval takes beyond the agent.

The session options, plus workflowOptions for the engine it opens per case when the agent declares workflows. That second one is not symmetry for its own sake: a workflow-starting tool’s STEPS make provider calls, and the only honest way to evaluate which tool the desk reached for — without paying for five gateway calls and a real web search per case, and without a 429 failing the run outright because a step’s maxRetries is inert here — is to script the step’s HTTP while leaving the SESSION’s model live. Both templates that hand off to a run had to install that inside the case body, which worked only because the engine publishes nothing when nobody passed one.

Type Declaration

network?

readonly optional network?: EvalNetwork | (() => EvalNetwork)

A fake network for every case — an evalNetwork(...), or a FACTORY returning one, called afresh for every case and every AAI_EVAL_REPEAT repeat.

It becomes all three fetches a case’s code can reach: the global fetch a custom tool calls, the fetch the builtins take, and the step fetch a workflow step (or sendToChannel) reads. A request no route answers is REFUSED and logged; only the live model’s own provider hosts pass through, worked out from the agent’s llm (in a scripted run, not even those). The global is swapped for the whole SUITE rather than per case, so an onSessionEnd still running after a case closed (the session waits 10 seconds for it, then stops waiting) is refused into that case’s log rather than reaching the real network.

Keep a route’s STATE (rows a fake database holds) in the network’s own state (evalNetwork({ state, routes })): an instance’s log AND state are rebuilt per case and per repeat, and the case reads it, typed, as ctx.network.state. State a handler keeps in its closure is not reset — use a factory for that — and state carried from one repeat into the next makes the second repeat measure the first. Mutually exclusive with fetch, which it replaces.

steps?

readonly optional steps?: EvalStepsOptions

The STEP fetch every case’s workflow steps dial, scripted — routes, an optional hold on the first answer, and a drain at case end: the harness releases the hold and settleAll()s the case’s workflow app before the routes are unpublished, so no run outlives its fakes. See EvalStepsOptions. Installed afresh per case and per repeat; the session’s own model traffic stays on the global fetch, so a live case still measures the agent. Mutually exclusive with network and workflowOptions.stepFetch, which answer the same requests.

workflowOptions?

readonly optional workflowOptions?: Omit<EvalWorkflowsOptions, "agent">

DescribeTextEvalOptions

DescribeTextEvalOptions = Omit<EvalTextAgentOptions, "agent">

Input

What describeTextEval takes beyond the agent.

EvalCaseOptions

EvalCaseOptions = object

Input

What a case gets to say about how it should be run.

Properties

call?

readonly optional call?: SessionCall | null

This case’s placed phone call, over the suite’s — what sessionContext receives as call. See EvalSessionOptions.call; a call the hook refuses lands on session.refused. null is “not a placed call”: a calling agent’s refusal of a session no carrier started, inside a suite whose other cases are all the one call.

clientId?

readonly optional clientId?: string | null

WHO this case’s session is, over the suite’s own (DescribeEvalOptions) — the client id sessionClientId(ctx) answers. See EvalSessionOptions.clientId. null is “no client id for this case”, whatever the suite set; absent is “the suite’s”.

Per case because a suite’s cases are rarely all the same caller: a speaker agent’s “a device with no client id is refused” case sits beside twenty that run as the kitchen speaker, and it is written { clientId: null }.

live?

readonly optional live?: boolean

This case only means something against a live model — it is SKIPPED in stub mode. Use it for a claim no script can honestly satisfy: a tool the model has to choose for itself, a refusal, a judgement.

network?

readonly optional network?: EvalNetwork | (() => EvalNetwork)

This case’s fake network, over the suite’s — see DescribeEvalOptions.network. A case needing routes of its own (a service that answers differently in this one scenario) passes them here.

phone?

readonly optional phone?: string | null

This case’s reported phone number, over the suite’s. See EvalSessionOptions.phone. null is “no number for this case”.

scripted?

readonly optional scripted?: boolean

The mirror: this case only means something against a SCRIPT, and is skipped against a live model.

It is not a symmetry for its own sake — three cases needed it. A gate can only be observed refusing if something CALLS the gated tool, and a competent model declines to (measured: tabletop-rpg-agent’s game-over route is a tool its own prompt forbids unprompted; a dispatcher calls resources_get_available first and never trips the busy-unit refusal; a visit_webpage at a private address is the SSRF screen’s own case and a live model sensibly refuses to try). Without this marker each cost a red live run and got weakened.

stubGenerate?

readonly optional stubGenerate?: StubScript

What a SCRIPTED ctx.generate answers with — its OWN script, walked by its own cursor.

Separate from EvalCaseOptions.stubReply because ctx.generate resolves a model INSTANCE of its own, in parallel with the turn’s: one script would need element 0 to be the turn’s first move and the first generate answer simultaneously. A tool that reasons with a model — a grader, a planner, a rewriter — is the shape this exists for, and two shipped templates’ central tools are exactly that. For the schema overload, write the object as the JSON string the model would have returned.

stubReply?

readonly optional stubReply?: StubScript

What a SCRIPTED model does when this suite runs without a key — one entry per model call, the last line repeating. A string is a line the agent says; { tool, args } is a tool call, which is what makes a stub run worth having for an agent that HAS tools:

no-check: the fence is one FIELD of this type, and its only compilable reading is a labelled statement inside a block — it would type-check whatever the field were called, so checking it asserts nothing about EvalCaseOptions.stubReply. Kept as a fragment deliberately, not because it cannot compile: a no-check that would pass is unclaimed headroom, and this one would pass for the wrong reason.

{ stubReply: [{ tool: "look_up", args: { orderId: "W1234" } }, "It shipped."] }

Choose it so the case’s own assertions still hold: the point of a stub run is that the case really executes, and a stub the case then fails against measures nothing.

EvalMode

EvalMode = "live" | "stub"

How the suite is running, and why.

EvalSteps

EvalSteps = object

Sealed

What a case is handed as ctx.steps.

Methods

release()

release(): void

Let the held request answer. Idempotent; the harness calls it at case end.

Returns

void

Properties

held

readonly held: boolean

Whether a request is parked on the gate right now.

hits

readonly hits: readonly FetchRouteHit[]

Every step request as it was ANSWERED, with its outcome and status — stubFetchRoutes’ own log, which records a request once it has resolved, so a held one is not here until released.

requests

readonly requests: readonly FetchRouteRequest[]

Every step request as it ARRIVED, in order — a held one included, which is what makes “the run is waiting on the provider” a claim about this list.

EvalStepsOptions

EvalStepsOptions = object

Input

The steps option: the routes a case’s step HTTP is answered from, and whether the first answer is HELD.

Properties

drain?

readonly optional drain?: boolean

Release the gate and settleAll() the case’s workflow app when the case ends, so no run outlives its routes. Defaults to true; false for a case that drains by hand or means to leave a run abandoned (the app’s close() still warns).

hold?

readonly optional hold?: "first" | "none" | ((request) => boolean)

Park the FIRST answered request until the case calls steps.release() (or ends), so a run is observably still in flight: a durable sleep is skipped in an eval, and an unheld run burns its whole poll loop in milliseconds. "first" holds the first request any route answered; a predicate holds the first request it matches (the first GET, say). Defaults to "none".

routes

readonly routes: FetchRouteTable | readonly FetchRouteHandler[]

What a step’s stepFetch is answered from — a stubFetchRoutes table, or an ordered handler list (how stubGatewayRoute’s route composes). The STEP fetch only: a live case’s own model traffic stays on the global. A request no route answers is a finding and rejects naming it.

EvalTest

EvalTest<Network, Client> = (name, body, options?) => void

Declare one eval case. The session is opened for it and closed after it.

Two things about this signature are decided by a LINTER rather than by taste, both A/B’d against Biome 2.5 and both invisible until a user’s own project lights up red on a file the SDK told them to write:

The parameter is named test. noMisplacedAssertion matches on the CALLEE IDENTIFIER and nothing else, so an expect inside evalTest(…) is an error while the identical body inside test(…) is fine.
The body takes a DESTRUCTURED context, not the session positionally. noDoneCallback reads the first parameter of an async test callback as jest’s done, so async (session) => … is an error; async ({ session }) => … is not — and it is vitest’s own fixture shape, which is what a reader already expects.

Type Parameters

Network

Network extends EvalNetwork = never

Client

Client extends WorkflowClient = never

Parameters

name

string

body

(ctx) => Promise<void>

options?

[Network] extends [never] ? EvalCaseOptions : Omit<EvalCaseOptions, "network"> & object

Returns

void

EvalTestContext

EvalTestContext = object

Sealed

What a case body is handed: its own session, which model it is on, and the workflow app behind it.

A simulated caller and a model-graded judge are NOT on it: a case that wants them builds the pair from session and mode with evalSimulation (on this same @alexkroman1/aai-runtime/eval/vitest), a surface versioned on its own.

Properties

mode

readonly mode: EvalMode

Which model this run got. A case may branch on it, and most should not.

The one branch that is always right

A value a SCRIPT determined may only be asserted under mode === "stub". stubReply and stubGenerate are what make a value predictable, so pinning one against a live model is pinning the script — and it presents as the agent misbehaving, which is the expensive part. Three shipped template evals had it, and each read as a defect in the template until the script was checked:

word-game-agent asserted playerSaid: "Is it a zebra crossing?", the exact remark of a scripted player, in a game whose word is drawn at random. Live, “Is it a zebra?” is a perfectly good wrong guess.
executive-inbox-agent pinned 2 closed / 6 queued, a split decided by eight live triage verdicts. A run that found every email worth answering failed on “expected [] to have a length of 2”.
topic-briefing-agent required a verdict word from a subagent its own tool documents as allowed to come back unusable.

So: assert the INVARIANT in both modes — the tool was called, the verdict and the score agree, nothing was sent before a yes — and put the exact strings behind the branch.

test("a wrong guess is relayed without a point", async ({ session, mode }) => {
  const relayed = await play(session);
  // True either way: the player answered and the round stands.
  expect(relayed.playerSaid.length).toBeGreaterThan(0);
  if (relayed.verdict === "wrong") expect(relayed.score).toBe(0);
  // Only a script can pin the words.
  if (mode === "stub") expect(relayed.playerSaid).toBe("Is it a zebra crossing?");
});

A case that cannot be written that way wants { scripted: true } instead, which skips it live rather than weakening it — see EvalCaseOptions.scripted.

network

readonly network: EvalNetwork | undefined

The fake network every fetch of this case went through. Its log holds THIS case’s requests (THIS repeat’s, under AAI_EVAL_REPEAT), so a case asserts on network.calls("textbelt.com") or network.expectNoOutbound(/twilio/) without filtering out another run’s traffic, and reads its routes’ network.state.

TYPED BY WHAT WAS PASSED, in the body a case hands EvalTest: in a suite given network it is exactly that network’s type — EvalNetwork<{ calls: Map<…> }> for one built with state — so there is no undefined to guard and no cast to reach the state. A case’s own network must be of the suite’s type, so the type holds for it too. In a suite given none it is EvalNetwork | undefined (this declaration) — undefined at runtime unless the case passed one.

session

readonly session: EvalSession

Open for this case, closed after it.

steps

readonly steps: EvalSteps | undefined

The case’s scripted step HTTP, when the suite passed steps: its request log, whether a request is held, and release() — which the harness calls at case end anyway, before draining the workflow app. undefined in a suite given none.

workflowClient

readonly workflowClient: WorkflowClient | undefined

The client this session’s ctx.workflows IS: the suite’s own (describeEval’s workflows — the one its factory built for THIS case and repeat), else the engine’s workflows.client, else undefined for an agent that declares no workflows and was given none.

Typed by what the suite passed, in the body a case hands EvalTest: a suite whose factory returns a recording client — () => createRecordingWorkflows({ workflows: agentDef.workflows }) from @alexkroman1/aai/testing, which records every start and runs nothing — reads workflowClient.started("remind") and seeds the runs find answers with workflowClient.seed(...), typed, with no module-level log to reset.

workflows

readonly workflows: EvalWorkflows | undefined

The workflow app behind this session’s ctx.workflows, for an agent that declares workflows — undefined for one that does not.

Opened per case and closed after it, and it is what makes a tool calling ctx.workflows.start runnable at all: the real client the runtime would build cannot start an untransformed body. A case reads the run its tool started with workflows.settle(runId).

The engine under it is NOT durable — see eval/workflow-engine.ts before writing a claim about a run.

EvalTextTest

EvalTextTest = (name, body, options?) => void

Declare one text eval case. The conversation is opened and closed for it.

Parameters

name

string

body

(ctx) => Promise<void>

options?

EvalCaseOptions

Returns

void

EvalTextTestContext

EvalTextTestContext = object

Sealed

What a text case body is handed: its own conversation and the mode. A simulated caller is evalSimulation({ target: agent, … }) (on this same @alexkroman1/aai-runtime/eval/vitest), as for a voice case.

Properties

agent

readonly agent: EvalTextAgent

Opened for this case, released after it.

mode

readonly mode: EvalMode

Which model this run got. A case may branch on it, and most should not.

EvalWorkflowCaseOptions

EvalWorkflowCaseOptions = object

Input

What a workflow case gets to say about how it should be run.

Properties

live?

readonly optional live?: boolean

This case only means something against real providers — it is SKIPPED in stub mode.

Reach for it when a step MUST reach the far side for the claim to mean anything: a transcript that has to be of the audio, a summary that has to be of the page. A case that can be scripted should be, because a scripted run is what a pipeline with no key can still gate on.

EvalWorkflowTest

EvalWorkflowTest = (name, body, options?) => void

Declare one workflow eval case. The app is opened for it and closed after it.

Parameters

name

string

body

(ctx) => Promise<void>

options?

EvalWorkflowCaseOptions

Returns

void

EvalWorkflowTestContext

EvalWorkflowTestContext = object

Sealed

What a workflow case body is handed.

Properties

app

readonly app: EvalWorkflows

Opened for this case, closed after it.

mode

readonly mode: EvalMode

Which mode this run got.

Unlike a voice case, a workflow case is EXPECTED to branch on it: it is what decides whether to install a fake for a provider a step would otherwise really dial.

steps

readonly steps: EvalSteps | undefined

The case’s scripted step HTTP, when the suite passed steps — see DescribeEvalOptions.steps; undefined in a suite given none.

RecordingWorkflows

RecordingWorkflows = WorkflowClient & object

Sealed

A WorkflowClient that records, plus its log.

start records and resolves a fresh run id, and the run it “started” is visible to get/find/recent as running — so a tool that checks for a pending run before starting a second one sees its own first start. cancel marks a known, unfinished run cancelled and resolves true (else false); wakeUp resolves 0; lastLine resolves undefined. The progress-channel reads (stream, streamTail, signal, publicWebhookUrl) REJECT, as createStubWorkflows’ do — spread over it to answer one.

Type Declaration

cancelled

readonly cancelled: string[]

Every run id cancel was called with, in order, whatever it resolved.

signals

readonly signals: RecordedSignal[]

Every signalRun, in order — delivered (true) without a run to deliver to.

starts

readonly starts: RecordedStart[]

Every start, in order.

seed()

seed(…runs): void

Add runs for the reads to answer from, e.g. inside a case before say().

Parameters

runs

…WorkflowRunSnapshot[]

Returns

void

started()

started(workflow?): RecordedStart[]

The starts of one workflow — by declared name or by def — or all of them.

Parameters

workflow?

string | AnyWorkflowDef

Returns

RecordedStart[]

RecordingWorkflowsOptions

RecordingWorkflowsOptions = object

Input

What createRecordingWorkflows takes.

Properties

runIdPrefix?

optional runIdPrefix?: string

Prefix of the minted run ids, numbered from 1. Defaults to "wrun_rec_".

runs?

optional runs?: readonly WorkflowRunSnapshot[]

Runs the reads answer from before anything starts — a reminder already pending, a job that failed. Build them with createRunSnapshot; more can be added later with seed.

workflows?

optional workflows?: Readonly<Record<string, AnyWorkflowDef>>

The agent’s declared workflows — pass agentDef.workflows — so a start by DEF is recorded under its declared name, and listing() reports them. A def not in it is refused, as the real client refuses it. Without it, a def is recorded under its description and matched by identity.

StubSpeech

StubSpeech = object

Sealed

What stubSpeech returns: the call log, and how to put the slot back.

Methods

restore()

restore(): void

Unpublish the synthesizer.

Calling it in an afterEach is not optional — a stub left published makes the next file’s steps speak into this one’s log, which is the kind of cross-file leak that presents as a passing test somewhere else.

Returns

void

Properties

calls

calls: StubSpeechCall[]

Every call, in order.

StubSpeechOptions

StubSpeechOptions = object

Input

What stubSpeech may be told.

Properties

error?

optional error?: Error

Fail instead of speaking, with this error.

The half a spec cannot write by leaving the slot empty: an unpublished slot is “no synthesizer here”, which is a different sentence and a different branch from a provider that answered and refused.

pcmBytes?

optional pcmBytes?: number

Bytes of PCM to answer with, per call.

Defaults to STUB_SPEECH_PCM_BYTES, which is enough that the WAV stepSpeak frames has a plausible duration and a spec asserting on one gets a number rather than zero. A caller that cares about the exact duration sets this: at the default 24 kHz mono 16-bit, one second is 48,000 bytes.

requireApiKey?

optional requireApiKey?: boolean

Refuse, as the real synthesizer does, when the step env holds no credential. Defaults to false: the stub presents the key to nobody, so a spec should not have to vi.stubEnv("ASSEMBLYAI_API_KEY", …) just to get past a check that guards a socket it never opens. Set it for the one spec whose subject IS the missing-key sentence.

StubStepFetch

StubStepFetch = object

Sealed

What stubStepFetch returns.

Properties

calls

calls: StubStepRequest[]

Every request the step made, in order.

restore

restore: () => void

Unpublish. Call it in an afterEach — see stubStepFetch.

Returns

void

StubTranscribe

StubTranscribe = object

Sealed

What stubTranscribe returns.

Methods

restore()

restore(): void

Unpublish.

Not optional — a stepFetch left published answers the next file’s steps. installStubTranscribe (@alexkroman1/aai/testing/vitest) is this with the registration already done.

Returns

void

Properties

calls

calls: StubTranscribeCall[]

Every request that reached the fake, in order, each tagged with its leg.

StubTranscribeOptions

StubTranscribeOptions = object

Input

What stubTranscribe may be told.

Properties

audioUrl?

optional audioUrl?: string

What the upload leg answers with. Defaults to a fixed fake CDN URL.

durationSec?

optional durationSec?: number

The provider’s own duration measurement, in seconds. Defaults to 60.

failure?

optional failure?: StubTranscribeFailure

Refuse at the HTTP level. See StubTranscribeFailure.

jobError?

optional jobError?: string

Fail the JOB rather than the request: the poll answers 200 with status: "error" and this reason.

A different branch from StubTranscribeOptions.failure and the one most likely to be got wrong in production code — the provider succeeded at answering and the answer is “no”. It is TERMINAL, and a flow that retried it would poll a dead job until its budget ran out.

jobIdPrefix?

optional jobIdPrefix?: string

Prefix for the job ids the submit leg mints. Defaults to "stub_transcript_", with a 1-based counter after it.

Minted rather than random for the reason stubUploads’s ids are: a spec asserting that a run journaled the job it later polled needs the id to be a value it can write down.

otherwise?

optional otherwise?: (request) => StubStepAnswer | undefined | Promise<StubStepAnswer | undefined>

Answer everything that is not a transcription call.

Publishing a stepFetch REPLACES, so a flow that transcribes AND calls a model cannot have two fakes installed — this is the seam for the second one. Returning undefined (or passing no handler) answers 404 with a body naming the URL, which is a better failure than an empty 200 a step would try to parse.

Parameters

request

StubStepRequest

Returns

StubStepAnswer | undefined | Promise<StubStepAnswer | undefined>

pendingPolls?

optional pendingPolls?: number

How many polls answer “still working” before the job completes. Defaults to 0 — the first poll finds it done.

Counted PER JOB ID, so a flow that submits two jobs sees each of them take the same number of polls. Keep it small: a caller’s polling loop usually sleeps between polls, and outside a real run that wait is not one a spec should be taking.

text?

optional text?: string | readonly string[]

The words a completed job or a sync request comes back with.

A list is consumed one per COMPLETED answer and the last repeats, matching stubGateway’s convention and for the same reason: a fan-out over segments wants a different line per segment, and a stub that ran out mid-fan-out would fail on the stub rather than on the code.

An EMPTY string is meaningful rather than a lazy default: the async API’s poll refuses it ("There is no speech in that recording", terminal), and the sync endpoint accepts it — a silent segment in a fan-out is ordinary. That asymmetry is real, and this is how a spec drives it.

StubUploads

StubUploads = object

Sealed

What stubUploads returns.

An OBJECT, like every other fake here (stubSpeech, stubReporter, stubStepFetch) — this one used to be the bare restore function, which made it the only stub in the family a spec had to remember was different, and left a spec asserting on a WRITE to round-trip through stepUploadInfo/stepReadUpload: the published slot, read back through the same seam the step wrote it through, to answer “did it write anything at all”.

Methods

read()

read(id): StubUploadWrite | undefined

What is stored under id right now — a seeded file or one a step wrote.

Synchronous and outside the published slot, so a spec asserting on bytes does not have to await stepReadUpload through the very seam it is testing.

Parameters

id

string

Returns

StubUploadWrite | undefined

restore()

restore(): void

Unpublish.

Not optional — a store left published makes the next file’s steps read this one’s bytes, which is the kind of cross-file leak that presents as a passing test somewhere else. installStubUploads (@alexkroman1/aai/testing/vitest) is this store with the registration already done.

Returns

void

Properties

writes

writes: StubUploadWrite[]

Every file a step wrote, in write order.

Empty unless the store was opened { writable: true }, which is what makes the pair readable as an assertion: a read-only store cannot accept a write, so writes staying empty is the same fact as the step never having tried.

StubUploadsOptions

StubUploadsOptions = object

Input

What stubUploads may be told beyond the files themselves.

Properties

idPrefix?

optional idPrefix?: string

Prefix for the ids writes are given. Defaults to "upl_stub_", with a 1-based counter after it — upl_stub_1, upl_stub_2 — so the id a step returned is a value a spec can assert on rather than a fresh UUID.

writable?

optional writable?: boolean

Accept WRITES, so a step calling stepWriteUpload can be tested.

Off by default, and deliberately: a store that silently accepts writes it was not asked for cannot fail a spec whose step wrote a file nobody meant it to, and stepWriteUpload naming a read-only store is a better failure than an upload appearing from nowhere. What a step writes is readable through stepReadUpload/stepUploadInfo on the id it was given, like any other upload.