testing/vitest

@alexkroman1/aai-runtime/testing/vitest — the vitest-coupled testing door: everything that INSTALLS or RESTORES, and the eval suites.

A test file imports from two places: @alexkroman1/aai-runtime/testing for every fake and reader that installs nothing, and this subpath for the rest. One import serves a unit spec and an eval file alike:

every installer of @alexkroman1/aai/testing/vitest (installStubGateway, installStubStepFetch, installStubTranscribe, …), each of which arms a fake and restores it with onTestFinished;
everything @alexkroman1/aai-runtime/eval/vitest provides — the describeEval / describeTextEval / describeWorkflowEval suites, the session and the readers, the simulated caller and the judge, and the SDK stubs a case composes with.

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Look an order up",
  inputSchema: z.object({ orderId: z.string() }),
  execute: ({ orderId }) => ({ orderId, status: "shipped" }),
});

// file: agent.eval.test.ts
import agentDef from "virtual:aai/agent";
import { describeEval, expectCalled } from "@alexkroman1/aai-runtime/testing/vitest";
import lookUp from "./tools/look_up.ts";

describeEval(agentDef, (test) => {
  test(
    "looks the order up before answering",
    async ({ session }) => {
      // The claim takes the tool's DEF, matched by the name the agent registered.
      expectCalled(await session.say("where is order W1234?"), lookUp);
    },
    { stubReply: "Order W1234 shipped yesterday." },
  );
});

Every name is a re-export of the SAME declaration as its original subpath, so ownership does not move: the eval names keep their eval* capabilities, the SDK stubs eval-stubs, and the other installers testing-stubs. vitest is an OPTIONAL peer dependency, and importing this subpath is what pulls it; /testing and /eval stay importable without it.

Functions

describeAgent()

describeAgent<D>(agent, define): void

Declare a suite for one deployed agent: the built-in “is deployable” case, then the body, handed the typed ToolRunner for the agent.

Every shipped template’s spec opened with the same two things — a describe(agentDef.name, …) whose first case was expectDeployable(agentDef), and const run = toolRunner(agentDef) above it. The deployable case is the one no agent spec should be without (a config the platform refuses is a spec measuring nothing), and binding the runner is the line every file repeated.

The body takes vitest’s own test/expect from vitest as usual; only the runner is handed in, so a case reads run(addItem, { … }) typed by the def — a tool file’s default export, or a speaker’s tools entry — and gated as the agent gates it.

Type Parameters

D

D extends ToolBearingAgent & object

Parameters

agent

D

define

(run, agent) => void

Returns

void

Example

import { agent } from "@alexkroman1/aai";

export default agent({ name: "Orders" });

// file: tools/add_item.ts
import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Add an item to the order",
  inputSchema: z.object({ item: z.string() }),
  execute: ({ item }) => ({ added: item }),
});

// file: agent.test.ts
import { deployedAgent } from "@alexkroman1/aai/testing";
import { describeAgent } from "@alexkroman1/aai/testing/vitest";
import { expect, test } from "vitest";
import authored from "./agent.ts";
import addItem from "./tools/add_item.ts";

// Under vitest, `import agentDef from "virtual:aai/agent"` is this line.
const agentDef = deployedAgent(authored, {
  tools: import.meta.glob("./tools/*.ts", { eager: true }),
});

describeAgent(agentDef, (run) => {
  test("adds the item", async () => {
    const { added } = await run(addItem, { item: "apple" });
    expect(added).toBe("apple");
  });
});

installStubClientInbox()

installStubClientInbox(options?): StubClientInbox

Publish a device inbox for stepNotifyClient, restored when this test finishes.

stubClientInbox with the bookkeeping done — see it for the call log and how to make the device answer busy or offline. The usual partner of installStubSpeech: a step that speaks a notice and pushes it to a speaker needs both, and each used to cost a try/finally of its own.

Parameters

options?

StubClientInboxOptions

Returns

StubClientInbox

Example

In a test body or a beforeEach:

import { stepNotifyClient } from "@alexkroman1/aai/step";
import { installStubClientInbox } from "@alexkroman1/aai/testing/vitest";

const inbox = installStubClientInbox();
await stepNotifyClient("kitchen", { id: "wrun_1", event: "reminder" });
console.log(inbox.calls[0]?.clientId); // "kitchen"

installStubFfmpeg()

installStubFfmpeg(script?): StubFfmpeg

Route every ffmpeg-family spawn through script, restored when this test finishes.

The fake never times out and never honours the run’s abort signal: a child that hangs is not something a script can express, and the deadline machinery it would exercise is the SDK’s own, tested in ffmpeg.test.ts.

Parameters

script?

StubFfmpegScript

Returns

StubFfmpeg

Example

import { probeMedia, runFfmpeg } from "@alexkroman1/aai/ffmpeg";
import { installStubFfmpeg } from "@alexkroman1/aai-runtime/testing/vitest";
import { writeFile } from "node:fs/promises";
import { expect, test } from "vitest";

test("a scripted ffmpeg answers each call by its args", async () => {
  const ffmpeg = installStubFfmpeg(async ({ args }) => {
    if (args.includes("-show_streams")) {
      return { stdout: '{"streams":[{"codec_type":"audio"}],"format":{"duration":"1.0"}}' };
    }
    // A transcode's output is whatever the script writes at the out path.
    await writeFile(args.at(-1) ?? "", new Uint8Array(32_000));
    return {};
  });
  const info = await probeMedia("call.m4a");
  expect(info.durationSec).toBe(1);
  await runFfmpeg(["-i", "call.m4a", "out.wav"]);
  expect(ffmpeg.calls).toHaveLength(2);
});

installStubGateway()

installStubGateway(replies, options?): StubGatewayCall[]

Install a fake LLM gateway as the global fetch, and return its call log.

stubGateway (@alexkroman1/aai/testing) installed for you. Import it under THIS name — aliasing it to stubGateway shadows the uninstalled fake of that name. A step whose HTTP goes through a published stepFetch wants stubGatewayRoute instead; stubGateway’s doc has the table.

The calls array is what a spec asserts on, and it is live — a reference taken before the code under test runs holds every call made after.

Lifetime is one test, as it is for any vi.stubGlobal under unstubGlobals — which this repo’s shared config and defineAgentTestConfig (@alexkroman1/aai/testing/vite) both set, so the real fetch is back before the next test. Install per test (the test body or a beforeEach); one installed in beforeAll is gone before the first test runs. Without unstubGlobals, vi.unstubAllGlobals() is the explicit undo.

It also seeds ASSEMBLYAI_API_KEY for the test when the process holds none, since stepGenerate refuses by name without one and the fake never reads it — apiKey: false ensures there is NONE, whatever the shell exports. Either way the previous value is put back when the test finishes.

Parameters

replies

string | readonly string[]

Completion contents, in order; the last repeats — see stubGateway in @alexkroman1/aai/testing, which this installs.

options?

InstallStubGatewayOptions

Returns

StubGatewayCall[]

Example

import { stepGenerateJson } from "@alexkroman1/aai/step";
import { z } from "zod";

export const summarize = (article: string) =>
  stepGenerateJson(`One headline for: ${article}`, { schema: z.object({ headline: z.string() }) });

// file: digest.test.ts
import { installStubGateway } from "@alexkroman1/aai/testing/vitest";
import { expect, test } from "vitest";
import { summarize } from "./workflows/digest.ts";

test("summarize sends the article", async () => {
  const calls = installStubGateway('{"headline":"Otters use tools"}');
  await summarize("Otters use tools.");
  expect(calls[0]?.prompt).toContain("Otters use tools.");
});

installStubReporter()

installStubReporter(): StubReporter

Capture what a step narrates and emits, restored when this test finishes.

stubReporter with the bookkeeping done — see it for why stepReport() and stepEmit() are separated the way the streams are.

Returns

StubReporter

installStubWorkflows()

installStubWorkflows(options?): WorkflowClient

Parameters

options?

StubWorkflowsOptions

Returns

WorkflowClient

installStubWorld()

installStubWorld(options?): StubWorld

Install every seam a workflow step reaches through, composed in the one order that works, and seed the credential — restored when this test finishes.

A step’s HTTP goes through ONE published stepFetch, so a flow that calls a model AND fetches a page AND transcribes a recording cannot install three fakes: whichever published last answers everything. The composition every workflow spec therefore hand-wrote is the gateway route in FRONT of the page stub (model.route(request) ?? page()), and in front of both the transcription fake holding the slot with the pair as its otherwise. Get the order wrong and the page stub answers the gateway’s request with HTML, which stepGenerate retries against until its budget runs out; a shipped template carries a paragraph warning about exactly this. Here the order is fixed, and a spec names only the legs it has.

It also seeds ASSEMBLYAI_API_KEY (apiKey: false ensures there is none instead), because stepGenerate refuses by name without one and the fake never reads it; the previous value is put back when the test finishes. stubGatewayRoute itself cannot seed anything — it installs nothing and owns no lifetime — so the seeding lives with the installers.

installStubReporter is installed too, so what the steps narrated is on reported without a second call — unless a reporter is already published (an eval’s), which is left in place: reporter decides. A SECOND world in one test replaces the first’s reporter along with its other legs, so the world a spec holds last is the one whose reported records.

Parameters

options?

StubWorldOptions

Returns

StubWorld

Example

import { stepFetch, stepGenerateJson } from "@alexkroman1/aai/step";
import { z } from "zod";

export async function digest(input: { url: string }) {
  const page = await (await stepFetch(input.url)).text();
  return stepGenerateJson(`One headline for: ${page}`, {
    schema: z.object({ headline: z.string() }),
  });
}

// file: digest.test.ts
import { installStubWorld } from "@alexkroman1/aai/testing/vitest";
import { expect, test } from "vitest";
import { digest } from "./workflows/digest.ts";

test("the digest reads the page and asks the model", async () => {
  const world = installStubWorld({
    model: '{"headline":"Otters use tools"}',
    pages: { "https://example.com/otters": { body: "<p>Otters use tools.</p>" } },
  });
  await digest({ url: "https://example.com/otters" });
  expect(world.model[0]?.prompt).toContain("Otters");
});

Type Aliases

InstallStubGatewayOptions

InstallStubGatewayOptions = StubGatewayOptions & object

Input

Options for installStubGateway: stubGateway’s, plus the credential.

Type Declaration

apiKey?

optional apiKey?: string | false

The ASSEMBLYAI_API_KEY to seed for this test, when the process holds none — see installStubWorld. Defaults to a recognizable stub value; false ENSURES there is none for the test (a key the shell exports is stubbed away and put back afterwards), for the one spec whose subject IS the missing-key refusal.

StubFfmpeg

StubFfmpeg = object

Sealed

The installed fake.

Properties

calls

readonly calls: StubFfmpegCall[]

Every spawn so far, in order.

StubFfmpegCall

StubFfmpegCall = object

Sealed

One spawn, as the fake saw it.

Properties

args

readonly args: readonly string[]

The argv, exactly as the caller built it (standing flags included).

binary

readonly binary: string

The binary the SDK resolved — ffmpeg, ffprobe, or whatever the env named.

stdin

readonly stdin: Uint8Array | undefined

Bytes the run handed to stdin, for an argv that reads pipe:0.

StubFfmpegReply

StubFfmpegReply = object

Input

What the scripted child does with one spawn.

Every field is optional: an empty reply is a child that printed nothing and exited 0.

Properties

exitCode?

readonly optional exitCode?: number

Exit code. Defaults to 0; anything else is an exit failure.

spawnError?

readonly optional spawnError?: Error

Fail the SPAWN itself with this error instead of running — ENOENT, say.

stderr?

readonly optional stderr?: string

What the child wrote to stderr — loudnorm’s block, an error line.

stdout?

readonly optional stdout?: string | Uint8Array

What the child wrote to stdout — ffprobe’s JSON, or a pipe:1 output.

StubFfmpegScript

StubFfmpegScript = (call) => StubFfmpegReply | undefined | Promise<StubFfmpegReply | undefined>

Input

Called once per spawn, with the argv. May be async, and may write files.

Parameters

call

StubFfmpegCall

Returns

StubFfmpegReply | undefined | Promise<StubFfmpegReply | undefined>

StubWorkflowsOptions

StubWorkflowsOptions = object

Input

What installStubWorkflows answers each read with.

Every field has a default, so installStubWorkflows() is a client whose reads all answer “nothing has run” — which is the arm a *_status tool branches on first and the one most specs of one want.

Properties

agent?

optional agent?: WorkflowAppLike

The agent under test, whose declared workflows are what listing() reports — so a renamed workflow is read off the def rather than restated. Pass agentDef; an explicit names wins over it.

lastLine?

optional lastLine?: unknown

What lastLine resolves with. Defaults to undefined, which means “the run has written nothing yet” — the arm a progress tool branches on, and the reason this has a default rather than being left to reject.

names?

optional names?: readonly string[]

Workflow names listing() reports, in order — normally the one the agent under test declares. Defaults to the keys of StubWorkflowsOptions.agent’s workflows, else none, i.e. an agent declaring no workflow.

runId?

optional runId?: string

What start resolves with. Defaults to "wrun_stub".

runs?

optional runs?: readonly WorkflowRunSnapshot[]

The runs get, find and recent answer from — get with the first, the other two with the whole list. One list rather than three, because a spec asserting what a tool REPORTS is describing one world, and three fixtures that can disagree about it is a way to write a passing test for a state the platform cannot produce. Build them with createRunSnapshot.

StubWorld

StubWorld = object

Sealed

What installStubWorld returns: every fake’s call log, live.

Properties

files

readonly files: StubUploads | undefined

The upload store itself — undefined when uploads was not named. Read a file back with read; what a step wrote (under writable) is writes.

model

readonly model: StubGatewayCall[]

Every completion request the gateway answered, decoded.

provider

readonly provider: StubTranscribeCall[]

Every transcription request, by leg — empty when transcribe was not named.

reported

readonly reported: StubReporter

What the steps narrated and emitted (installStubReporter) — empty, and recording nothing, when the world left a published reporter in place (see StubWorldOptions.reporter).

speech

readonly speech: StubSpeechCall[]

Every line the synthesizer was asked to say — empty when speech was not named.

StubWorldOptions

StubWorldOptions = object

Input

What installStubWorld may be told: one entry per seam a step reaches through, each optional, each installed only when named.

Properties

apiKey?

readonly optional apiKey?: string | false

The ASSEMBLYAI_API_KEY to seed for this test, when the process holds none. Defaults to a recognizable stub value; false ensures there is none for the test, whatever the shell exports.

model?

readonly optional model?: string | readonly string[]

What the LLM gateway answers, in order, the last repeating — stubGatewayRoute’s replies. Routed FIRST, on the completions path, before every other leg: a page stub answering the gateway’s request with HTML is the footgun this ordering exists for (the summarize step retries six times against it). Absent, a completion request falls through to the other legs.

pages?

readonly optional pages?: FetchRouteTable | readonly FetchRouteHandler[] | FetchRouteHandler

Everything else a step fetches — a page, a vendor API — as a stubFetchRoutes table, an ordered handler list (what describeEval’s steps.routes takes too) or one handler. The LAST leg: a request it does not answer is a finding and throws naming the method and URL, never an empty 200 a step would try to parse — after a transcribe.otherwise of your own has declined it.

reporter?

readonly optional reporter?: boolean

Publish the recording reporter (installStubReporter). Defaults to publishing one only when NO reporter is published yet — inside describeEval the harness’s reporter is what run.reported reads, and a stub published over it emptied that log — or when the one published is an EARLIER world’s, which this one replaces like every other leg (its reported then records; the earlier world’s stops). false never publishes one; true always does.

speech?

readonly optional speech?: StubSpeechOptions | true

Publish a recording synthesizer — installStubSpeech’s options, or true for the defaults.

transcribe?

readonly optional transcribe?: StubTranscribeOptions

Answer AssemblyAI’s transcription endpoints — installStubTranscribe’s options. When present, the transcription fake OWNS the published step fetch and the gateway and pages legs are its otherwise, which is the only order that works: publishing replaces, so two fakes cannot both hold the slot. An otherwise of your own runs after both.

uploads?

readonly optional uploads?: StubUploadsOptions & object

Publish an in-memory upload store — installStubUploads’s two arguments as one object: the files it holds by id, beside its options (writable, idPrefix). {} publishes an empty read-only store; { writable: true } one a step may write into, read back on files.