eval/simulate

@alexkroman1/aai-runtime/eval/simulate — a simulated caller, and a model-graded judge.

simulateCall has a SECOND model play the caller — a persona and a goal — driving the same say()/send() a scripted case does until it calls end_call or maxTurns runs out. The result is ordinary EvalTurns plus the call’s metrics, so every reader on @alexkroman1/aai-runtime/eval takes it unchanged. judgeCall rules on each criterion over a call, a list of turns or a transcript, and computes the verdict from those rulings rather than asking for one. Deterministic readers stay the first instrument; a judge is for the claims only visible as meaning, and it is a noisy one — run it under AAI_EVAL_REPEAT and read the spread.

In a describeEval / describeTextEval case, evalSimulation builds the pair from the case’s own session and mode, live or scripted the way the rest of the suite is:

import type { AgentDef } from "@alexkroman1/aai";
import { evalSimulation } from "@alexkroman1/aai-runtime/eval/simulate";
import type { EvalTestContext } from "@alexkroman1/aai-runtime/eval/vitest";

declare const agentDef: AgentDef;

// The body of a `describeEval` case: `session` and `mode` come from its context.
export async function forecastCase({ session, mode }: EvalTestContext): Promise<boolean> {
  const { simulate, judge } = evalSimulation({ agent: agentDef, mode, target: session });
  const call = await simulate({ persona: "a commuter", goal: "the forecast" });
  return (await judge(call, ["It answered the question."])).pass;
}

Its own subpath and capability rather than names on /eval and fields on the case context: the harness a case runs in and the second and third models a simulation adds move for unrelated reasons, and one epoch for both would version neither honestly. Runner-free, like /eval.

Exports are enumerated explicitly (no export *) so the public surface is deliberate.

Functions

evalSimulation()

evalSimulation(settings): EvalSimulationContext

Build the simulate/judge pair for one case. Every stub it installs is released before the call that installed it returns, so a case owes nothing back.

Parameters

settings

EvalSimulationOptions

Returns

EvalSimulationContext

---

judgeCall()

judgeCall(input, options): Promise<CallVerdict>

Have a model rule on criteria over input, and hand back the verdict.

import { llm } from "@alexkroman1/aai/llm";
import { judgeCall, type SimulatedCall } from "@alexkroman1/aai-runtime/eval/simulate";

export async function grade(call: SimulatedCall): Promise<void> {
  const verdict = await judgeCall(call, {
    criteria: [
      "The agent looked the order up before saying whether it shipped.",
      "The agent never asked for a card number.",
    ],
    llm: llm({ provider: "anthropic", model: "claude-sonnet-5" }),
  });
  if (!verdict.pass) throw new Error(verdict.explain());
}

Parameters

input

JudgeInput

options

JudgeCallOptions

Returns

Promise<CallVerdict>

Throws

if criteria is empty — a judge with nothing to rule on passes vacuously, which is the silent green this module exists not to produce.

---

simulateCall()

simulateCall(target, options): Promise<SimulatedCall>

Run a simulated call against target and hand back every turn, the way it ended, and what was measured.

import { agent } from "@alexkroman1/aai";
import { llm } from "@alexkroman1/aai/llm";
import { openEvalSession } from "@alexkroman1/aai-runtime/eval";
import { simulateCall } from "@alexkroman1/aai-runtime/eval/simulate";

export async function hurriedCaller(): Promise<void> {
  const session = await openEvalSession({ agent: agent({ name: "Order Desk" }) });
  try {
    const call = await simulateCall(session, {
      caller: {
        persona: "a polite but hurried customer",
        goal: "find out whether order W1234 has shipped",
      },
      llm: llm({ provider: "anthropic", model: "claude-haiku-4-5" }),
    });
    if (call.endedBy !== "caller") throw new Error(call.transcript());
    console.log(call.metrics.toolCallCounts, call.metrics.latencyMs);
  } finally {
    await session.close();
  }
}

The target is used as is and left OPEN — whoever opened it closes it, the same ownership every other door here keeps.

Parameters

target

SimulationTarget

options

SimulateCallOptions

Returns

Promise<SimulatedCall>

Type Aliases

CallVerdict

CallVerdict = object

The judge’s verdict over a whole conversation.

Methods

explain()

explain(): string

The failed rulings, one per line — what a failure message should print.

Returns

string

Properties

criteria

readonly criteria: readonly CriterionVerdict[]

One ruling per criterion, in the order they were given.

pass

readonly pass: boolean

Every criterion passed.

scripted

readonly scripted: boolean

The rulings came from a script (stubJudge), not a model’s reading.

summary

readonly summary: string

The judge’s overall summary.

---

CriterionVerdict

CriterionVerdict = object

One criterion’s ruling.

Properties

criterion

readonly criterion: string

pass

readonly pass: boolean

reason

readonly reason: string

The judge’s reason, in a sentence or two.

---

EvalSimulationContext

EvalSimulationContext = object

What a case gets for running a simulated caller and grading the result.

Methods

judge()

judge(input, criteria, options?): Promise<CallVerdict>

Have a model rule on criteria over a simulated call, a list of turns, or a transcript. See judgeCall.

Parameters

input

JudgeInput

criteria

readonly string[]

options?

context?

string

Returns

Promise<CallVerdict>

simulate()

simulate(caller, options?): Promise<SimulatedCall>

Run a simulated caller against this case’s session (or text agent) until it hangs up or maxTurns runs out. See simulateCall.

Parameters

caller

SimulatedCaller

options?

maxTurns?

number

Returns

Promise<SimulatedCall>

---

EvalSimulationOptions

EvalSimulationOptions = object

What evalSimulation takes.

Properties

agent

readonly agent: AgentDef

The agent under evaluation. Live, the caller and the judge default to its model.

callerLlm?

readonly optional callerLlm?: LlmProvider

The model that PLAYS the caller when live. Defaults to the agent’s model.

env?

readonly optional env?: Record<string, string>

The agent env, for live credentials. Defaults to none.

judgeLlm?

readonly optional judgeLlm?: LlmProvider

The model that JUDGES when live. Defaults to the agent’s model.

llm?

readonly optional llm?: LlmProvider

The model the AGENT was evaluated on, when the suite overrode it (its llm option) — the live default for both of the above.

mode

readonly mode: EvalMode

Which model this run got — the case context’s mode. "stub" scripts the caller and the judge too; a keyless simulation checks wiring, not behaviour.

providerEnv?

readonly optional providerEnv?: ProviderEnv

Provider credentials when live. Defaults to env plus the host’s own.

stubCaller?

readonly optional stubCaller?: StubScript

The simulated caller’s lines in a keyless run, one per caller turn. End it with { tool: "end_call", args: { reason } }; absent, the stub caller says one line and hangs up.

stubJudge?

readonly optional stubJudge?: readonly boolean[]

The rulings a keyless judge hands back, one per criterion in order — missing entries pass. Absent, every criterion passes, marked scripted.

target

readonly target: SimulationTarget

What the simulated caller talks to — the case’s session, or its text agent.

---

JudgeCallOptions

JudgeCallOptions = object

What judgeCall takes.

Properties

context?

readonly optional context?: string

Extra context the judge should know — the agent’s purpose, a policy.

criteria

readonly criteria: readonly string[]

What must be true of the conversation, one claim each — “the agent confirmed the order number before cancelling”. Phrase each so it can be ruled on from the transcript alone.

llm

readonly llm: LlmProvider

The JUDGING model. Any @alexkroman1/aai/llm descriptor.

providerEnv?

readonly optional providerEnv?: ProviderEnv

Where the judge’s credential is resolved from. Defaults to this machine’s.

---

JudgeInput

JudgeInput = SimulatedCall | readonly EvalTurn[] | string

What a judge may be handed: a simulated call, a list of turns, or a transcript.

---

SimulateCallOptions

SimulateCallOptions = object

What simulateCall takes.

Properties

caller

readonly caller: SimulatedCaller

Who is calling.

llm

readonly llm: LlmProvider

The model PLAYING the caller. Any @alexkroman1/aai/llm descriptor — including one from installStubLlm, which is how a keyless run scripts the caller’s lines ({ tool: "end_call", args: { reason } } ends it).

maxTurns?

readonly optional maxTurns?: number

The most caller turns before the harness hangs up for them. Default DEFAULT_MAX_TURNS.

providerEnv?

readonly optional providerEnv?: ProviderEnv

Where the caller model’s credential is resolved from. Defaults to this machine’s environment, the same trust decision openEvalSession makes.

---

SimulatedCall

SimulatedCall = object

A finished simulated call.

Methods

transcript()

transcript(): string

The call as Agent:/Caller: lines — what a judge or a failure message reads.

Returns

string

Properties

caller

readonly caller: SimulatedCaller

endedBy

readonly endedBy: "caller" | "max-turns"

"caller" — it called end_call. "max-turns" — the harness hung up after SimulateCallOptions.maxTurns, which usually means the goal was never met.

endReason

readonly endReason: string | undefined

The reason the caller gave to end_call, when it gave one.

greeting

readonly greeting: readonly string[]

The agent’s opening line(s) before the caller spoke — empty for a text agent.

metrics

readonly metrics: SimulationMetrics

turns

readonly turns: readonly SimulatedTurn[]

---

SimulatedCaller

SimulatedCaller = object

Who the simulated caller is, and what they called for.

Properties

goal

readonly goal: string

What they want out of the call, stated as the CALLER would know it — including the facts they hold (“order W1234”, “a table for four on Friday”). The simulating model is told to reveal them only when asked, as a caller would.

opening?

readonly optional opening?: string

The caller’s first line. Absent, the model writes one — after the greeting, when the target has one.

persona

readonly persona: string

Who they are and how they talk — “a hurried commuter who answers in fragments”, “an elderly caller who asks for things to be repeated”.

---

SimulatedTurn

SimulatedTurn = object

One exchange: what the caller said and the turn it produced.

Properties

caller

readonly caller: string

The caller’s line.

latencyMs

readonly latencyMs: number | undefined

Milliseconds from the committed utterance to the first reply text — undefined for a turn that produced no text.

turn

readonly turn: EvalTurn

The agent’s turn in reply, exactly as say()/send() returned it.

---

SimulationMetrics

SimulationMetrics = object

What was measured over the whole call.

Properties

durationMs

readonly durationMs: number

Wall-clock time of the whole simulation, caller model included.

latencyMs

readonly latencyMs: object

Reply latency over the turns that produced text.

max

readonly max: number | undefined

mean

readonly mean: number | undefined

p50

readonly p50: number | undefined

toolCallCounts

readonly toolCallCounts: Readonly<Record<string, number>>

Tool calls per tool name.

toolCalls

readonly toolCalls: readonly EvalToolCall[]

Every tool call the agent made, in order.

turns

readonly turns: number

Caller turns taken.

---

SimulationTarget

SimulationTarget = { said: readonly string[]; say: Promise<EvalTurn>; } | { said: readonly string[]; send: Promise<EvalTurn>; }

What a simulation drives: an EvalSession (say) or an EvalTextAgent (send). Structural, so either handle passes as is.

Variables

DEFAULT_MAX_TURNS

const DEFAULT_MAX_TURNS: 12 = 12

How many caller turns a simulation may take unless told otherwise.

---

END_CALL_TOOL

const END_CALL_TOOL: "end_call" = "end_call"

The name of the caller-side hang-up tool.