Type Alias AgentDef<S>

Built-in server-side tools enabled for this agent. Unset defaults to the cognitive set DEFAULT_BUILTIN_TOOLS (think, remember, recall, calculate); set explicitly — including [] — to override.

Pipeline mode only. Phrase spoken when the turn's LLM stream fails, so a provider outage hands the conversation back instead of going silent — a failed turn produces no text, so nothing would otherwise reach TTS. Defaults to DEFAULT_ERROR_PHRASE; set "" to disable.

Pipeline mode only. False-interruption recovery window (ms): when a barge-in aborts the agent's reply but no user turn commits within this window (STT noise, hallucinated partial), the agent resumes the interrupted reply. Defaults to DEFAULT_FALSE_INTERRUPTION_TIMEOUT_MS (2000); 0 disables recovery.

Sentence spoken when a session starts. Defaults to DEFAULT_GREETING; set "" to start silent.

Pipeline mode only. Phrase spoken when the model's first action in a turn is a tool call with no preceding speech, so the caller never hears dead air. Defaults to "One moment."; set "" to disable.

How long the session may go with no inbound audio before it is closed (ms). Measures silence, not call length — re-armed on every audio frame. Defaults to DEFAULT_IDLE_TIMEOUT_MS (300 000, 5 minutes); 0 or a non-finite value disables the timer entirely.

Pipeline mode only. Minimum sustained speech (ms since the utterance's first interim transcript) before an interim-triggered barge-in aborts the agent's reply — a duration gate alongside minBargeInWords, mirroring LiveKit's min_interruption_duration. Committed turns (STT finals) are never gated. Defaults to DEFAULT_INTERRUPTION_MIN_DURATION_MS (500); set 0 to disable the gate.

Pluggable LLM provider descriptor from @alexkroman1/aai/llm (e.g. anthropic({ model })) for pipeline mode. Unset (with no s2s), the stage defaults to the AssemblyAI LLM Gateway. Note this is pure serializable data, not a Vercel AI SDK LanguageModel instance — the host resolves the descriptor into a LanguageModel at session start, using credentials from the agent's env.

Max tool calls per reply — bounds runaway tool loops. Defaults to DEFAULT_MAX_STEPS (10).

Pipeline mode only. Minimum words in an interim transcript before user speech barges in on (aborts) the agent's in-flight reply. Defaults to DEFAULT_MIN_BARGE_IN_WORDS (2) so one-word backchannels ("yeah", "mm-hmm") don't cut the agent off; set 1 to interrupt on any word.

Display name shown by the default client UI.

Env var names this agent's tools read from ToolContext.env (beyond provider credentials, which are derived from the stt/llm/tts/s2s descriptors automatically). Deploys check that every listed name is present in the agent's stored env, so a missing key surfaces at deploy time instead of as a runtime failure on the first tool call.

Pluggable S2S provider descriptor — the explicit opt-in to speech-to-speech mode (e.g. assemblyAIS2s() for AssemblyAI's Voice Agent API, or openaiRealtime()). Unset, the agent runs the default cascaded pipeline. Mutually exclusive with the stt/llm/tts pipeline triple.

Instruction injected as a synthetic user turn when silenceTimeoutMs elapses. Never shown as a user transcript. Defaults to DEFAULT_SILENCE_PROMPT. Requires silenceTimeoutMs.

Pipeline mode only. When set, the assistant proactively takes a turn after this many ms of user silence (no speech since the last reply finished). Unset disables the behavior. Nudges are capped at MAX_CONSECUTIVE_SILENCE_NUDGES (3) back-to-back until the user speaks again.

Pipeline mode only. Phrase spoken when a provider fails to open, so a session that cannot start says so instead of holding an open line in silence. Only reachable when TTS itself came up — which is the usual case, since STT and TTS open independently. Defaults to DEFAULT_START_FAILURE_PHRASE; set "" to disable.

Factory creating this session's mutable state — the value tools read and write as ctx.state. Called once per session; unset leaves ctx.state an empty object.

Pluggable STT provider for pipeline mode. Unset (with no s2s), the stage defaults to AssemblyAI STT — each pipeline stage is individually optional, and unset stages are filled from the all-AssemblyAI pipeline (assemblyAIPipeline()).

Pipeline mode only. Bias prompt for the streaming STT — use it to teach the transcriber the agent's own vocabulary (product names, spelled-out identifiers). Defaults to empty (unbiased transcription); see DEFAULT_STT_PROMPT for what an effective prompt looks like.

Project per-session state to the browser client, so a custom UI can render it without the agent hand-rolling a sync channel.

A PROJECTION rather than a flag, for three reasons. ctx.state routinely holds things that should not reach a browser or cannot be serialized, so the author decides what leaves. Returning plain data makes serializability the author's call rather than a runtime surprise. And it doubles as the client's contract: whatever this returns is exactly what useAgentState receives.

Called after every tool call, and pushed only when the projection actually changed — most turns do not touch state, and this shares a socket with 384 kbps of PCM.

import { agent } from "@alexkroman1/aai";
type Item = { sku: string; qty: number };

agent({
  name: "Cart",
  state: () => ({ cart: [] as Item[], staffPin: "" }),
  syncState: (s) => ({ cart: s.cart }),   // staffPin stays server-side
});
Copy

Without it, the pattern agents reach for is: return a state snapshot from every tool, declare a result type describing it, and mirror it into useState via useToolResult. Measured across generated agents, 58% built some version of that by hand.

System prompt driving the LLM. Defaults to DEFAULT_SYSTEM_PROMPT when not set on agent().

How the LLM selects tools each step. Defaults to "auto" (DEFAULT_TOOL_CHOICE): the model decides when to call a tool. Honored in pipeline mode and by the OpenAI Realtime transport; the AssemblyAI S2S service runs the tool loop service-side and does not take a tool-choice parameter.

Custom tools the agent may invoke, keyed by tool name.

Pluggable TTS provider for pipeline mode. Unset (with no s2s), the stage defaults to AssemblyAI TTS (agent()'s voice shorthand picks its voice).