Voices and models

An agent hears, thinks and speaks in three stages: speech-to-text (STT), a language model (LLM), and text-to-speech (TTS). All three are AssemblyAI by default, billed to the one key in your .env. Come here when you want a different voice, a different model, or a different provider for any stage.

Each stage is a field on agent(), and anything you leave unset stays on the default — so swapping one thing is one line.

A voice

import { agent } from "@alexkroman1/aai";
import { assemblyAITts } from "@alexkroman1/aai/tts";

export default agent({ name: "My Agent", tts: assemblyAITts({ voice: "aai-michael" }) });

The ids are the voice_id slugs in ASSEMBLYAI_TTS_VOICES (@alexkroman1/aai/tts) — the aai- prefix is part of the id. Every library voice speaks all six supported languages; language picks which.

The type autocompletes but does not guard. The catalog belongs to the service, so a voice added after your SDK release still has to work.

A misspelled voice id is refused when the session opens

answers it with a 3012 error. So aai build and aai dev warn about an id they do not recognize, and name the ones it is closest to. :::

A model

llm takes a bare model id:

import { agent } from "@alexkroman1/aai";

export default agent({ name: "My Agent", llm: "claude-sonnet-4-6" });

A bare id routes through the AssemblyAI LLM gateway on your existing key. A "creator/model" id routes through the Vercel AI Gateway and needs AI_GATEWAY_API_KEY in your secrets.

Bare ids autocomplete from AssemblyAIGatewayModel (@alexkroman1/aai), the union generated from what the gateway advertises. Like a voice id, it is autocomplete rather than a guard.

A wrong model id is a gateway error on the first turn

catches it at build time the way a voice id is caught. An id the gateway does not carry comes back as a 400 the first time the agent tries to think, so the session opens and then fails on the caller’s first sentence. :::

A whole stage

Import the provider you want and pass it to the matching field:

import { agent } from "@alexkroman1/aai";
import { deepgramStt } from "@alexkroman1/aai/stt";
import { cartesiaTts } from "@alexkroman1/aai/tts";

export default agent({
  name: "My Agent",
  stt: deepgramStt(),
  tts: cartesiaTts(),
  // `llm` unset → still AssemblyAI
});

Each factory reads one key from the environment:

Factory	Import from	Key it reads
assemblyAIStt, assemblyAITts, llm({ provider: "assemblyai" }), assemblyAIS2s	/stt, /tts, /llm, /s2s	ASSEMBLYAI_API_KEY
deepgramStt	@alexkroman1/aai/stt	DEEPGRAM_API_KEY
elevenLabsStt	@alexkroman1/aai/stt	ELEVENLABS_API_KEY
sonioxStt	@alexkroman1/aai/stt	SONIOX_API_KEY
cartesiaTts	@alexkroman1/aai/tts	CARTESIA_API_KEY
rimeTts	@alexkroman1/aai/tts	RIME_API_KEY
llm({ provider: "anthropic" })	@alexkroman1/aai/llm	ANTHROPIC_API_KEY
llm({ provider: "openai" }), openAIS2s	/llm, /s2s	OPENAI_API_KEY
llm({ provider: "google" })	@alexkroman1/aai/llm	GOOGLE_GENERATIVE_AI_API_KEY
llm({ provider: "mistral" })	@alexkroman1/aai/llm	MISTRAL_API_KEY
llm({ provider: "xai" })	@alexkroman1/aai/llm	XAI_API_KEY
llm({ provider: "groq" })	@alexkroman1/aai/llm	GROQ_API_KEY
llm({ provider: "openrouter" })	@alexkroman1/aai/llm	OPENROUTER_API_KEY
llm({ provider: "cerebras" })	@alexkroman1/aai/llm	CEREBRAS_API_KEY
llm({ provider: "gateway" })	@alexkroman1/aai/llm	AI_GATEWAY_API_KEY

The LLM stage is one factory: llm({ provider, model }), where provider is a string. Any other provider name also compiles — give it a baseUrl (and an apiKeyEnv naming its key) and it is called as an OpenAI-compatible endpoint.

Put that key in .env locally, and in your agent’s secrets in production — see Publish. It is read on the server and never reaches the browser. Each factory’s options are in the SDK reference.

A backup provider

fallback([primary, secondary]) takes two or more providers for one stage and tries the next when one fails. It is on /stt, /llm and /tts:

import { agent } from "@alexkroman1/aai";
import { llm } from "@alexkroman1/aai/llm";
import { assemblyAIStt, deepgramStt, fallback } from "@alexkroman1/aai/stt";

export default agent({
  name: "My Agent",
  stt: fallback([assemblyAIStt(), deepgramStt()]),
  llm: fallback([
    llm({ provider: "assemblyai", model: "gpt-5.6-luna" }),
    llm({ provider: "anthropic", model: "claude-sonnet-5" }),
  ]),
});

It switches only before the stage has produced anything, so a caller never hears half an answer from one vendor and the rest from another:

• STT and TTS switch when the connection fails to open, or when the session reports an error before its first transcript (STT) or its first audio (TTS).
• The LLM switches when a request fails, or when its stream errors before the first word or tool call. It decides per request: the next turn tries the primary again.
• Nothing switches on an interruption or after output has started.

Each switch is a provider.failedOver session event naming the stage, both providers and the reason, so an events handler can count them. Every provider in the list needs its key: aai dev and Publish check all of them.

Your own provider

A provider this SDK does not ship is registered by the program that runs the agent — aai dev does not, but a self-hosted server.mjs can. Register the kind with registerSttKind, registerTtsKind or registerLlmKind from @alexkroman1/aai-runtime before the agent server starts, and the agent names it with a plain descriptor:

server.mjs

import { registerSttKind } from "@alexkroman1/aai-runtime";

registerSttKind("my-stt", {
  envVar: "MY_STT_API_KEY",
  open: ({ options }) => myStt(options), // an `SttOpener`
});

// agent.ts
// stt: { kind: "my-stt", options: { model: "fast" } },

The key is read from the agent’s environment like any other provider’s. The opener is written against SttOpener / TtsOpener (also on @alexkroman1/aai-runtime); registerLlmKind takes a function that builds a Vercel AI SDK model.

Speech-to-speech

Speech-to-speech (S2S) replaces all three stages with one socket: the transcription, the model loop and the voice all run inside a single service. It is an explicit opt-in, never something you reach by omission.

import { agent } from "@alexkroman1/aai";
import { openAIS2s } from "@alexkroman1/aai/s2s";

export default agent({ name: "My Agent", mode: "s2s", s2s: openAIS2s() });

assemblyAIS2s() is the other one, from the same subpath.

What you buy is one round trip instead of three hops. What you give up is the seams. Providers can no longer be mixed, the S2S descriptor owns its own voice, and the tuning groups below are implemented by the three-stage pipeline alone — so setting one on an S2S agent is a compile error naming the rule rather than a silent no-op.

Stay on the three-stage default unless response time is the specific problem you are trying to fix.

Tuning the conversation

Three groups on agent() — turnTaking, interruption and silence — decide how a pipeline agent handles pauses, interruptions and monologues. Reach for them once you have heard a specific problem, not before.

Field	What it decides	Default
turnTaking.maxSilenceMs	How long a caller may pause mid-sentence before the turn is force-ended. Raise it for callers who dictate addresses or confirmation numbers.	1500
interruption.minWords	How many words of caller speech interrupt the agent’s reply. The default interrupts on any word, so a one-word “Hello?” is heard; minDurationMs filters a short backchannel.	1
interruption.minDurationMs	How long that speech must be sustained before it counts as an interruption. 0 disables the gate.	500
silence.deadAirCoverMs	How long a turn may send nothing before the agent speaks a short filler, so a long tool chain does not sound like a dropped call. 0 disables.	2400
turnTaking.userTurnLimit	A cap on one caller turn, { maxWords, maxDurationMs } — either or both. A caller who never pauses never ends a turn; past the cap the transcriber ends it as a pause would, the agent answers what it heard, and the rest opens the next turn. Each cut is a userTurn.exceeded event.	unset (no cap)
turnTaking.detection	Who ends a caller turn. "manual" is push-to-talk: the client opens and closes each turn (usePushToTalk() in @alexkroman1/aai-ui), audio outside a held turn is never transcribed, everything said while held is answered as one turn on release, and pressing is what interrupts the agent.	"auto" (the transcriber ends a turn on a pause)

The rest, including the phrases spoken on a provider failure, are in the SDK reference.

Next

• Your agent — the other fields on agent()
• Publish — where a provider key goes in production