stt

@alexkroman1/aai/stt subpath barrel — the speech-to-text stage of a pipeline agent.

Four vendors, one shape: each factory returns a serializable DESCRIPTOR ({ kind, options }), and you hand it to agent({ stt }). Nothing here opens a socket or reads a credential — the host resolves the descriptor at session start, so importing this barrel pulls in no vendor SDK.

Example

Swap the STT stage of an otherwise default agent

import { agent } from "@alexkroman1/aai";
import { deepgramStt } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  // `llm` and `tts` keep their AssemblyAI defaults.
  stt: deepgramStt({ model: "nova-3", language: "en" }),
});

Credentials are never passed here. Each factory’s vendor names the env var its key is read from — ASSEMBLYAI_API_KEY, DEEPGRAM_API_KEY, ELEVENLABS_API_KEY, SONIOX_API_KEY — and the host reads it out of the agent’s own environment when the session starts. That is what keeps a descriptor safe to serialize across the CLI → server → guest boundary. The variable NAMES are not published: an author never types one, and the one case for repointing a stage is apiKeyEnv on the AssemblyAI descriptor.

What an unset language means, per vendor

The field is spelled language for a single code and languages for a list, which is the only difference between the four — but their DEFAULTS do not line up, and that is the one cross-provider fact you cannot assemble from the per-symbol docs:

factory	field	unset means
assemblyAIStt	`languages`	English on Universal-3.5 Pro (the default model) — `["en"]` is sent for you; `[]` code-switches across 18
deepgramStt	`language`	English — `"en"` is sent for you
elevenLabsStt	`language`	auto-detect (the field is omitted)
sonioxStt	`languages`	auto-detect (the field is omitted)

So moving an agent from elevenLabsStt or sonioxStt to assemblyAIStt or deepgramStt silently drops multilingual transcription, and moving the other way silently gains auto-detection — read AssemblyAISttOptions.languages before you do, because code-switching on a monolingual line has a measured failure mode with no obvious symptom.

Each vendor’s own wire spelling (language_codes, languageCode, language_hints) is applied by the host opener, not written here.

The descriptor type is on the ROOT barrel TOO

SttProvider — what a factory here returns — is also exported from @alexkroman1/aai, beside the other three stage types, so an agent annotating two stages writes one import rather than two. It stays here as well: this is where the factory that produces one lives. ProviderDescriptor, the base all four narrow, is on the root ALONE now — one interface with four reference pages was three too many.

The host-side opener contract is on `/runtime`

Implementing an STT vendor of your own — SttOpenOptions, SttSession, SttEvents, SttError, SttTurnMeta, Unsubscribe — is a HOST job, and those types live on @alexkroman1/aai-runtime beside registerSttKind, which is what you hand the opener to.

Functions

assemblyAIStt()

assemblyAIStt(options?): SttProvider

Build an AssemblyAI STT descriptor.

The API key is resolved host-side from the agent’s env (ASSEMBLYAI_API_KEY); there is no factory-time key parameter, so the descriptor stays free of secrets and safe to serialize.

Named assemblyAIStt (not assemblyAI) so the STT, LLM (llm({ provider: "assemblyai" })), and TTS (assemblyAITts) factories can be imported side by side without aliasing.

Parameters

options?

AssemblyAISttOptions

Returns

SttProvider

Example

import { agent } from "@alexkroman1/aai";
import { assemblyAIStt } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: assemblyAIStt({ languages: ["en"] }),
});

Unset languages means English only on Universal-3.5 Pro (code-switching OFF; other models get no default); pass [] to code-switch across every supported language, or several codes to bias toward a subset — see AssemblyAISttOptions.languages.

deepgramStt()

deepgramStt(options?): SttProvider

Build a Deepgram STT descriptor.

The API key is resolved host-side from the agent’s env (DEEPGRAM_API_KEY); there is no factory-time key parameter, so the descriptor stays free of secrets and safe to serialize.

Parameters

options?

DeepgramSttOptions

Returns

SttProvider

Example

import { agent } from "@alexkroman1/aai";
import { deepgramStt } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: deepgramStt({ model: "nova-3", language: "en" }),
});

Deepgram’s unset language is not auto-detect: "en" is sent for you. Name the code you mean.

elevenLabsStt()

elevenLabsStt(options?): SttProvider

Build an ElevenLabs Scribe STT descriptor.

The API key is resolved host-side from the agent’s env (ELEVENLABS_API_KEY); there is no factory-time key parameter, so the descriptor stays free of secrets and safe to serialize.

Parameters

options?

ElevenLabsSttOptions

Returns

SttProvider

Example

import { agent } from "@alexkroman1/aai";
import { elevenLabsStt } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: elevenLabsStt({ model: "scribe_v2_realtime", language: "en" }),
});

Unset, language is omitted from the request and Scribe auto-detects — which is not the same as English.

fallback()

Call Signature

fallback(providers): SttProvider

Try providers in order, failing over to the next when one cannot serve.

import { agent } from "@alexkroman1/aai";
import { llm } from "@alexkroman1/aai/llm";
import { assemblyAIStt, deepgramStt, fallback } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: fallback([assemblyAIStt(), deepgramStt()]),
  llm: fallback([
    llm({ provider: "assemblyai", model: "gpt-5.6-luna" }),
    llm({ provider: "anthropic", model: "claude-sonnet-5" }),
  ]),
});

When it fails over — only before the stage has produced anything, so a caller never hears half an answer from one vendor and the rest from another:

STT / TTS: the connection fails to open (connect, auth, a refused handshake), or the open session reports an error before its first output (a transcript for STT, audio for TTS). Text sent to a TTS member that failed before speaking is replayed into the next one.
LLM: a request throws (network, any HTTP error status), or its stream errors before the first content part. Once text or a tool call has streamed, an error is the turn’s, as it is without a fallback.
Never on an abort (a barge-in, a session ending).

The failed member is skipped for the rest of that session (STT/TTS) or that request (LLM); the next request tries the primary again. When every member fails, the LAST error is the stage’s, exactly as a lone provider’s would be.

Every member’s key is required. The credential preflight (aai dev, deploy) demands each one, because a fallback whose secondary has no key fails at the moment it is needed.

The list is typed as two or more: a fallback of one member is not a fallback. Members are flattened, so a fallback inside a fallback is one list. Each member keeps its own options; the descriptors must all be the same stage.

Parameters

providers

readonly [SttProvider, SttProvider, SttProvider]

Returns

SttProvider

Call Signature

fallback(providers): LlmProvider

Try providers in order, failing over to the next when one cannot serve.

import { agent } from "@alexkroman1/aai";
import { llm } from "@alexkroman1/aai/llm";
import { assemblyAIStt, deepgramStt, fallback } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: fallback([assemblyAIStt(), deepgramStt()]),
  llm: fallback([
    llm({ provider: "assemblyai", model: "gpt-5.6-luna" }),
    llm({ provider: "anthropic", model: "claude-sonnet-5" }),
  ]),
});

When it fails over — only before the stage has produced anything, so a caller never hears half an answer from one vendor and the rest from another:

STT / TTS: the connection fails to open (connect, auth, a refused handshake), or the open session reports an error before its first output (a transcript for STT, audio for TTS). Text sent to a TTS member that failed before speaking is replayed into the next one.
LLM: a request throws (network, any HTTP error status), or its stream errors before the first content part. Once text or a tool call has streamed, an error is the turn’s, as it is without a fallback.
Never on an abort (a barge-in, a session ending).

Every member’s key is required. The credential preflight (aai dev, deploy) demands each one, because a fallback whose secondary has no key fails at the moment it is needed.

Parameters

providers

readonly [LlmProvider, LlmProvider, LlmProvider]

Returns

LlmProvider

Call Signature

fallback(providers): TtsProvider

Try providers in order, failing over to the next when one cannot serve.

import { agent } from "@alexkroman1/aai";
import { llm } from "@alexkroman1/aai/llm";
import { assemblyAIStt, deepgramStt, fallback } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: fallback([assemblyAIStt(), deepgramStt()]),
  llm: fallback([
    llm({ provider: "assemblyai", model: "gpt-5.6-luna" }),
    llm({ provider: "anthropic", model: "claude-sonnet-5" }),
  ]),
});

When it fails over — only before the stage has produced anything, so a caller never hears half an answer from one vendor and the rest from another:

STT / TTS: the connection fails to open (connect, auth, a refused handshake), or the open session reports an error before its first output (a transcript for STT, audio for TTS). Text sent to a TTS member that failed before speaking is replayed into the next one.
LLM: a request throws (network, any HTTP error status), or its stream errors before the first content part. Once text or a tool call has streamed, an error is the turn’s, as it is without a fallback.
Never on an abort (a barge-in, a session ending).

Every member’s key is required. The credential preflight (aai dev, deploy) demands each one, because a fallback whose secondary has no key fails at the moment it is needed.

Parameters

providers

readonly [TtsProvider, TtsProvider, TtsProvider]

Returns

TtsProvider

sonioxStt()

sonioxStt(options?): SttProvider

Build a Soniox STT descriptor.

The API key is resolved host-side from the agent’s env (SONIOX_API_KEY); there is no factory-time key parameter, so the descriptor stays free of secrets and safe to serialize.

Parameters

options?

SonioxSttOptions

Returns

SttProvider

Example

import { agent } from "@alexkroman1/aai";
import { sonioxStt } from "@alexkroman1/aai/stt";

export default agent({
  name: "Support",
  stt: sonioxStt({ model: "stt-rt-v3", languages: ["en", "es"] }),
});

Unset, languages is omitted from the request and Soniox auto-detects — which is not the same as English.

Interfaces

AssemblyAISttOptions

Input

Options for assemblyAIStt.

Extends

ProviderCredentialOptions

Properties

apiKeyEnv?

optional apiKeyEnv?: string

Env var holding this stage’s credential, replacing the provider default. Names a VARIABLE, not a key.

Inherited from

ProviderCredentialOptions.apiKeyEnv

connectTimeoutMs?

optional connectTimeoutMs?: number

Deadline for one streaming connect attempt — socket open and the server’s Begin message. Defaults to STT_CONNECT_TIMEOUT_MS (2500 ms), overriding the SDK’s own 1000 ms, which a healthy handshake can exceed. 0 waits indefinitely.

formatTurns?

optional formatTurns?: boolean

Whether the service applies punctuation, casing and inverse text normalization to a committed turn — “my number is nine seven two” becomes “My number is 972…” — sent as the format_turns connection parameter.

Not a parameter on universal-3-5-pro, where formatting is ALWAYS ON; setting it there is not sent, and the opener says so at warn level rather than letting an author believe they turned formatting off. On universal-streaming-english the service default is false, so that model’s transcripts are lowercase, unpunctuated and spelled-out until this is set.

It is a single explicit flag because the thing it changes is the tool-call ARGUMENT the model emits from a dictated identifier, and that is worth A/B-ing rather than inheriting. Note what true costs mechanically: that model then emits TWO end_of_turn messages per turn — the unformatted one first, the formatted one right after — so the opener demotes the unformatted one to a PARTIAL and commits only the formatted text. Without that the agent answers the same sentence twice, the second time on a history already containing its own reply; isCommittingTurn in aai-runtime’s providers/stt/_assemblyai-turn.ts is where it lives, and it is load-bearing rather than tidy-up.

languages?

optional languages?: readonly string[]

Languages to bias the model toward, sent as the language_codes connection parameter (e.g. ["en"], ["en", "es"]).

Unset means ENGLISH on Universal-3.5 Pro (the default model; ASSEMBLYAI_STT_DEFAULT_LANGUAGES, ["en"]) — the same default deepgramStt has, and the opposite of elevenLabsStt and sonioxStt, which auto-detect when unset. On any other model unset sends no language_codes and the model’s own language behaviour applies.

unset — Universal-3.5 Pro: ["en"], English only, code-switching off. Other models: nothing sent.
one code (["es"]) — pins that language, code-switching off.
several codes (["en", "es"]) — biases toward that subset and keeps code-switching between them. Sent exactly as given.
[] — sends no language_codes at all, so Universal-3.5 Pro code-switches across all 18 languages it supports. This is the opt-in for a genuinely multilingual line.

Code-switching is not the default because it costs accuracy on a monolingual line in a way that is easy to misread as an audio problem: measured against tau2-bench, English utterances came back transliterated into Devanagari and Hebrew script (Hello? Any update? → हेलो एनी अपडेट), and caller backchannels (mm-hm) came back as نن. (21x) and और (16x) — non-English text the pipeline took as real caller turns, cutting the agent off mid-reply. Nothing in the transcript says “wrong language” — it reads as a mis-hearing.

maxConnectRetries?

optional maxConnectRetries?: number

Extra connect attempts after a transient failure (timeout, network drop, unexpected close); permanent failures such as auth are never retried. Defaults to STT_CONNECT_MAX_RETRIES (2). 0 disables retries.

Raising either knob widens the worst-case open time ((1 + retries) * connectTimeoutMs plus the retry delays), which has to stay under DEFAULT_SESSION_START_TIMEOUT_MS — see the connect-budget note in sdk/constants.ts.

maxTurnSilenceMs?

optional maxTurnSilenceMs?: number

Maximum silence (ms) before the service force-ends a turn regardless of content, sent as the max_turn_silence connection parameter. This is the pause-tolerance knob: it bounds only utterances that never read as complete, so raising it costs an ordinary finished sentence nothing. Defaults to DEFAULT_MAX_TURN_SILENCE_MS (3500); the service’s own default is 1536. Raise it for callers who dictate confirmation numbers or addresses, and keep it above minTurnSilenceMs.

minTurnSilenceMs?

optional minTurnSilenceMs?: number

Silence (ms) before the service runs its end-of-turn check, sent as the min_turn_silence connection parameter. At this point the model asks whether the turn reads as COMPLETE — if it does the turn ends, if not a partial is emitted and the turn stays open. So this is the latency floor on utterances that really did finish. Defaults to DEFAULT_MIN_TURN_SILENCE_MS (800), which relies on continuation (interruption.continuationWindowMs) to re-join a turn split mid-entity — set 1600 here if you turn continuation off.

To tolerate longer mid-utterance pauses, raise maxTurnSilenceMs instead — and never above it. This is a minimum and that is a maximum, so a value above the ceiling means the check can never fire before the content-blind force-end closes the turn, which is the split this knob is usually reached for in order to prevent.

model?

optional model?: string & object | "universal-3-5-pro"

Streaming speech model. Defaults to "universal-3-5-pro" (Universal-3.5 Pro Real-Time). Arbitrary strings are forwarded to the SDK unchanged.

region?

optional region?: "us" | "eu"

EU data-residency — routes both streaming and sync transcription to AssemblyAI’s EU endpoints (streaming.eu.assemblyai.com / sync.eu.assemblyai.com). Required for EU-region API keys, which the US endpoints reject. Defaults to "us".

streamingUrl?

optional streamingUrl?: string

Streaming WebSocket endpoint override, sent as the SDK’s websocketBaseUrl. Must include the versioned path (e.g. wss://streaming.sandbox000.assemblyai-labs.com/v3/ws) — the SDK only supplies that path for its own default host, so a bare origin connects to the wrong route.

Takes precedence over AssemblyAISttOptions.region: an explicit endpoint is a deliberate choice and must not be silently overwritten by the residency shorthand. Intended for pre-release/staging clusters and A/B measurement against the default host; leave unset in production.

vadThreshold?

optional vadThreshold?: number

Confidence (0-1) the service’s voice-activity detector needs before a frame counts as speech, sent as the vad_threshold connection parameter. Defaults to DEFAULT_VAD_THRESHOLD (0.02), well under the service’s own 0.2.

Lower it when a caller’s speech is going UNHEARD — no speech.started, no transcript, not even a wrong one — rather than mis-heard. The gate sits in front of the model, so a voice that scores just under it never reaches the model at all. Measured on tau2-bench retail: one caller voice, after 8 kHz mu-law telephony encoding, produced zero service events at the default; replayed across that voice’s calls, caller transcripts went 12 -> 19 of 30 lines at 0.01 (0.02: 18) while background speech leaking into the transcript stayed at 2-3 throughout.

Not a substitute for voiceFocusThreshold: background SPEECH passes any VAD, because it is speech. And a lower gate means the service reports speech more readily, which is what opens a barge-in — check interruptions on a live run before shipping a low value.

voiceFocus?

optional voiceFocus?: string & object | "off" | "near-field" | "far-field"

Voice focus (voice isolation) mode, sent as the voice_focus connection parameter. Defaults to "near-field" to suppress background noise for close-mic / phone audio. Set to "" (or "off") to disable.

voiceFocusThreshold?

optional voiceFocusThreshold?: number

How aggressively Voice Focus suppresses background audio, sent as the voice_focus_threshold connection parameter (0-1, higher is more aggressive). Defaults to DEFAULT_VOICE_FOCUS_THRESHOLD (0.9), above the service’s own 0.7.

Raise it when BACKGROUND SPEECH — a television, a radio, another conversation — is reaching the transcript. That is the case the default is tuned for, and the case no VAD setting can fix: those frames really are speech, so a frame gate cannot distinguish them from the caller (see the constant’s doc for the measurement). Lower it if the caller’s own quiet or distant speech is being suppressed.

Ignored when voiceFocus is off — it tunes that filter.

DeepgramSttOptions

Input

Options for deepgramStt.

Extends

ProviderCredentialOptions

Properties

apiKeyEnv?

optional apiKeyEnv?: string

Env var holding this stage’s credential, replacing the provider default. Names a VARIABLE, not a key.

Inherited from

ProviderCredentialOptions.apiKeyEnv

endpointing?

optional endpointing?: number

Deepgram endpointing window (ms of trailing silence before a final is emitted). Defaults to DEEPGRAM_DEFAULT_ENDPOINTING_MS. Endpointing is the provider’s job — the pipeline transport commits a turn on every final — so this window is what keeps a mid-utterance pause from splitting one request across turns.

language?

optional language?: string

BCP-47 language code for transcription. Examples: "en", "es", "fr", "de".

Unset means ENGLISH, not auto-detect. DEEPGRAM_DEFAULT_LANGUAGE ("en") is filled in and sent on every connection — the same default assemblyAIStt has — where elevenLabsStt/sonioxStt omit the field entirely so the vendor auto-detects. So an agent moved from either of those two to deepgramStt() silently loses non-English transcription — and the symptom is a caller whose speech comes back as plausible English words, which reads as a mis-hearing rather than as a language setting.

Name the code you mean. There is no value for “detect”: Deepgram’s multilingual support is selected by naming a multilingual model.

model?

optional model?: string & object | "nova-3" | "nova-2"

Streaming speech model. Defaults to "nova-3". Any string is forwarded to the SDK unchanged, which allows opt-in to future models.

ElevenLabsSttOptions

Input

Options for elevenLabsStt.

Extends

ProviderCredentialOptions

Properties

apiKeyEnv?

optional apiKeyEnv?: string

Env var holding this stage’s credential, replacing the provider default. Names a VARIABLE, not a key.

Inherited from

ProviderCredentialOptions.apiKeyEnv

language?

optional language?: string

BCP-47 language code hint. Passing one reduces ambiguity for short utterances.

Unset means AUTO-DETECT, not English. The field is omitted from the request entirely, so ElevenLabs decides — which is the same default sonioxStt has, and the opposite of deepgramStt and assemblyAIStt, whose unset value is English. Pass a code for a line you know is monolingual.

model?

optional model?: string

Streaming speech model. Defaults to "scribe_v2_realtime". Any string is forwarded to the SDK unchanged so users can opt in to future models without an SDK release.

SonioxSttOptions

Input

Options for sonioxStt.

Extends

ProviderCredentialOptions

Properties

apiKeyEnv?

optional apiKeyEnv?: string

Env var holding this stage’s credential, replacing the provider default. Names a VARIABLE, not a key.

Inherited from

ProviderCredentialOptions.apiKeyEnv

languages?

optional languages?: readonly string[]

Language codes (ISO 639-1) that bias decoding toward the expected languages, sent as Soniox’s language_hints. Example: ["en", "es"].

Unset means AUTO-DETECT, not English. The field is omitted from the request entirely, so Soniox decides — which is the same default elevenLabsStt has, and the opposite of deepgramStt and assemblyAIStt, whose unset value is English. Pass the codes for a line you know is monolingual, or the handful you expect on one that is not.

model?

optional model?: string

Streaming model. Defaults to "stt-rt-v3". Any string is forwarded verbatim so users can opt in to future models.

Variables

ASSEMBLYAI_STT_EU_URL

const ASSEMBLYAI_STT_EU_URL: string

EU data-residency streaming endpoint.

References

ProviderCredentialOptions

Re-exports ProviderCredentialOptions

SttProvider

Re-exports SttProvider

stt

Example

What an unset language means, per vendor

The descriptor type is on the ROOT barrel TOO

The host-side opener contract is on /runtime

Functions

assemblyAIStt()

Parameters

options?

Returns

Example

deepgramStt()

Parameters

options?

Returns

Example

elevenLabsStt()

Parameters

options?

Returns

Example

fallback()

Call Signature

Parameters

providers

Returns

Call Signature

Parameters

providers

Returns

Call Signature

Parameters

providers

Returns

sonioxStt()

Parameters

options?

Returns

Example

Interfaces

AssemblyAISttOptions

Extends

Properties

apiKeyEnv?

Inherited from

connectTimeoutMs?

formatTurns?

languages?

maxConnectRetries?

maxTurnSilenceMs?

minTurnSilenceMs?

model?

region?

streamingUrl?

vadThreshold?

voiceFocus?

voiceFocusThreshold?

DeepgramSttOptions

Extends

Properties

apiKeyEnv?

Inherited from

endpointing?

language?

model?

ElevenLabsSttOptions

Extends

Properties

apiKeyEnv?

Inherited from

language?

model?

SonioxSttOptions

Extends

Properties

apiKeyEnv?

Inherited from

languages?

model?

Variables

The host-side opener contract is on `/runtime`