step-files
@alexkroman1/aai/step-files — moving bytes between the upload store and a local FILE.
A FACADE. The subpath resolves here rather than at step-files.ts, which buys two
things the direct form could not. That module can be SPLIT as it grows without
moving the published entry point — the path an implementation file happens to
have is not a thing to promise anyone — and a name it gains next reaches the
public surface only when a line is added below, rather than the moment it is
written.
Named re-exports rather than export * for the second half of that: the
wildcard form re-exports whatever arrives, and needs a noReExportAll
suppression the escape-hatch ratchet only lets move down.
Functions
Section titled “Functions”readUploadToFile()
Section titled “readUploadToFile()”readUploadToFile(
uploadId,path,options?):Promise<number>
Write an upload to a local path, a window at a time, and answer with the byte count that landed.
The windows are read CONCURRENTLY, because the cost here is the REMOTE read
and not the local write. This was a for loop, argued as “the bytes land
in one file at one offset each, so concurrency buys nothing and costs exactly
the memory the windows are here to bound” — which is true of the WRITE and
says nothing about the read. On a deployed guest every window is a brokered
302 + Range GET against object storage
(aai-runtime/uploads/blobs-brokered.ts), measured at 1.9-4.3 MB/s per
request under UPLOAD_PART_BYTES, so window N+1’s request did not start until
window N’s bytes had fully landed and the whole leg was latency-bound. The
other half of the same round trip has always fanned out — putWindows
(aai-runtime/uploads/store-blobs.ts) runs UPLOAD_WINDOW_CONCURRENCY wide —
so a step normalizing a recording pulled it in one window at a time and pushed
it back out four at a time. See STEP_FILE_READ_CONCURRENCY, which is
that same 4 and the same 32 MiB held.
The walk advances by what was READ, not by the window it asked for, and that
contract only holds IN ORDER. A stepReadUpload window is clamped to the bytes
that have ARRIVED, so on a STREAMED upload — or on any stale size — a short
answer means the file ends there, and the returned count is how the caller
learns it. A fan-out cannot preserve that by itself: window 5 may land in full
while window 2 comes back short, and writing 5 leaves a HOLE the returned count
claims is not there — silent truncation, which is the failure this whole module
keeps being rewritten to refuse. So the two paths are cut on exactly that line:
- No
size—stepRequireCompleteUploadhas established the file is whole, so every window but the last is full BY CONSTRUCTION and landing order cannot change the result. This path fans out. - A
size— the caller is judging completeness itself, which is what makes a polling body expressible (see ReadUploadToFileOptions.size). This path stays serial, so it can stop at the first window the store came back short on rather than discovering it four windows later.
The concurrent path is still short-safe, because a store may answer short
for reasons of its own. What it returns is the length of the CONTIGUOUS PREFIX
that landed — never the sum of the bytes it wrote — and it TRUNCATES the file to
that prefix, so a hole is neither reported as present nor left on disk in front
of bytes the count denies. That makes the two paths produce the same file, which
is what step-files.test.ts asserts rather than assumes.
With no size, an upload that is still arriving is REFUSED. That count was
documented as how a caller learns the store came back short, and against a
defaulted size it could never say so: the default was stepUploadInfo(id).size,
the contiguous readable PREFIX, so the walk copied the prefix and returned a
number equal to it. What reached ffmpeg was a truncated recording with nothing
anywhere reporting it. See sdk/step-uploads-complete.ts.
Parameters
Section titled “Parameters”uploadId
Section titled “uploadId”string
The id a run input carried.
string
Where to write. Created, or truncated if it exists.
options?
Section titled “options?”Returns
Section titled “Returns”Promise<number>
Bytes written — equal to the upload’s size unless the store came back short, which is the case a caller polling a streamed upload has to notice.
Throws
Section titled “Throws”when no size was given and the upload is
still arriving.
stepSpeakToUpload()
Section titled “stepSpeakToUpload()”stepSpeakToUpload(
text,options):Promise<SpokenUpload>
Speak text, master it, and store it — answering with the record and the
duration.
Parameters
Section titled “Parameters”string
What to say. Refused when blank, as stepSpeak refuses it.
options
Section titled “options”See SpeakToUploadOptions.
Returns
Section titled “Returns”Promise<SpokenUpload>
Throws
Section titled “Throws”whatever stepSpeak throws (no synthesizer published, no
credential), and for master: "mp3" the verdict throwFfmpegStepError
reaches on a failed encode — fatal on an exit or a missing binary.
transcodeUpload()
Section titled “transcodeUpload()”transcodeUpload(
uploadId,options):Promise<TranscodedUpload>
Read an upload to a temp file, run ffmpeg over it, and store what it wrote.
Parameters
Section titled “Parameters”uploadId
Section titled “uploadId”string
An upload in the agent’s own store. Must be complete — one
still arriving is refused before anything is read, as readUploadToFile
refuses it.
options
Section titled “options”Returns
Section titled “Returns”Promise<TranscodedUpload>
Throws
Section titled “Throws”on an ffmpeg exit, a missing binary, an output past the
cap, or an output ffmpeg wrote nothing to; RetryableError-shaped
(the FfmpegError unchanged) on a timeout or an abort — throwFfmpegStepError’s
verdict, which its doc argues.
Throws
Section titled “Throws”when the upload is still arriving.
withTempDir()
Section titled “withTempDir()”withTempDir<
T>(work,options?):Promise<T>
Run work with a private temp directory, and remove it afterwards.
join(tmpdir(), …) rather than a /tmp literal, which is this repo’s rule
(guard-invariants rule 11) and not merely portability theatre: on Windows a
literal /tmp/x is DRIVE-RELATIVE, so it resolves somewhere that does not
exist and every write fails with ENOENT. A step runs in a Linux guest when it
is deployed and on the developer’s own machine under aai dev, which is the
half that makes it matter.
The removal is in a finally, so it also runs on the failure paths — a guest’s
disk is small, and a step that leaves a copy of every recording it touched
fills it. force, so a run that never created its output does not fail HERE
and replace the real error with this one.
Type Parameters
Section titled “Type Parameters”T
Parameters
Section titled “Parameters”(dir) => Promise<T>
Called with the directory. Its result is this call’s result, so a step returns an upload id out of the scope rather than a path into it.
options?
Section titled “options?”See WithTempDirOptions.
Returns
Section titled “Returns”Promise<T>
writeUploadFromFile()
Section titled “writeUploadFromFile()”writeUploadFromFile(
path,options?):Promise<UploadInfo>
Store a local file as an upload, streaming it, and answer with the record.
The composition rather than the generator, and that is the whole design of this
function: stepWriteUpload(fileChunks(path), { … }) is three lines a caller can
write, and one of the three is a trap that has to be re-explained every time it
is written (see below). Handing over the composition means the trap is tested
once, here, by step-files.test.ts — where deleting the .slice() fails a
spec — rather than being a warning comment in every template that copies it.
A stream rather than readFile for the reason the windows above exist: the
converted audio is usually the largest thing a media step touches, and handing
the store an AsyncIterable is what keeps it off the heap.
Parameters
Section titled “Parameters”string
The file to store. Read to EOF; never modified or removed, so a withTempDir scope is still what owns its lifetime.
options?
Section titled “options?”name and type are stored verbatim and neither is inferred —
pass both, since type is what the byte route serves as Content-Type and a
browser will not play a file it was handed as bytes. See
WriteUploadFromFileOptions.
Returns
Section titled “Returns”Promise<UploadInfo>
Type Aliases
Section titled “Type Aliases”ReadUploadToFileOptions
Section titled “ReadUploadToFileOptions”ReadUploadToFileOptions =
object
Input
Options for readUploadToFile.
Properties
Section titled “Properties”concurrency?
Section titled “concurrency?”
optionalconcurrency?:number
How many windows to read at once. Defaults to
STEP_FILE_READ_CONCURRENCY; rounded down and floored at 1, as
mapConcurrent does.
Read only when size is absent. That option puts the completeness
judgement on the caller, and judging it means seeing the windows in order —
see readUploadToFile, which is where the two paths are cut apart.
optionalsize?:number
How many bytes the upload holds. Defaults to what stepRequireCompleteUpload
reports — so with no size, an upload that is still ARRIVING is refused.
Pass it only when you already have the record — a step that reported the
file’s name and size before starting has one, and this saves a second look.
Passing a size LARGER than the store holds is not an error: stepReadUpload
clamps its window to what has arrived, and this walk stops at what it was
actually given rather than at what it asked for.
Passing one moves the completeness judgement to the CALLER, which is
what makes a polling body expressible: this option means “I have read the
record”, and a caller who has read it can see complete for itself. It
therefore has to read it — stepUploadInfo(id).size threaded in here is the
whole bug this default now refuses, since that number IS the prefix.
windowBytes?
Section titled “windowBytes?”
optionalwindowBytes?:number
Bytes per read. Defaults to STEP_FILE_WINDOW_BYTES.
SpeakMaster
Section titled “SpeakMaster”SpeakMaster =
SpeakMasterFormat| {args?: readonlystring[];format:SpeakMasterFormat; }
Input
What stepSpeakToUpload masters to: the format alone, or the format
with the ffmpeg flags to apply on the way — a loudnorm pass, a sample
rate, a bitrate — inserted between -i <source> and the encoder’s own.
format: "wav" with args still runs ffmpeg (the WAV is re-encoded with the
flags applied); without args it stores stepSpeak’s WAV as it is.
Union Members
Section titled “Union Members”Type Literal
Section titled “Type Literal”{ args?: readonly string[]; format: SpeakMasterFormat; }
optionalargs?: readonlystring[]
Filter and encoder flags, e.g. ["-af", "loudnorm=I=-16:TP=-1.5:LRA=11"].
format
Section titled “format”format:
SpeakMasterFormat
SpeakMasterFormat
Section titled “SpeakMasterFormat”SpeakMasterFormat =
"mp3"|"wav"
The container stepSpeakToUpload stores.
SpeakToUploadOptions
Section titled “SpeakToUploadOptions”SpeakToUploadOptions =
SpeakOptions&object
Input
What stepSpeakToUpload accepts: stepSpeak’s options, plus the
container and the name to store under.
Type Declaration
Section titled “Type Declaration”master?
Section titled “master?”
optionalmaster?:SpeakMaster
"wav" stores the synthesized WAV as it is; "mp3" masters it with ffmpeg
first; the object form adds ffmpeg flags to either. Defaults to "wav".
name:
string
Filename to store, e.g. "digest.mp3" — what a page puts on the download
link. Required, because the right name depends on master and a default
that disagreed with it (summary.wav holding MPEG frames) is a file a
browser will not play.
timeoutMs?
Section titled “timeoutMs?”
optionaltimeoutMs?:number
Wall-clock budget for the mastering run. Defaults to DEFAULT_FFMPEG_TIMEOUT_MS.
SpokenUpload
Section titled “SpokenUpload”SpokenUpload =
UploadInfo&object
Sealed
What stepSpeakToUpload resolves with: the stored record, plus what was spoken.
Type Declaration
Section titled “Type Declaration”durationMs
Section titled “durationMs”durationMs:
number
How long the audio lasts — stepSpeak’s own measurement, from the byte count.
ffmpegMs
Section titled “ffmpegMs”ffmpegMs:
number
Wall-clock milliseconds the mastering run took — 0 when no ffmpeg ran.
voice:
string
The voice that actually spoke, with the default filled in.
TranscodedUpload
Section titled “TranscodedUpload”TranscodedUpload =
object
Sealed
What transcodeUpload resolves with — a step’s whole return value, so everything here survives the journal.
Properties
Section titled “Properties”bytes:
number
How many bytes of the SOURCE were materialized — the recording’s size.
ffmpegMs
Section titled “ffmpegMs”ffmpegMs:
number
Wall-clock milliseconds the ffmpeg run took.
id:
string
The stored output’s id, as writeUploadFromFile minted it.
optionalinfo?:MediaInfo
What ffprobe made of the source, when probe was asked for.
size:
number
The stored output’s size in bytes.
stderr
Section titled “stderr”stderr:
string
The tail of ffmpeg’s log, on success too: it carries the encode summary.
TranscodeUploadOptions
Section titled “TranscodeUploadOptions”TranscodeUploadOptions =
object
Input
What transcodeUpload accepts.
Properties
Section titled “Properties”args: (
source,out,info) => readonlystring[]
The argv AFTER the standing flags — -i <source>, the filter and encoder
flags, and <out> last — given the two paths this call chose and, when
probe was asked for, what ffprobe made of the source: an argv that
depends on the probed rate or channel count reads info rather than
probing again in a step of its own.
ffmpegBaseArgs() (-hide_banner -loglevel error -nostats -nostdin -y)
is prepended by this call and is not the caller’s to spell: -nostdin is
the flag whose absence is a hung process in a guest, and it was the one
most often left out. A filter that reports through the LOG needs
loglevel: "info" instead.
Parameters
Section titled “Parameters”source
Section titled “source”string
string
MediaInfo | undefined
Returns
Section titled “Returns”readonly string[]
loglevel?
Section titled “loglevel?”
optionalloglevel?:string
ffmpegBaseArgs’s loglevel. Defaults to "error".
output
Section titled “output”output:
object
What to store the result as. Both fields are stored verbatim and neither is
inferred (writeUploadFromFile says why type matters); the name’s
EXTENSION also names the output file ffmpeg writes, which is how ffmpeg
picks a container when the argv names no -f.
name:
string
type:
string
probe?
Section titled “probe?”
optionalprobe?:boolean
Also run ffprobe over the materialized source FIRST, hand its
MediaInfo to args, and answer it as info. Off by default — a
probe is a second subprocess, and most transcodes do not need to know what
they were handed.
signal?
Section titled “signal?”
optionalsignal?:AbortSignal
Kill the run when this aborts. Combined with timeoutMs, not replaced by it.
timeoutMs?
Section titled “timeoutMs?”
optionaltimeoutMs?:number
Wall-clock budget for the ffmpeg run. Defaults to DEFAULT_FFMPEG_TIMEOUT_MS.
WithTempDirOptions
Section titled “WithTempDirOptions”WithTempDirOptions =
object
Input
Options for withTempDir.
Properties
Section titled “Properties”prefix?
Section titled “prefix?”
optionalprefix?:string
Prefix for the directory’s name, under the OS temp directory.
Defaults to "aai-step-". Worth setting to something naming the pipeline
("aai-normalize-"): the directory is gone by the time anyone looks, so the
prefix’s real audience is a person reading ls /tmp during a run that hung,
and a spec asserting that nothing was left behind.
WriteUploadFromFileOptions
Section titled “WriteUploadFromFileOptions”WriteUploadFromFileOptions =
WriteUploadOptions&object
Input
Options for writeUploadFromFile — WriteUploadOptions, plus the window.
Type Declaration
Section titled “Type Declaration”windowBytes?
Section titled “windowBytes?”
optionalwindowBytes?:number
Bytes per read. Defaults to STEP_FILE_WINDOW_BYTES.
Variables
Section titled “Variables”STEP_FILE_READ_CONCURRENCY
Section titled “STEP_FILE_READ_CONCURRENCY”
constSTEP_FILE_READ_CONCURRENCY:number
Windows readUploadToFile reads at once, when the file is known to be whole.
What it costs is memory, and exactly this much: the width times
STEP_FILE_WINDOW_BYTES, i.e. 32 MiB held while a copy is in flight,
because a window is buffered before its write starts. That is the same budget
the WRITE half of this round trip already accepts — UPLOAD_WINDOW_CONCURRENCY
(aai-runtime/uploads/store.ts) is 4 over the same 8 MiB window, and its doc
calls that “the number that decides whether the uplink and the bucket work at
the same time or take turns”. Matching it is the whole argument for this value:
a step that pulls a recording in and pushes it back out should not hold two
different amounts of it, and 4 is a width the guest is already sized for.
It is UNMEASURED on the READ side, and that is stated rather than dressed up
in a table — the write width was swept against a bucket and an uplink, and
nothing here has been swept against the brokered read path. What is known is
the shape of the cost it attacks: on a deployed guest each window is a brokered
302 + Range GET against object storage
(aai-runtime/uploads/blobs-brokered.ts), which UPLOAD_PART_BYTES
(sdk/upload-constants.ts) measures at 1.9-4.3 MB/s per request, so a serial
walk cannot start window N+1 until window N has fully landed. Re-measure before
moving it; a wider default costs a guest’s resident set linearly, and a metered
link takes back throughput that width alone tries to buy.
STEP_FILE_WINDOW_BYTES
Section titled “STEP_FILE_WINDOW_BYTES”
constSTEP_FILE_WINDOW_BYTES:number
Bytes moved per store round trip, in either direction.
8 MiB is large enough that a two-hour recording is a few hundred round trips rather than tens of thousands, and small enough that a step’s resident set is a constant rather than a function of the recording. The number this must NOT be is “the whole file”, which is the shape every first draft has — and the reason the window is nameable at all is that both functions below take it as an option, which is what makes their multi-window paths reachable from a spec without writing 16 MB to a disk.