Class 3: Evaluation and Deployment

Time: 1 hour (15 min lecture + 45 min hands-on)

Goal: Measure your model, find out where it fails, and put it in front of other people.

Before you start. You need a model on the Hugging Face Hub. If your Class 2 run has pushed at least one checkpoint, use that. If it hasn't finished, or you skipped the cloud run, use the published model instead; every command below works the same way.

export MODEL=your-username/tiny-audio-yourname   # or: mazesmazes/tiny-audio

The commands below refer to $MODEL, and to $NAME for its short name (the part after the slash, e.g. tiny-audio-yourname), which is how the analysis tools identify it.

---

Part A: Lecture (15 min)

Word Error Rate (WER)

WER = (Substitutions + Insertions + Deletions) / Words in the reference

Align the prediction against the reference word by word, count the edits, divide by the reference length. A WER of 10% means roughly one word in ten is wrong. It can exceed 100% if the model hallucinates a lot of extra words.

WER	Quality
< 5%	Excellent. Commercial APIs on clean speech
5-10%	Very good
10-20%	Good. A realistic target for the course run
> 30%	Poor. Something is wrong with the model or the data

Normalization matters. Before scoring, both reference and prediction go through Whisper's EnglishTextNormalizer: lowercase, punctuation stripped, numbers and common spellings standardized ("twenty five dollars" and "$25" agree). Without this, a perfect transcript with different comma placement would count as errors. It also means WER says nothing about your model's punctuation or capitalization. The raw pair is saved alongside the normalized one so you can inspect those by hand.

This is also why evaluations disagree. A WER depends as much on how the reference transcripts were written, and what was tidied before scoring, as on the model. If a customer's references spell out numbers and the model writes digits, or their references have no punctuation and a scorer without normalization compares raw text, every one of those counts as an error even though the model heard every word. Two systems can swap places on the same test set just by changing the normalizer. Before trusting any WER, yours or a vendor's, ask two questions: how were the references written, and what was normalized before scoring? Often the gap between two models is a formatting gap, not a hearing gap.

Evaluation Datasets

Tiny Audio's registry covers 13 test sets. Each stresses something different:

Name (-d)	Domain	Why it's hard
loquacious	Mixed read and spontaneous English (default)	Broad benchmark
librispeech, librispeech-other	Audiobooks	Clean baseline; other is harder speakers
tedlium	TED talks	Presentational speech, technical vocabulary
commonvoice	Crowd-sourced read sentences	Thousands of speakers, accents, cheap mics
voxpopuli	European Parliament	Non-native accents
peoples	Public-domain speech	Varied recording quality
gigaspeech	Podcasts and YouTube	Conversational, noisy
earnings22	Earnings calls	Financial jargon, names, numbers, phone-quality audio
spgispeech	Financial presentations	Formatted numbers and entities
ami, ami-sdm	Meetings	Overlapping speakers; sdm is a single distant microphone
expresso	Expressive read speech	Emotional and whispered speech

A model that does well on LibriSpeech and badly on AMI is normal. A model that does badly on LibriSpeech has a problem. One benchmark is not an evaluation: the gap between domains is your roadmap, because it tells you which data to add to the training mix next.

Error Analysis

An aggregate WER tells you how much the model fails. To improve it you need to know how. Start with the split from Class 1 and ask, for each wrong word, which part failed:

Kind of error	Example	Where it lives	Usual fix
Hearing mistake	A mumbled name becomes an unrelated word	Encoder	More audio like it (accent, noise, mic) in the mix
Language mistake	"there" for "their"; a name heard right, spelled wrong	Decoder	More text like it; a larger or better-tuned decoder
Formatting gap	"$25" vs "twenty five dollars"; missing punctuation	Labels / scorer	Check the normalizer and the references, not the model
Mode failure	Empty output, endless repetition	The whole run	Check the loss curve and gradient flow

Then look at the data three ways:

• Worst samples: sort by per-sample WER. The top of the list is usually noisy audio, mislabeled references, or a mode failure (empty output, endless repetition).
• Entity errors: names, places, organizations, and numbers. These carry the most meaning and are the hardest for a small decoder to spell.
• Comparison: the same clips through another model. If everyone fails on a clip, blame the clip.

Deployment Options

Option	Cost	When
Hub repo	Free	Already done: training pushed your checkpoints there
Hugging Face Space (Gradio)	ZeroGPU	A public demo anyone can try in a browser
ta serve on RunPod	Paid GPU	A production HTTP API with batched GPU inference
Local server	Your hardware	Privacy, or wiring into your own app

---

Part B: Hands-On (45 min)

Exercise 1: Evaluate (10 min)

# Default benchmark: LoquaciousSet test split
poetry run ta eval -m $MODEL -n 200

# A second domain
poetry run ta eval -m $MODEL -d earnings22 -n 100

# Several at once
poetry run ta eval -m $MODEL -d loquacious -d tedlium -d ami -n 100

Each sample prints as it's scored:

Sample 1: WER=8.3%, Time=1.23s
  Ref:  the quick brown fox jumps over the lazy dog
  Pred: the quick brown fox jumped over the lazy dog

and each dataset ends with a summary table (WER, sample count, average time per clip).

Results land in outputs/<timestamp>_<short-name>_<dataset>/:

• results.txt: every sample with its WER, the normalized reference and prediction, and the raw (unnormalized) pair
• metrics.txt: the corpus-level numbers

Keep these directories: they are how you compare runs later.

Score the published model on the same datasets so you have a comparison point:

poetry run ta eval -m mazesmazes/tiny-audio -n 200

Exercise 2: Analyze Errors (15 min)

Worst samples. Every sample in results.txt starts with a Sample N - WER: X% line, so the worst ones are a sort away:

grep -h "WER:" outputs/*_${NAME}_loquacious/results.txt | sort -t: -k2 -rn | head -20

Open results.txt and read a dozen of the worst. Sort each into a bucket from the table above: hearing mistake, language mistake, formatting gap, or mode failure, plus "bad reference label" for clips where the reference itself is wrong. The bucket that dominates tells you which part, or which data, to fix.

Compare models. Evaluate the published model on the same datasets and sample count, then read the corpus WER from each run's metrics.txt:

grep -H "^wer:" outputs/*_loquacious/metrics.txt

Exercise 3: Check Your Hub Repo (5 min)

Open https://huggingface.co/$MODEL. Training pushed each checkpoint there as it was saved, so the repo already contains:

• model.safetensors: the trained projector weights. The frozen encoder and decoder are not stored; config.json names them and they are downloaded from their own repos at load time
• adapter_model.safetensors and adapter_config.json: the decoder's LoRA adapters
• config.json: the ASRConfig, including which encoder and decoder to load
• asr_modeling.py, projectors.py, and the other custom code files that make trust_remote_code=True work
• Tokenizer and feature-extractor files

Two things it does not have yet: a model card, and any code fixes you made after training started. ta push uploads exactly those. It needs a Hub write token: the hf auth login cache, HF_TOKEN, or --hf-token:

poetry run ta push --repo-id $MODEL

It stages the tiny_audio/*.py files, the repo's MODEL_CARD.md as the Hub README.md, and requirements.txt, then uploads them. It does not upload weights. Edit MODEL_CARD.md to describe your model before you push it.

Test the result the way a stranger would:

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="your-username/tiny-audio-yourname",
    trust_remote_code=True,
)
print(pipe("audio.wav")["text"])

Exercise 4: Deploy a Demo to Hugging Face Spaces (10 min)

The demo/ directory is a complete Gradio app. It reads the model ID from the MODEL_ID environment variable, so the same code serves any Tiny Audio model.

1. Edit demo/README.md. Its front matter is the Space's card: set title, and change the models: list and preload_from_hub: to your model ID.

2. Deploy. The command creates the Space if it doesn't exist:

poetry run ta deploy --repo-id your-username/tiny-audio-demo

1. In the Space's Settings → Variables, add MODEL_ID = your-username/tiny-audio-yourname. The Space restarts. (Without this it serves the published model.)

ta deploy puts the Space on ZeroGPU (hosting a ZeroGPU Space needs a Hugging Face PRO account; without one, pick CPU in the Space's settings: slow, but it works). The first build takes a few minutes. Share the link.

To test locally before deploying:

poetry run ta demo --model $MODEL --port 7860

Exercise 5: Production Server (5 min)

For an HTTP API on a GPU, run ta serve on a RunPod pod. It loads the model once, merges the LoRA adapters, and batches audio chunks from concurrent requests on the GPU:

1. Try it locally first (CUDA, Apple MPS or CPU):

poetry run ta serve --model $MODEL --port 8000

1. Create an inference pod (an RTX 4090 by default), install, and start the server:

poetry run ta runpod up --serve
poetry run ta runpod wait <pod-id>          # prints <ip> <port>
poetry run ta runpod deploy <ip> <port>
poetry run ta runpod serve <ip> <port> --model $MODEL --no-attach

1. Load-test it (latency and throughput at 1, 8 and 32 requests in flight):

poetry run ta bench --url https://<pod-id>-8000.proxy.runpod.net

1. Call it with any HTTP client, or evaluate through it by passing its URL as the model:

poetry run ta eval -m https://<pod-id>-8000.proxy.runpod.net --endpoint -n 50 -w 8

---

Advanced Evaluation

Comparing with Commercial APIs

export ASSEMBLYAI_API_KEY='...'
poetry run ta eval -m assemblyai -d loquacious -n 200 -w 4          # universal-3-pro by default

export DEEPGRAM_API_KEY='...'
poetry run ta eval -m deepgram -d loquacious -n 200 -w 4            # nova-3

export ELEVENLABS_API_KEY='...'
poetry run ta eval -m elevenlabs -d loquacious -n 200 -w 4          # scribe-v2

# macOS only: Apple's on-device recognizer
poetry run ta eval -m apple-speech -d loquacious -n 200

-w runs API calls in parallel. Each run's metrics.txt holds its corpus WER, so the grep from Exercise 2 puts them side by side.

Every Dataset at Once

poetry run ta eval -m $MODEL -d all -n 100

all expands to the 12 plain ASR sets. It skips expresso and the speaker-attribution sets (ami-speakers, ami-speakers-long, ami-meetings-24, notsofar-test); name those explicitly when you want them.

Running Evaluation on the Pod

If you still have a RunPod instance up, evaluation is much faster there:

poetry run ta runpod eval <HOST> <PORT> -m $MODEL -d loquacious -d ami -n 500

---

Debugging Poor Performance

Symptom	Likely cause	What to do
High WER everywhere	Undertrained	Check the loss curve; train longer or on more data
High WER on one domain	Domain gap	Add that domain's data to your data config
High WER on accented speech	Training data bias	Add CommonVoice or VoxPopuli to the mix
Empty or one-word outputs	Audio too quiet, or the run collapsed	Inspect the clips and the W&B loss and grad-norm curves
Runaway repetition	Decoder loop	The pipeline truncates repeats; check max_new_tokens; more training usually fixes it
Great eval loss, bad WER	Mismatch between eval split and test set	Compare label formats; check normalization
Wrong casing or punctuation but good WER	Training labels lacked them	Expected with LoquaciousSet; add cased, punctuated data

---

Congratulations

You now have:

• A speech recognition model you trained, on the Hub, loadable with three lines of Python
• WER numbers on several domains, and a comparison against the published model
• A list of its worst failures and what kind they are
• A public demo

Where to go next:

• Retrain with a change and compare: a narrower projector (model.projector_hidden_dim=2048), a different stride (model.projector_pool_stride=2), or a different LoRA rank (model.lora_rank=32 model.lora_alpha=32)
• Add a second dataset to your data config, targeting the domain where you failed worst
• Read configs/data/multiasr.yaml to see how the production mix was assembled, and why corpora were added and removed
• Try a different encoder or decoder (model.audio_model_id, model.text_model_id); the comments in configs/config.yaml note what has to change with them, such as model.encoder_conv_layers

---

← Class 2 | Quick Reference →