EchoBenchDOCS
OPEN-SOURCE VOICE EVALUATION

Test every conversation.
Trust every release.

EchoBench simulates real callers, measures what happened, and shows exactly why a voice agent passed or failed.

Start in 5 minutes →See how it works
5 ready-to-run scenarios0 API keys required1 command to start
call_7f3a · date-correctionPassed
TTFA407ms
BARGE-IN138ms
ASSERTIONS6/6
CALLER
AGENT
C

Wait, make that Wednesday instead.

E

Got it. I’ve updated the date before booking.

01
QUICKSTART

From clone to first result

The reference suite works locally without accounts, credentials, or downloaded AI models.

1

Install everything

From the EchoBench project folder, install the backend and dashboard dependencies.

Terminal
make install
2

Start the API

Open a terminal and start the EchoBench control plane and scenario runner.

Terminal
make api
3

Start the dashboard

Keep the API running. Open a second terminal in the same project folder.

Terminal
make dashboard
4

Run the suite

Visit http://localhost:5173, select Run test suite, then open any completed call.

▶Run test suite5 scenarios
02
FIRST RUN

What happens when you click Run?

EchoBench executes five practical reservation scenarios and evaluates the full conversation.

01

Standard booking

Happy path

→
02

Caller changes date

Barge-in

→
03

Party-size correction

State update

→
04

Tool recovery

Failure retry

→
05

Unavailable time

Alternative offer

→
Passed5
Failed0
Median TTFA407 ms

Your values are measured on each run and may differ by machine and provider.

03
CALL TRACE

Read one conversation in layers

A call trace combines the audio your user heard with the internal events that explain it.

01Recording

Play the complete caller and agent audio.

02Waveforms

See speech, silence, and overlapping turns.

03Events

Inspect LLM, tool, TTS, and cancellation timing.

04Assertions

Compare expected behavior with observed evidence.

One turn · 407 msfast
EndpointSTTLLMTTS

Click any event dot in the dashboard to inspect the event at that moment.

04
SCENARIOS

Describe behavior, not test code

A scenario defines what the caller wants, what changes during the call, and what a correct outcome looks like.

scenarios/date-correction.yaml
name: Caller changes the date
caller:
  personality: impatient
  goal: Correct the booking before confirmation
turns:
  - text: Book a table for two Tuesday at seven.
  - text: Wait, make that Wednesday instead.
    interrupt: true
  - text: It's Maya Shah. Go ahead.
expected:
  date: Wednesday
  time: 7:00 PM
  party_size: 2
  name: Maya Shah
conditions:
  noise: coffee_shop
caller

The goal and conversational personality.

turns

What the caller says and when they interrupt.

expected

Facts that must reach the agent’s tool.

conditions

Noise, availability, and failure behavior.

05
ARCHITECTURE

One test, end to end

EchoBench stays separate from the agent being tested, so it measures the experience instead of trusting internal text.

1CallerGoal + behavior
audio →
2Agent under testSTT · LLM · tools · TTS
events →
3EvaluatorMetrics + assertions
results →
4DashboardTrace + comparison
◎

Black-box by default

Audio in and audio out gives the most realistic test.

⌁

Instrument when useful

Optional events reveal internal latency and tool behavior.

◇

Provider neutral

The protocol does not require a specific model vendor.

06
CONNECT AN AGENT

The EchoBench audio contract

External agents connect over WebSocket using mono PCM audio and optional JSON trace events.

ENCODINGPCM S16LE
SAMPLE RATE16 kHz
CHANNELSMono
FRAME SIZE20 ms
Session start
{
  "type": "session.start",
  "session_id": "call_123",
  "audio": {
    "encoding": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1,
    "frame_duration_ms": 20
  }
}
Optional instrumented event
{
  "type": "trace.event",
  "session_id": "call_123",
  "timestamp_ms": 812,
  "name": "tool.completed",
  "data": {
    "tool": "create_reservation",
    "status": "success",
    "duration_ms": 184
  }
}
07
PROVIDERS

Bring the stack you already use

EchoBench is designed so callers, STT, reasoning, and speech can be local, hosted, or mixed.

LAYERLOCAL OPTIONHOSTED OPTION
Caller LLMOllamaOpenAI-compatible API
Speech to textWhisperCloud STT adapter
Text to speechSystem voiceCloud TTS adapter
AgentLocal endpointStaging endpoint

Credentials are read only by the API server. They are never sent to the browser or stored in scenario files.

08
CLI & API

Use EchoBench without the dashboard

The CLI is ideal for CI and quick checks. The API supports custom workflows and integrations.

Terminal
uv run echobench demo
GET/health

Check whether the API is ready.

POST/api/runs

Start an evaluation run.

GET/api/runs/:id

Read status and call results.

GET/api/calls/:id

Inspect one complete trace.

GET/api/comparisons

Compare baseline and candidate.

WS/api/protocol/echo

Validate protocol frames.

Open interactive API reference ↗
09
TROUBLESHOOTING

Common setup fixes

Most problems are one of these four things.

The dashboard says “API unavailable” +

Confirm that make api is still running and that http://localhost:8000/health returns an OK response.

Docker cannot connect to its daemon +

Start Docker Desktop, wait until it reports that Docker is running, then retry docker compose up --build.

A call is stuck as queued after a restart +

Refresh the dashboard. EchoBench marks interrupted runs as cancelled when the API starts again.

Recordings use tones instead of speech +

The system speech tool is unavailable. Tone artifacts preserve the evaluation flow; connect a TTS adapter for spoken recordings.

READY TO LISTEN CLOSER?

Run the first conversation.

Five scenarios. No API key. Evidence on every result.

Start with EchoBench →