Voice notes

Receiving a person's voice notes with their transcript, fetching the audio, and answering with a voice note — your own audio or one nmbr speaks for you.

Markdown version: /developers/docs/voice.md · Spec: /developers/openapi.yaml

Voice in

A person's voice note arrives as a normal message.received with message.type: "voice", message.audio: { data, duration } and — because nmbr transcribes every voice note in the background, after the event was queued — no transcript yet. The words follow as their own event:

{ "type": "message.transcript", "payload": { "conversationId": "…", "messageId": "…", "from": { … }, "transcript": "call me back about the invoice" } }

Treat the pair as one message: keep the messageId, act when message.transcript lands (usually within seconds). Both are masked without the person's messages:read — transcript is null with accessDenied, so your agent knows a voice note exists that it may not read.

audio.data is an /objects/… path only nmbr's apps can open. To run your own speech model, fetch the file with GET /agent/v1/messages/:messageId/audio (the SDK's agent.getMessageAudio(id) returns the Response); it streams the audio with its content type and supports Range. The other side's notes need their messages:read; your own need nothing.

Voice out

POST /messages with type: "voice" in one of two ways:

Your own audio. audioData is a base64 data URL — mp3, m4a/aac, webm, ogg/opus or wav, at most 10 MB — plus audioDuration in seconds. nmbr stores the file and the bubble plays it like any voice note; the row never keeps an inline data URL. Add transcript so the words show under the note and travel with it: nmbr does not transcribe agent audio. A path you got from nmbr (/objects/…, e.g. forwarding a note) is accepted as is. Anything else is 400 invalid_audio.

Let nmbr speak. tts: { text, voice? } synthesizes the audio server-side (OpenAI tts-1; voices alloy, echo, fable, onyx, nova — the default — and shimmer; up to 4096 characters), stores the mp3, sets the transcript to the text and, when you leave content out, uses the text as the caption too. The duration is estimated from the text so the bubble shows a length before playback.

curl -s -X POST https://nmbr.ai/api/agent/v1/messages -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"conversationId":"…","type":"voice","tts":{"text":"Your build finished. Two tests failed — want the log?","voice":"onyx"}}'

await agent.speak({ conversationId }, "Your build finished. Two tests failed — want the log?", { voice: "onyx" });

503 tts_unavailable means speech isn't configured on this deployment (send your own audio); 502 tts_failed means the synthesis call failed — retry or fall back to text. A voice note counts as a message for limits and ceilings like any other.

Answering a voice note with a voice note

The loop this phase was built for: on message.transcript, think, then agent.speak({ conversationId }, answer). If the answer is long, stream the text with the stream first and follow with the spoken version — or skip the audio when the transcript came back null, and say so.