How to build a voice agent for a physical device, end to end
Published November 4, 2026 · By Port OS Team
A voice agent on a physical device has the same core as any voice agent: speech recognition, a language model and speech out. The device changes how listening starts and adds parts that app and phone agents never need: a wake word with an audio ring buffer, echo cancellation, session management, terminal states, arbitration between several devices and audio ducking. This guide shows the whole architecture, end to end, and changes the pipeline one step at a time.
Why this guide
Most voice agent guides stop at the software loop. AssemblyAI's guide to building AI voice agents is a good example: speech to text, an LLM, text to speech, and turn detection with interruption handling, built for phone calls and browser apps. That covers the core well. It does not cover a microphone in a room.
At the same time, people keep asking for real-time voice AI with a body. On the OpenAI developer forum, for example, a feature request asks for ChatGPT in a home speaker with a custom wake word and smart home control, and similar threads ask for desktop companions and robots. We could not find a guide that explains how to build one end to end with a physical device.
We have built production voice assistants and the Port OS device platform. This guide gives the high-level architecture of a voice agent on a device. It starts from the two common pipelines, shows the device variant of each, then adds the remaining device parts one at a time. In every diagram, yellow marks what that step adds or changes.
| Part | Phone or app voice agent guides | A physical device |
|---|---|---|
| Speech to text, LLM, text to speech | Covered | Needed |
| Turn detection and interruption | Covered | Needed |
| Wake word and ring buffer | Not needed | Needed |
| Echo cancellation of the device's own audio | Handled by the phone or browser | Needed |
| Session lifetime and false wakes | Ends with the call | Needed |
| Timeouts and terminal states | Ends with the call | Needed |
| Several devices hearing one request | Not needed | Needed |
| Ducking music and TV audio | Not needed | Needed |
The standard voice pipeline
The standard (cascaded) voice pipeline, as most guides describe it for a phone call or an app.
The standard pipeline has five stages:
- ASR and endpointing. Speech recognition turns audio into text, and an endpointing model decides when the person has finished.
- NLU or LLM. An intent model or a language model works out what the person wants.
- Request classification. The request is sorted into a kind, such as home control, music or a timer, which decides what to fetch next.
- Tool calls and context building. Device data, skills and tools are fetched based on the request, and the model calls the tool that takes the action.
- TTS. Text to speech says the result.
Classic intent-based assistants turned off a light in about 1.3 seconds from the end of speech in our measurements. That is the bar people compare you to. Our post on how Alexa works shows the timing of each stage.
1. ASR and endpointing
Endpointing decides when to stop listening. Wait too long and every request feels slow. Stop too early and you lose "...in the kitchen".
- Fix: start the next stages on a partial transcript, and cancel them if the person keeps talking.
- Cost: wasted compute on cancelled work. It saves real time on every request.
Device names are often not dictionary words: brand names, "Lamp 2 living", a child's name for a night light. A general speech model misspells them, and the misspelling then fails to match.
- Fix: bias recognition with the home's own device and room names.
- Cost: a per-home vocabulary to keep current. Little extra latency.
2. NLU or LLM
Out-of-the-box models in 2023 succeeded less than 30% of the time for us. Fine-tuned Llama models reached about 70 to 80%.
- Fix: fine-tune a small model on real device-control requests.
- Cost: training data and evaluation sets, both scarce. A small fine-tuned model is faster than a large general one.
Models are also slow at generating long IDs, and some cannot reproduce a UUID reliably. In our measurements, each output token cost a few milliseconds, so a single UUID added about 120 ms.
- Fix: replace device IDs with short aliases in the prompt, and map them back after the call.
{
"devices": [
{ "id": "d1", "name": "Kitchen light", "room": "Kitchen", "type": "LIGHT" },
{ "id": "d2", "name": "Lamp 2 living", "room": "Living room", "type": "LIGHT" }
]
}3. Request classification
A general assistant has tools for music, calendars, timers and devices. Loading every tool on every request makes the prompt long and the model slow.
- Fix: classify the request first, and load only that kind's skills and tools.
- Cost: a classification step, which can pick the wrong kind. Keep the classifier small and fast.
4. Tool calls and context building
A large home can have more than a hundred devices, each with many settings. Putting all of them in the prompt makes the model slower and less accurate.
- Fix: search the home for the devices that match the request, and pass only those, with rooms and groups. Fetch device context, skills and tools in parallel.
- Cost: one search step before the model call.
Real homes also have a light group called "Living room", a switch called "Living room switch" and lamps called "Lamp 1 living room" and "Lamp 2 living". Asked to turn off the living room lights, models target only some of them, or pick the group in one request and the lamps in the next.
- Fix: give the model rooms, groups and device types as data, plus facts like which switch controls which light. Prompt rules alone were flaky for us.
- Cost: a better device data model. No extra latency, and the largest accuracy gain we found.
Most devices are controlled through the maker's cloud. That call can be slow, and it can succeed without the device changing. Read the state back when it matters, and say "queued" instead of "done" when the service has not confirmed. Port OS gives an agent one control API across brands, with rooms, groups and device types already modeled. See the Port OS API and the smart home MCP server.
5. TTS
Start speaking as soon as you know the result, and stream the audio. A short confirmation ("Done") is faster than a sentence.
The standard duplex pipeline
The standard duplex pipeline: a voice model holds the conversation, and a separate agent does the work.
Newer assistants, such as Alexa+ and Gemini Live, split the job in two:
- Duplex audio. A speech-to-speech model listens and talks at the same time. It handles turn taking and interruptions and answers at once.
- Hand-off to another model. For anything that needs business logic, the duplex model sends a text request to an agent. The agent fetches context, calls tools and returns a result for the duplex model to say.
This makes the conversation feel fast while the task runs at the agent's speed, which can be many seconds.
The device variant: wake word and endpointing
On a device, listening no longer starts when an app opens a stream. It starts when the device hears its wake word, from microphones that may be across the room. That changes the first stages of both pipelines.
The device variant of the standard pipeline.
The device variant of the duplex pipeline. The same wake word and audio changes apply.
Wake word. The device runs a small wake word model locally and sends no audio until it fires. Near-field devices (a phone, a remote) hear the person close up. Far-field devices (a speaker across the room) need a microphone array and beamforming to pick one voice out of a noisy room.
Ring buffer. People often run the wake word straight into the request: "Alexa turn off the light", with no pause. By the time the wake word fires, the start of the request has already passed.
- Fix: keep a short ring buffer of recent audio, a second or two, and send it from the start of the wake word, so ASR or the duplex model hears the whole utterance.
- Cost: a little memory on the device. No extra latency.
Endpointing. Endpointing now starts at the wake word, on far-field audio with room noise. In the duplex variant, the duplex model's turn taking takes this job, but it still needs the session to begin at the wake word, with the buffered audio.
The remaining steps build on the device variant of the standard pipeline. Each one applies to the duplex variant in the same way.
Echo cancellation
The device's own speaker is the loudest sound its microphones hear. Without help, it hears its own replies, its music and its TTS, and can wake itself or transcribe itself.
- Fix: acoustic echo cancellation (AEC) subtracts what the device is playing from what the microphones capture, using the playback signal as a reference. It is what lets a person interrupt (barge in) while the device is talking.
- Cost: a few milliseconds of signal processing. It must run on the device, where the playback reference is.
Session management
A session is one conversation with the device. Two questions decide how it behaves.
How does it stay coherent over many turns and long durations? Two approaches work:
- Keep one context stream for the session and compact it over time, summarizing older turns as it grows.
- Keep only the last N turns. This is simpler and bounded, and it forgets older context.
How do you know a wake was false? End the session early based on what ASR and AEC report:
- The audio after the wake was too short, or held no speech.
- Nothing was captured during the duplex conversation or after TTS.
- ASR returned nothing that sounds like a request.
A false wake that ends quietly costs nothing. One that answers "Sorry, I didn't get that" to a TV line is the failure people remember.
Terminal states and the runtime controller
A device session has to end cleanly, or the device keeps listening, keeps a stream open and keeps paying for it. Give the voice runtime a controller, an orchestrator that consumes events from the device and from the agent and owns the session's state.
- Timeouts. If no event reaches the core runtime from the device or the agent for a set time, say one minute, the controller ends the session.
- Direct ends. The person can end it ("stop", "never mind"), and the agent can end it when the task is done.
- Control channel. Every stage reports to the controller, and only the controller closes streams, so a slow agent cannot leave the microphone open.
Multi-device arbitration
In a home with several speakers, one request wakes more than one device. Without arbitration, two devices answer at once, or the wrong room acts.
- Fix: collect the audio from every device that woke, synchronize the streams in time, and merge them, or pick the best one by signal quality. Then answer from one device, usually the closest.
- Cost: a short wait for every device's audio to arrive.
Arbitration also changes context building: the device that heard the request is the source device, and "turn off the light" means the light in its room.
Audio ducking
When music or the TV is playing, the device has to hear the person over it and be heard over it.
- Fix: when the wake word fires, lower (duck) the media playing on the device, and on other devices in the room when you can control them. AEC removes what the device itself is playing. Restore the volume when the session reaches a terminal state.
- Cost: the ducking must follow the controller's session state exactly, or the music stays quiet after a false wake or a timeout.
What each fix costs: summary
| Fix | Accuracy gain | Latency cost |
|---|---|---|
| Start work on partial transcripts | None | Saves time |
| Short device IDs | Small | Saves time |
| Device search instead of the full home | Medium | One search step |
| Request classification | Medium | One classification step |
| Rooms, groups and device facts as data | Large | None |
| Fine-tuned small model | Large | Saves time compared with a large model |
| Ring buffer at the wake word | Large for fast talkers | None |
| Echo cancellation | Large while audio plays | A few milliseconds |
| Multi-device arbitration | Large in homes with several devices | A short wait for every stream |
| Multi-step agent with checks | Largest | Seconds per extra step |
Most accuracy in voice agents has been bought with time. The fixes that cost no time are about data and audio: put the facts in front of the model, and give it clean audio of the whole request.
Frequently asked questions
What latency should a voice agent for device control aim for?
People compare it with classic voice assistants, which turned off a light in about 1.3 seconds from the end of speech in our measurements. Aim to start the reply within about a second, and confirm the action as soon as the device service answers.
Should I use a speech-to-speech model or a pipeline?
A duplex speech-to-speech model gives a more natural conversation and can talk while work runs. A pipeline gives you more control over each stage, including recognition of device names. Many systems use a duplex model for the conversation and a separate agent for the device call. Both need the same device parts.
Why does a voice device need echo cancellation?
Its own speaker is the loudest thing its microphones hear. Echo cancellation removes the device's own playback from the microphone signal, so the device does not wake itself and the person can interrupt it.
How do I stop several speakers from answering at once?
Arbitrate: collect the audio from every device that woke, synchronize and merge it, and answer from one device, usually the one closest to the person.
How do I stop the agent from controlling the wrong device?
Give the model rooms, groups, device types and facts such as which switch controls which light. Pass only the devices that match the request, and use the room of the device that heard the request. Read the state after the command when a mistake would matter.