How Alexa works, and why AI assistants got slower: from slot filling to Alexa+
Published November 4, 2026 · By Port OS Team
Alexa works in five steps. The Echo listens on the device for the wake word. It streams what you say next to Amazon's cloud. Speech recognition turns the audio into text. A language system works out what you want and which device you mean. Alexa then sends a command to that device's service and speaks a reply.
That pipeline has changed three times since the first Echo shipped in 2014. Each change made the assistant understand more, and each one made it slower. We have built production voice assistants and the Port OS device platform, and this post explains each generation with the numbers we saw.
| Generation | Years | How it understands you | Typical time to act | What it gets wrong |
|---|---|---|---|---|
| Slot filling | 2014 to 2022 | Matches your words against fixed patterns | About 1.3 s | Anything that does not fit a pattern |
| LLM in the loop | 2023 to 2025 | One model call with selected tools and device context | About 2 to 3 s | Large homes, similar device names |
| Duplex voice plus background agent | 2025 onward | A voice model talks while an agent does the work | Reply at once, task 30 s to 1 min | The task is still slow |
| Chat agents over MCP | 2025 onward | The model loops through tool calls until it is done | 8 to 30 s in our tests | Speed, and asking you questions it could settle |
Step by step: how Alexa turns "turn off the kitchen light" into a command
1. Wake word on the device
The Echo runs a small wake word model locally. It does not send audio to the cloud until it hears "Alexa" (or the wake word you chose). After the wake word, it streams the audio of your request.
2. Speech recognition and endpointing
In the cloud, automatic speech recognition (ASR) turns audio into text. At the same time, an endpointing model decides when you have stopped talking. Endpointing matters more than people expect. If it waits too long, every request feels slow. If it cuts in early, it chops off "...in the kitchen".
3. Understanding: intents and slots
Classic Alexa used natural language understanding (NLU) built on intents and slots. An intent is the action, such as turning something off. Slots are the blanks the action needs, such as the device and the room. Developers of Alexa skills describe this with sample phrases like turn {state} the {device} in the {room}. Under the hood, weighted finite-state transducers (FSTs) and statistical models matched your words to the best intent and filled the slots.
4. Finding the device and sending the directive
For smart home requests, Alexa resolves the slot ("kitchen light") against the devices linked to your account. It then sends a directive to the device maker's cloud service. Amazon's Smart Home Skill API defines these directives by interface: a light that can switch on and off implements Alexa.PowerController, and "turn off" becomes a TurnOff directive for that endpoint. The device service answers with the new state.
5. Speech out
Alexa turns its reply into speech with text to speech (TTS), or simply plays a confirmation tone.
From the end of your sentence to the light going off, a matched request took about 1.3 seconds.
Generation 1 (2014 to 2022): slot filling and computer speak
Slot filling was fast and cheap. It was also brittle. It handled the phrases its designers expected and failed on most others. People learned to talk to it in short fixed commands, which is why "computer speak" became normal.
It also struggled as homes grew. If you had a light group called "All Hue Lights" and you said "turn off all lights", the match could be ambiguous enough that Alexa asked you which one you meant. The model had no way to reason about the house. It could only match names.
Generation 2 (2023 to 2025): an LLM in the loop
In 2023, teams started to replace the intent matcher with a large language model (LLM) that could call tools. The rest of the pipeline stayed the same.
Out of the box, this went badly. In our experience, early models turned off the right light less than 30% of the time. They were slow to generate device IDs, and some models could not reproduce a long UUID at all, so teams replaced IDs with short aliases. Fine-tuned Llama models raised accuracy to about 70 to 80%. That was usable for small homes and fragile for large ones.
By 2024 and 2025, production systems worked for simple requests in homes with about a dozen devices. They still failed for homes with a hundred or more. To get there, the pipeline grew:
- A router picks which tools and instructions to load for this request, because loading all of them is too slow.
- A search step pulls only the relevant devices into the prompt.
- Many of these steps run in parallel, because each one adds delay.
- Every token counts. In our measurements, each generated token cost a few milliseconds, so one UUID could add about 120 ms.
All that work held the time to act at about 2 to 3 seconds. That is still about twice the old target.
Generation 3 (2025 onward): Alexa+ and the duplex voice model
Amazon announced Alexa+ in February 2025. It is built on large language models and coordinates many services, which Amazon calls experts. Other assistants moved the same way, such as Gemini Live on Android phones.
The common design splits the job in two:
- A duplex voice model talks with you in real time. It can listen while it speaks, and it answers at once.
- A background agent does the actual work. It plans, calls tools, reads the results and repeats until it is done.
This makes the conversation feel fast. The task itself is slower than before. The agent has to collect context in several steps before it can act, and providers share compute across many users. In our experience, turning off the lights through an agent like this can take 30 seconds to a minute, for about the same device call that classic Alexa made in 1.3 seconds.
Generation 4: chat agents over MCP
ChatGPT, Claude, Grok and similar assistants can control a home through the Model Context Protocol (MCP), a standard way for an AI app to call outside tools. They do not have a voice pipeline tuned for speed. They loop: think, list devices, search, call, check.
We timed four of them on the same home. With clear device names, they took 8 to 11 seconds to turn off a light. On an ambiguous request, they took up to two minutes and then asked which light we meant.
These agents are the most capable of the four generations. They can also do things no voice assistant can, such as running a robot vacuum and reporting what it found.
Streaming pipelines and agent loops: the core design difference
Voice assistants and chat agents do much of the same work. Both classify the request, fetch context about the home (devices, rooms, what media is playing), choose skills and choose tools. The difference is when that work happens and how tightly it is tied to the rest of the system.
A streaming voice pipeline does as much as it can before the model runs. When you start talking, it fetches context about the device you are talking to: its name, its room and any media playing on it. When you stop, it classifies the request. Then it fetches the relevant devices, skills and tools in parallel, each searched with what you said. Once all of those return, one model call chooses the action, the tool call reaches the device, and a second model call writes the spoken reply. The whole design aims to get the request right zero shot, with as little time as possible inside model inference.
A streaming pipeline gathers context, skills and tools in parallel, so the model gets one focused try. Times are illustrative, and signals such as speaker recognition and device arbitration are left out.
That speed costs flexibility. Context fetching, skills and tools are coupled to the pipeline. A request the designers did not plan for gets the wrong context, and the model has no way to go and look for the right one.
An agent loop keeps device data, media state, skills and tools separate from the core logic and runtime. The model decides what to search for, which skill to load and which tool to call, one turn at a time, and it reads each result before it decides the next step. It does not try to get the request right in one shot.
An agent loop fetches what it needs after you ask. Each model call takes about 4 to 5 seconds, and the vertical line marks where the single-shot streaming pipeline is already done.
This is why an agent can handle almost any request and why it is slow at simple ones. Products that stream voice, such as Alexa+, still run an agent loop like this behind the conversation. The voice part answers quickly, and the loop sets how long the task takes.
| Design choice | Streaming voice pipeline | Agent loop |
|---|---|---|
| When context is fetched | The source device while you speak, the rest in parallel right after classification | After you ask, by the model, turn by turn |
| Skills and tools | Chosen by the pipeline | Chosen by the model, kept separate from the runtime |
| Model calls per request | Two: one chooses the action, one writes the reply | Several, about 4 to 5 s each |
| Optimized for | Getting it right zero shot | Getting it right eventually |
| Handles requests nobody planned for | Poorly | Well |
| Time to turn off a light | About 1.3 to 3 s | 8 to 30 s or more in our tests |
The goal, and where assistants are today
People want four things at once from a home assistant. No design delivers all four today.
| Goal | Target | Streaming voice pipeline today | Agent loop today |
|---|---|---|---|
| Fast | Reply and act within about 1.3 s of the end of speech | Close: about 1.3 s for fixed commands, 2 to 3 s with an LLM | Far: 8 to 30 s, minutes when stuck |
| Good | The right devices on the first try, in any home, for any phrasing | Good for small homes and planned requests, weak on the long tail | Best understanding, but often asks which device you meant |
| Safe | Never takes an action you did not want | Strong: a fixed set of tools limits mistakes | Depends on what the connection allows and on confirmations |
| Cheap | Low cost per request at millions of requests a day | Low: one small model call | High: several large model calls per request |
Six ways to run inference, compared
Each architecture trades these goals differently. In the table, fluent means it holds a natural conversation, and reactive means it responds at once while you talk or while work is still running, for example when you interrupt it.
| Architecture | Cost | Latency to act | Accuracy | Safety | Fluent | Reactive |
|---|---|---|---|---|---|---|
| NLU with pFSTs (classic slot filling) | Very low | About 1.3 s | High on planned phrases, fails the rest | Strong: fixed intents only | No: fixed commands | No |
| LLM backend for NLU (streaming pipeline) | Low: one or two small model calls | About 2 to 3 s | Good in small homes, weak when the pipeline fetches the wrong context | Strong: the pipeline picks the tools | Partly: any phrasing in, short replies out | No |
| Voice to voice agent | Medium to high: an audio model runs for the whole session | Replies in under a second, actions vary | Weaker at tool calls than text models in our experience | Harder to check before it acts | Yes | Yes |
| LLM backend on fast inference hardware, such as Cerebras | High: tokens about 10 times faster at a correspondingly higher price | Close to 1.3 s, with room for a larger model or a check step | Better than the plain LLM backend | Strong: the same pipeline picks the tools | Partly | No |
| Asynchronous long running agent | High: several large model calls | 8 to 30 s, minutes when stuck | Best on open requests | Depends on what the connection allows and on confirmations | Through chat | No: you wait for the result |
| Realtime voice plus async agent (Alexa+, Gemini Live) | Highest: both models run | Reply at once, task 30 s to 1 min | Agent level, eventually | Depends on the agent | Yes | Yes, while the task itself stays slow |
The last row is the pattern most new assistants follow. The realtime model makes the assistant conversational and lets you interrupt it, and the agent behind it still sets how long the work takes. It talks like a person and acts at agent speed.
Accuracy for device control has mostly been bought with time. A model that thinks more, calls more tools and checks its work makes fewer mistakes, and each of those steps costs seconds. The open question for the next generation is how to get agent-level understanding inside the streaming pipeline's time budget, and which jobs should not try.
What comes next
Faster inference hardware and better small models will shorten the agent loop. The bigger gains are outside the model:
- Clean device data. When the assistant already knows rooms, groups, device types and which plug powers which lamp, it does not need extra steps to work it out.
- One consistent control API. Every brand behaves differently. A single model of devices removes guesswork.
- Better jobs for agents. Agents fit tasks that take minutes anyway, like watching a camera for a delivery. Our post on the AI smart home covers those jobs.
Port OS gives AI assistants that clean model of your home and one way to control every connected brand. To try it, read how to control your smart home with ChatGPT or Claude.
More in this series
- Why is Alexa so bad? An engineer's explanation
- Gemini vs Google Assistant for smart home: why the smarter one is slower
- AI in the smart home: what it can do today
- How to build a voice agent that controls devices
- Why AI still struggles to turn off a light
Frequently asked questions
Does Alexa listen all the time?
The Echo listens locally for the wake word. Amazon says audio is sent to the cloud only after the device detects the wake word, and the light ring shows when it is streaming.
How does Alexa control smart devices?
Alexa sends a directive, such as TurnOff, to the cloud service of the device's maker through a smart home skill. That service then talks to the device. This is why most smart home devices need their own app and account before Alexa can use them.
How does Alexa+ work differently?
Alexa+ uses large language models to understand requests and to coordinate services and devices. It can handle requests that do not fit a fixed phrase, and it can do several steps in a row. Those extra steps take more time than a single matched command.
Why is Alexa slower than it used to be?
Newer assistants use language models that do more work per request: retrieving context, choosing tools and sometimes checking results. Each step adds delay. A fixed command matched by the old system finished in about 1.3 seconds in our measurements.