Skip to content
Port OS

Grok vs ChatGPT vs Claude: which AI runs a smart home best? (2026 test)

Published November 3, 2026 · By Port OS Team

I've been asking Grok to turn off my lights. Then I asked ChatGPT and Claude the same things, in the same house, through the same Port OS MCP server, and timed every request from pressing send to the answer.

Each assistant got the same tasks:

  1. Turn off the living room lights.
  2. Turn off the kitchen light.
  3. Start the robot vacuum (a Roborock S8 MaxV Ultra).
  4. Play a YouTube video on the living room TV.

All three used the same server, https://api.portoperatingsystem.com/mcp, and the same Port OS account with the same devices. The only thing that changed was the assistant.

Results

TaskGrokChatGPTClaude
Living room lights off✗ about 1 min, asked which light✗ 29 s, asked which light✗ under 45 s, asked which light and named a likely switch
Kitchen light off✓ 9 s✓ 8 s, after one connection question✓ 11 s
Start the vacuum✓ 9 s✓ 15 s✓ 15 s
YouTube on the TV✓ 18 s✓ 23 s✓ 29 s, checked the TV screen

Times run from send to the reply, measured once per task on November 3, 2026, with Grok in Fast mode and Claude on Opus 5.5. One run each is a snapshot, not a benchmark. A Meta AI run will be added to this page.

Short version: once the device had a clear name, Grok was the fastest, at 9 seconds for a light or the vacuum. ChatGPT gave up fastest on an ambiguous request. Claude was the slowest, but it was the only one that looked at the TV screen to check what was playing, and it told me when a device had only queued a command rather than confirmed it.

The living room trick question

None of the three could turn off the living room lights, and that is the right answer. This house has no light assigned to a room called "living room". The living room exists in Port OS, but only the TV, the speakers and the vacuum's map use it.

Grok ran about a dozen Port OS lookups over a minute: rooms, lights, switches, then a control request that matched nothing. Then it said so plainly, listed the rooms that do have lights, and asked which one I meant. ChatGPT did the same thing in 29 seconds, because it stopped searching after the first empty result.

Claude gave the most useful version of the same answer. It explained what "living room" does contain, pointed out an unassigned SwitchBot switch that might control the lights, and warned that the "All Hue Lights" group would turn off every light in the house.

An assistant that guessed would have turned off the wrong light. If you want "living room lights" to work, put the lights in that room in your smart home app. Room names are what the assistant goes by. We wrote more about this in why AI still struggles to turn off a light.

Lights and the robot vacuum

Named directly, the kitchen light took Grok 9 seconds, ChatGPT 8 and Claude 11. The vacuum took Grok 9 seconds and ChatGPT and Claude 15 each. After each run the vacuum was sent back to its dock.

Claude was the most careful about what it knew. It said the kitchen light command was queued, not yet confirmed, and that it had left two other kitchen lights alone.

YouTube on the TV

All three found the Living Room TV (an Android TV) and started a lofi stream. Grok took 18 seconds and mentioned that an ad was playing first. ChatGPT took 23 seconds. Claude took 29 seconds because it also took a snapshot of the TV screen to check, and reported the ad with 2 seconds left before it could be skipped.

How to connect Grok to your smart home

Grok supports MCP through custom connectors on grok.com.

  1. Sign in to Port OS and add your devices, for example Philips Hue, Roborock or an Android TV.
  2. In Grok, open Connectors and choose New Connector.
  3. Enter the server URL https://api.portoperatingsystem.com/mcp and complete the Port OS sign-in.
  4. Start a new chat and try something you can undo: "Using Port OS, turn off the kitchen light."

For ChatGPT and Claude, see how to control your smart home with ChatGPT or Claude.

FAQ

Does Grok support MCP?

Yes. Grok can use remote MCP servers as custom connectors on grok.com. That is how this test connected Grok to Port OS.

Is Grok free for smart home control?

The account in this test did not have Grok Premium (Grok was showing an upgrade prompt for it), and the custom connector worked. xAI can change what each plan includes, so check the connector screen on your own account.

Which AI was fastest at smart home control?

In this test, Grok, once the device was named: 9 seconds for a light and the vacuum, and 18 for YouTube. ChatGPT was fastest when the request was ambiguous.

Grok vs Claude: which is better for a smart home?

Grok was faster on every named task. Claude was slower but more careful: it checked the TV screen, separated queued commands from confirmed ones, and gave the most specific follow-up when a room had no lights.

Why did every assistant ask which light I meant?

The room names in your smart home app decide what "living room" means. If no light is in that room, a careful assistant asks before acting.

All posts