EVAL Engine
Blog

ChromBot: what we built on top of open-source XiaoZhi

ChromBot is a small ESP32-S3 voice companion built by EVAL Engine and Chromia. It runs on XiaoZhi, the open-source voice assistant firmware from 78/xiaozhi-esp32, and the xiaozhi-esp32-server backend from xinnan-tech. Upstream provides the hard parts: wake word detection, Opus audio over WebSocket, the streaming ASR, LLM and TTS loop, and device-side MCP. This post covers what we added.

Firmware

Our fork's main tracks upstream. Our work lives on branches:

  • A chrombot board definition with an ES8311 codec, a 240x240 SPI display, camera pins, a UART link and LED light modes, plus a GIF emoji display.
  • BLE Wi-Fi provisioning on NimBLE. GATT characteristics cover SSID, password, network scan, saved networks, PIN authentication, connection verification and device status, with specific error codes for timeouts, wrong credentials, missing networks and DHCP failures. A Web Bluetooth page drives the flow from a browser.
  • An idle screen that shows a token price from CoinGecko when the device is idle and online, and a device-side MCP tool, self.show_price_display, so the user can turn it on or off by voice.
  • Fixes for a Wi-Fi netif double-initialization crash and an idle screen crash, plus flashing and MAC listing scripts.

Server

Our server fork adds wallet-based accounts through Privy, device bind and unbind keyed by MAC address, usage and token tracking, telemetry and LangSmith tracing. New plugins cover CoinGecko prices (with number formatting tuned for speech), crypto news, Elfa social mentions and web search. Logs and prompts were translated from Chinese to English.

An interaction summary endpoint aggregates a user's activity over a number of days, can generate a short LLM summary, and on POST writes that summary to Chromia through a Rell contract using the Postchain Python client.

MCP tools

xiaozhi-mcp is a FastMCP server that loads plugin modules and registers every function prefixed mcp_ as a tool. Plugins include CoinGecko, Elfa, Kaito Yaps, a calculator, Chromia FT4 balance lookups and Chromia docs search. It runs over stdio for desktop MCP clients, or through a pipe script that connects to a XiaoZhi MCP endpoint.

We also run a fork of xiaozhi-mcphub, itself built on MCPHub, for managing multiple XiaoZhi endpoints. Our changes there are limited to a cookbook and deployment config.

Website

chromia-robotics is the ChromBot site: Next.js with English, Korean and Chinese locales, an MDX blog with meetup recaps, a setup guide and FAQ, Supabase auth, and Polar for checkout and billing.