ML in Production · 3 min read

Brain-to-AI EEG Tool Publishes a 40% Failure Rate Next to Its Wins

bci-mcp streams live EEG into AI assistants via MCP. Its author published a 105-of-109 'calm' result and a blink detector that fails on ~40% of windows.

By Marcus Bennett682 words

Key takeaways

  1. bci-mcp's 'calm' reading increased with eyes closed in 105 of 109 PhysioNet EEG recordings.
  2. The blink detector produced false detections in roughly 40 percent of eyes-closed windows.
  3. The tool is available on npm and PyPI with about 1,000 downloads and 20+ MCP directory listings.
  4. Inky Ganbold published both the strong and weak results rather than only the strongest finding.
Inky Ganbold’s Brain-to-AI Tool Publishes Its Failures Alongside Its Results
Inky Ganbold’s Brain-to-AI Tool Publishes Its Failures Alongside Its Results — GinaRe77 / pixabay

The open-source bci-mcp project, which streams live EEG (electroencephalography, the recording of electrical activity at the scalp) into AI assistants, published a benchmark result that most developers would have buried: its blink detector misfired on roughly 40 percent of eyes-closed windows. Inky (Enkhbold) Ganbold, the hardware and embedded-systems engineer behind the tool, put that number in the project record alongside his strongest result — a 'calm' reading that rose with eyes closed in 105 of 109 recordings from the public PhysioNet EEG Motor Movement/Imagery dataset.

The contrast is deliberate. bci-mcp connects live brain-signal data to an AI assistant such as Claude through the Model Context Protocol (MCP, a standard that lets AI assistants connect to external tools and data sources). The assistant stops responding only to conversation text and starts receiving changing input from an external sensor. That makes honest characterization of the measurements a prerequisite, not a courtesy.

What did the benchmark actually test?

Ganbold compared the system's output between eyes-open and eyes-closed conditions across all 109 participants in the PhysioNet dataset. The 'calm' measure increased with closed eyes in 105 cases — a checkable, dataset-wide result rather than a single demonstration. The blink detector told a different story: false detections appeared in roughly 40 percent of eyes-closed windows. He reported it as a limitation, not as a finished feature.

"I think the responsible thing is to show where a system works and where it breaks," Ganbold said. "If a detector fails often enough to matter, that should be visible to the person using it instead of hidden behind the strongest demo."

Why the distinction matters for EEG data

EEG signals are small, noisy, and influenced by movement and other artifacts. A measurable change at the scalp does not automatically support a claim about what a person is thinking. Ganbold kept his framing narrow: bci-mcp streams EEG data and produces testable outputs. He has been explicit that it does not read minds.

"The useful question is not whether a graph moves," Ganbold said. "The useful question is whether it moves consistently for the reason you think it does, and whether you have tested enough examples to know where that interpretation stops being reliable."

The blink-detector number exists precisely to mark that boundary. Without it, a weak detector could inherit credibility from the stronger 'calm' result sitting next to it in the same project.

How much traction does the tool have?

bci-mcp is distributed through npm and PyPI and has accumulated about 1,000 downloads, plus listings in more than 20 MCP directories. Ganbold treats adoption and technical validity as separate questions. Download counts show that developers are finding the tool; they say nothing about whether each detector performs reliably.

"I would rather publish a limitation that makes the project look less impressive than let somebody build on a capability that is not actually there," Ganbold said. "If the evidence improves later, then the claim can improve with it."

What should data teams take from this?

The project sits at the intersection of two developing layers: consumer EEG hardware making non-invasive brain-signal collection cheaper, and MCP giving assistants a standardized external-input channel. For teams evaluating any such integration, the operating lessons are concrete:

  • Check the dataset, not the demo. A result replicated across 105 of 109 participants carries weight a single successful screenshot does not.
  • Read the failure rates before the feature list. A 40 percent false-detection rate on a defined condition is a signal to test or replace, not to ship.
  • Do not let adjacent features transfer credibility. Each capability should stand on evidence collected for that capability, which is how Ganbold, coming from embedded systems where measured performance is separated from expectation, reports his results.

The benchmark may prove more durable than the tool itself. As consumer neurotechnology and AI tooling converge, a 105-of-109 success and a 40 percent failure rate published in the same document give downstream developers exactly what they need: a line between demonstrated behavior and unfinished work.

linkedin.com (Original)

Price tags

Share this article:

More from Marcus Bennett

Marcus Bennett

Show full bio

News editor covering people, launches and day-to-day practice in Analytics Digest is a professional daily at Analytics Digest.

13 articles

Read also

« Previous article