BI Platforms · 4 min read
Agentic BI Fails 79% of Real Workflows, Survey Finds
A 68-study survey under review at IEEE IT Professional finds top agents complete 14–21% of realistic workflows, with trust failures to match. Vendors sell them regardless.
Key takeaways
- Spider 2.0 (632 workflows): best agent completes ~21%; DABStep hard tasks: 14.55%
- MAST study of 1,642 traces reports 41–86.7% multi-agent failure rates
- PrivacyLens: 90–95% accuracy on privacy questions, but 25–40% of actions leak sensitive data
- Open Semantic Interchange spec finalized January 2026, now incubating at Apache as Ossie
- Over 90% of 45 surveyed data-science agents lack explicit trust and safety mechanisms

The best enterprise analytics agents complete roughly 21 percent of realistic workflows on the Spider 2.0 benchmark — and the industry is selling them anyway. That number, drawn from 632 authentic professional data-analysis tasks, is the hardest fact in a sweeping survey of 68 studies and systems now under review at IEEE IT Professional, titled "Agentic AI Meets BI: Architectures, Evaluation, and Deployment for Analytics Agents."
The author's conclusion is blunt: the products are real, the capability claims are not. Every major BI vendor now ships an agent — Microsoft Copilot in Power BI plus build-your-own Fabric Data Agents, Salesforce's Tableau rebuilt around a semantics layer, ThoughtSpot's Spotter replacing the old Sage interface, plus agents from Sigma and Hex. The gap between demo and production, the survey argues, is a measurement problem, a semantics problem, and a trust problem. In that order.
How bad are the workflow numbers?
DABStep, a benchmark of more than 450 multi-step tasks from a real financial analytics platform, is harsher than Spider 2.0: the best agent solves only 14.55 percent of its hard tasks. Because DABStep scores factoid answers with automatic checks and no LLM judge, a fluent answer can't talk its way into a pass.
Multi-agent architectures — the ones behind most vendor marketing — fare no better. The MAST study, built on 1,642 expert-annotated traces, reports failure rates between 41 and 86.7 percent. Failures concentrate in specification and coordination breakdowns, not in undersized models. Adding agents multiplies coordination failures and token costs faster than it adds capability.
Architecturally, the survey finds four recurring patterns:
- Planner–executor loops with an optional critic
- Role-specialized pipelines passing structured artifacts
- Graph-orchestrated swarms
- Search-augmented planners exploring multiple solution paths
No pattern dominates. The best-evaluated systems share discipline — explicit coordination and aggressive verification of intermediate results — rather than agent count.
Why does 'revenue' break the agents?
Mature BI practice queries a governed semantic layer, not raw tables, where "revenue" has exactly one certified definition. Two dashboards disagreeing on revenue is a trust-destroying event that takes quarters to repair. An agent that quietly invents a plausible KPI definition produces output that looks authoritative and is wrong in precisely the way enterprises fear most.
The serious vendors get this: Tableau grounds agents in Tableau Semantics, and Microsoft's Fabric data agents answer from Power BI semantic models. The Open Semantic Interchange — launched by Snowflake, Salesforce, dbt Labs and others in 2025, finalized as a specification in January 2026, now incubating at Apache as Ossie — standardizes metric exchange between tools. It does not yet standardize the agent side: how an agent consumes a certified definition at query time, proves a reported number came from it, or resolves conflicting governed definitions. Most deployments still wire that grounding by hand, platform by platform.
Can agents be trusted with access rules?
The trust evidence is worse than the accuracy evidence. Rahman and colleagues, surveying 45 data-science agents, found over 90 percent lack explicit trust and safety mechanisms. ToolEmu, covering 144 high-stakes tool-use cases, found even the safest agent fails 23.9 percent of cases, including private-data leaks and financial-loss scenarios.
PrivacyLens is the instructive one. Across 493 scenarios, models answered privacy-appropriateness questions with 90 to 95 percent accuracy — then leaked the same sensitive data in 25 to 40 percent of agent actions. Knowing the policy is not following it. Prompt injection, the OWASP Top 10's LLM01 entry, is a design condition for agents that ingest untrusted spreadsheets, logs and documents.
Evaluation, meanwhile, still can't grade insight. InsightBench plants known insights in 100 business datasets and checks whether agents recover them — a proxy, not the thing. A credible harness would pair deterministic checks with blinded human judgment and inter-rater agreement. None exists as a validated instrument.
What should a data team do?
The survey's buying checklist: demand Spider 2.0- and DABStep-style completion rates, not single-query accuracy; require certified metric consumption and per-number lineage; enforce privacy at runtime through row-level access controls rather than prompts; and apply NIST AI RMF and OWASP frameworks to analytics agents specifically.
The industry has decided what it wants. The benchmarks say it doesn't reliably work yet.
Google News: Business Intelligence (Source)
More from Grace Kim
Show full bio
Market editor covering companies, deals and strategy in Analytics Digest is a professional daily at Analytics Digest.
18 articles