University of Illinois at
Abstract
Large Language Model (LLM) agents, which integrate planning, memory,
reflection, and tool-use modules, have shown promise in solving complex,
multi-step tasks. Yet their sophisticated architectures amplify vulnerability
to cascading failures, where a single root-cause error propagates through
subsequent decisions, leading to task failure. Current systems lack a framework
that can comprehensively understand agent error in a modular and systemic way,
and therefore fail to detect these errors accordingly. We address this gap with
three contributions. First, we introduce the AgentErrorTaxonomy, a modular
classification of failure modes spanning memory, reflection, planning, action,
and system-level operations. Second, we construct AgentErrorBench, the first
dataset of systematically annotated failure trajectories from ALFWorld, GAIA,
and WebShop, grounding error analysis in real-world agent rollouts. Third, we
propose AgentDebug, a debugging framework that isolates root-cause failures and
provides corrective feedback, enabling agents to recover and iteratively
improve. Experiments on AgentErrorBench show that AgentDebug achieves 24%
higher all-correct accuracy and 17% higher step accuracy compared to the
strongest baseline. Beyond detection, the targeted feedback generated by
AgentDebug enables LLM agents to iteratively recover from failures, yielding up
to 26% relative improvements in task success across ALFWorld, GAIA, and
WebShop. These results establish principled debugging as a pathway to more
reliable and adaptive LLM agents. The code and data will be available at
https://github.com/ulab-uiuc/AgentDebug
Abstract
Recent advances in large language models (LLMs) have enabled a new class of
AI agents that automate multiple stages of the data science workflow by
integrating planning, tool use, and multimodal reasoning across text, code,
tables, and visuals. This survey presents the first comprehensive,
lifecycle-aligned taxonomy of data science agents, systematically analyzing and
mapping forty-five systems onto the six stages of the end-to-end data science
process: business understanding and data acquisition, exploratory analysis and
visualization, feature engineering, model building and selection,
interpretation and explanation, and deployment and monitoring. In addition to
lifecycle coverage, we annotate each agent along five cross-cutting design
dimensions: reasoning and planning style, modality integration, tool
orchestration depth, learning and alignment methods, and trust, safety, and
governance mechanisms. Beyond classification, we provide a critical synthesis
of agent capabilities, highlight strengths and limitations at each stage, and
review emerging benchmarks and evaluation practices. Our analysis identifies
three key trends: most systems emphasize exploratory analysis, visualization,
and modeling while neglecting business understanding, deployment, and
monitoring; multimodal reasoning and tool orchestration remain unresolved
challenges; and over 90% lack explicit trust and safety mechanisms. We conclude
by outlining open challenges in alignment stability, explainability,
governance, and robust evaluation frameworks, and propose future research
directions to guide the development of robust, trustworthy, low-latency,
transparent, and broadly accessible data science agents.