AI Insights

• The task granularity is flexible, and every reasoning chain must start from the raw data or a logically prior step. (ML: 0.97)👍👎
• The instructions may be too complex or detailed for some users, potentially leading to confusion. (ML: 0.95)👍👎
• The provided Jupyter Notebook content is a template for generating data science questions based on an answered notebook. (ML: 0.95)👍👎
• QRA: Question-Reasoning-Answer triplet JSON: JavaScript Object Notation Generating high-quality data science questions based on an answered notebook requires careful analysis and adherence to specific guidelines. (ML: 0.94)👍👎
• The output format requires a valid JSON object with specific keys such as 'data_type', 'domain', 'task_type', 'language', 'question', 'reasoning', 'answer', 'best_score (Optional)', and 'confidence'. (ML: 0.89)👍👎
• The final output must be a valid JSON object with the specified structure. (ML: 0.82)👍👎
• The instructions provide detailed guidelines for generating QRA triplets, including the importance of not mentioning the notebook and ensuring diversity across task types. (ML: 0.79)👍👎
• The output format must conform to a valid JSON object with specified keys, ensuring that the generated QRA triplets are accurate and comprehensive. (ML: 0.77)👍👎

Abstract
Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., vision and text). DSAEval incorporates three distinctive features: (1) Multimodal Environment Perception, which enables agents to interpret observations from multiple modalities including text and vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across reasoning, code, and results. We systematically evaluate 11 advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance, GPT-5.2 is the most efficient, and MiMo-V2-Flash is the most cost-effective. We further demonstrate that multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04% to 11.30%. Overall, while current data science agents perform well on structured data and routine data anlysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions to advance the development of data science agents.

Why we are recommending this paper?
Due to your Interest in Managing teams of data scientists

AI Insights

• They also note that the results may not generalize to other domains or tasks beyond text-to-SQL. (ML: 0.99)👍👎
• The authors acknowledge the limitations of their study, including the small size of the dataset and the potential bias in the persona modeling prompt. (ML: 0.98)👍👎
• The authors propose a framework for generating high-quality datasets that align with business intelligence (BI) settings and evaluate the performance of various LLMs using this framework. (ML: 0.97)👍👎
• Previous studies have shown that LLMs can be effective on certain NLP tasks, but struggle with others, highlighting the need for more research in this area. (ML: 0.96)👍👎
• The proposed framework generates high-quality datasets that align with BI settings and evaluates the performance of various LLMs using this framework. (ML: 0.96)👍👎
• The text is a research paper on developing a benchmark for evaluating large language models (LLMs) on real-world text-to-SQL tasks. (ML: 0.96)👍👎
• The results show that some LLMs perform well on certain aspects of text-to-SQL, but struggle with others, highlighting the need for more research in this area. (ML: 0.95)👍👎
• Text-to-SQL: A task where a natural language question is converted into an SQL query to retrieve relevant data from a database. (ML: 0.94)👍👎
• LLMs: Large Language Models, which are artificial intelligence models that can process and generate human-like text. (ML: 0.94)👍👎
• ReAct paradigm: A prompt-based agentic framework for evaluating LLMs on complex tasks such as text-to-SQL. (ML: 0.93)👍👎

Abstract
Evaluating Text-to-SQL agents in private business intelligence (BI) settings is challenging due to the scarcity of realistic, domain-specific data. While synthetic evaluation data offers a scalable solution, existing generation methods fail to capture business realism--whether questions reflect realistic business logic and workflows. We propose a Business Logic-Driven Data Synthesis framework that generates data grounded in business personas, work scenarios, and workflows. In addition, we improve the data quality by imposing a business reasoning complexity control strategy that diversifies the analytical reasoning steps required to answer the questions. Experiments on a production-scale Salesforce database show that our synthesized data achieves high business realism (98.44%), substantially outperforming OmniSQL (+19.5%) and SQL-Factory (+54.7%), while maintaining strong question-SQL alignment (98.59%). Our synthetic data also reveals that state-of-the-art Text-to-SQL models still have significant performance gaps, achieving only 42.86% execution accuracy on the most complex business queries.

Why we are recommending this paper?
Due to your Interest in Data Science Engineering