Problem

The paper addresses the increasing interest in Generative AI (GenAI) for research, particularly in statistical methodology. Despite models like ChatGPT showing promise in information synthesis and programming, there's a notable lack of evaluation regarding their application in statistical research tasks. The core problem explored is whether current GenAI models possess the depth and contextual understanding required to effectively assist in resource-intensive tasks such as conducting comprehensive literature reviews and accurately generating programming code from complex academic methodologies, specifically focusing on Dynamic Treatment Regime (DTR) estimation using dynamic weighted ordinary least squares (dWOLS). The authors aim to identify the strengths and limitations of GenAI in these areas.

Method

This case study utilized ChatGPT-5 and ScholarAI (Sept-Nov 2025 releases) to evaluate GenAI's capabilities across two main tasks: literature review creation and code generation. For the literature review, the models were tasked with identifying relevant sources on dWOLS methodology, summarizing papers from abstracts and full texts, and identifying research gaps. A pre-existing human-curated literature review served as a benchmark. For code generation, ChatGPT-5 was prompted to reproduce simulation studies, analyze simulated datasets using dWOLS, and extend an existing R package (DTRreg) to incorporate the Generalized dWOLS (G-dWOLS) method. Both zero-shot prompting and scenarios with uploaded papers or specified models were employed. The evaluation assessed execution, methodological correctness, and expected parameter estimates, with expert review for accuracy.

Results

For literature review tasks, GenAI models successfully identified relevant papers without fabricating entire sources, a notable improvement over past models. However, they frequently provided incorrect citation details and had a significant number of false positives and negatives when classifying paper relevance. While summaries were generally factual, they lacked contextual depth, and models struggled with consistent formatting without explicit instructions. Identifying research gaps proved challenging, with models often reiterating existing literature gaps or offering high-level, less actionable suggestions, occasionally demonstrating a lack of domain-specific insight. ChatGPT generated a draft literature review but struggled with citation accuracy and contextual understanding. For code generation, ChatGPT-5 produced executable R code quickly (1-3 prompts) but consistently introduced minor errors (syntax, undefined variables) and, more critically, incorporated incorrect methodologies (estimating equations, weight functions) that appeared functional but yielded slightly inaccurate results. The model struggled with extending existing R packages and exhibited a tendency to "agree" with incorrect user suggestions.

Implications

The study concludes that current GenAI models are not adequate substitutes for human researchers, especially where deep domain expertise and critical thinking are paramount. While GenAI can increase efficiency for tasks like initial literature searching and basic summarization, significant human oversight and verification are essential due to issues with accuracy, contextual understanding, and methodological correctness in both literature review synthesis and code generation. Researchers should treat GenAI as a supplementary tool to streamline routine tasks, emphasizing transparency and responsible use. Future research should explore more formal evaluation metrics and different prompting techniques, acknowledging the rapid evolution of these technologies. Ultimately, GenAI can make research more accessible but cannot replace the intellectual originality and expertise of a human researcher.