Problem

Medical training relies heavily on hands-on clinical practice, such as residency, to convert theoretical knowledge into practical expertise. Clinical reasoning involves complex, multi-turn interactions with patients, requiring the elicitation of history, the refinement of diagnostic hypotheses, and the management of treatment plans under uncertainty. While large language models (LLMs) have achieved impressive results on static medical assessments, there is a significant gap in methods for optimizing the entire sequence of clinical decisions in dynamic, conversational settings. Common issues include premature closure—a tendency to conclude information gathering too early—and the underdeveloped nature of existing multi-turn reinforcement learning (RL) approaches for clinical dialogue, which often suffer from short interaction horizons and limited action spaces. This paper aims to overcome these limitations by introducing a new RL method for training AI agents in comprehensive, multi-turn simulated clinical encounters.

Method

The authors present ResidencyRL, an online multi-turn reinforcement learning framework designed to train clinical AI agents. The core idea is to foster clinical mastery through extensive practice in simulated environments, similar to how human physicians gain experience. The system pairs a policy agent, initialized with a Gemini 3.5 Flash model, with an LLM-based patient simulator capable of complex and adversarial behaviors. Training involves a diverse curriculum that includes routine telehealth consultations, targeted history-taking scenarios designed to uncover hidden information, and adversarial safety cases to test robustness against challenging patient behaviors. The agents engage in multi-turn clinical encounters, scaling up to 60 dialogue turns and 8 structured tool calls for tasks like documenting, diagnosing, and managing patients.

Training is guided by a structured, hierarchical reward system, implemented by an LLM-as-judge, which optimizes across six clinical dimensions: diagnostic accuracy, management quality, screening completeness, patient-centered communication, clinical documentation, and conversational style. This reward also includes critical safety penalty flags. A sophisticated scenario generation pipeline creates medically rigorous cases by sampling demographic and behavioral profiles, using LLMs for condition-targeted generation, verifying clinical coherence, and deduplicating scenarios. The patient simulator adapts its communication style based on health literacy and information asymmetry, with adversarial specializations for safety scenarios. The RL algorithm used for policy updates is Group Relative Policy Optimization (GRPO), iteratively improving the agent based on live environment feedback.

Results

The ResidencyRL-trained agent demonstrated significant improvements across various evaluation benchmarks. In held-out in-domain scenarios, diagnostic accuracy improved by 7.0% under adversarial conditions (from 81.0% to 88.0%). Management quality scores increased from 3.98 to 4.52, and patient-centered communication showed notable gains, particularly in responding to emotions (from 2.63 to 3.06). Screening completeness, especially for social and lifestyle history, saw substantial improvement, indicating the agent's enhanced ability to inquire thoroughly. Crucially, the agent reduced missed red flag rates by 31% (from 45.5% to 31.5%) and missed critical questions by 22% (from 65.5% to 43.5%) in adversarial settings, showcasing a rigorous mitigation of premature closure.

Beyond in-domain evaluations, the agent's capabilities generalized effectively. On the AMIE Mx multi-visit benchmark, it outperformed the base model in all six evaluation categories, with the largest improvements in management reasoning (80.1% to 88.4%) and patient communication (83.7% to 92.2%). When evaluated on expert-curated specialist oncology cases, a domain not seen during training, the ResidencyRL agent showed significantly better performance in completeness, clinical accuracy, and actionability of its management plans. On external benchmarks like AgentClinic and CRAFT-MD, designed to assess dynamic reasoning and premature closure, the agent showed consistent directional improvements in diagnostic accuracy, particularly in free-response formats requiring active information gathering and diagnostic construction.

Blinded evaluations by board-certified clinicians provided strong human-grounded validation, with the ResidencyRL-trained agent preferred in 87.6% of side-by-side comparisons for overall clinical impression. Clinicians specifically highlighted superior completeness of information gathering (90.7% win rate) and management plan appropriateness (75.3% win rate). The trained model also maintained or improved safety and communication quality without introducing factual degradation.

Implications

The findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn reinforcement learning in simulated environments, yielding robust and generalizable AI capabilities. This approach allows AI agents to develop procedural competencies—such as efficient information gathering, knowing when to probe further, and managing clinical uncertainty—which transfer effectively across diverse evaluation frameworks and previously unseen medical specialties. The use of capability-specific scenarios, including targeted history-taking and adversarial safety conditions, is shown to steer the development of critical clinical skills, significantly reducing diagnostic pitfalls like premature closure and improving overall patient safety.

The study highlights that these gains compound with expert-designed agentic scaffolding, and the trained agent is consistently preferred by practicing clinicians. While there are limitations, such as the current confinement to text-based telehealth, single-visit encounters, and the absence of multimodal sensory inputs, the core principle holds: a frontier model already possessing broad medical knowledge can be made into a meaningfully more competent and helpful clinician through extensive reinforcement learning in high-fidelity simulation. Future work will focus on expanding clinical coverage, enhancing simulation fidelity (e.g., multimodal data, physiological effects), and extending the temporal horizon to multi-visit longitudinal care.