Problem

Existing AI evaluation primarily focuses on model accuracy, which is insufficient for real-world human-AI collaboration. Failures often stem from miscalibrated human reliance—overusing AI when it's wrong or underusing it when helpful. Current methods fail to capture how human-AI teams prepare for safe, effective collaboration, leading to persistent issues after deployment. Evaluations often rely on proxies like self-reported trust or explanation fidelity, which poorly predict actual reliance behavior and can obscure critical safety concerns. This mismatch between evaluation practices and real-world deployment needs to be addressed.

Method

The paper proposes a measurement framework centered on human-AI team readiness, rather than just model performance. It introduces a four-part taxonomy of evaluation metrics: outcome quality, reliance behavior, safety and harm signals, and learning over time. These metrics are linked to the Understand–Control–Improve (U–C–I) lifecycle, which describes how users learn to collaborate with AI (Understand model behavior, Control reliance, Improve strategies). The framework emphasizes operationalizing evaluation through observable interaction traces (e.g., acceptance/override patterns, error recovery actions) rather than inferred attitudes or model properties. This trace-based approach enables assessment of calibration, error recovery, and governance in deployment-relevant contexts.

Results

This framework enables a more comprehensive and deployment-relevant assessment of human-AI collaboration. By using observable interaction traces, it allows for the measurement of key aspects like calibrated reliance (e.g., accept-on-wrong, changed-to-wrong, reliance slope), safety signals (e.g., AI-induced harm, near-misses, governance-in-use), and the development of durable skills over time (e.g., calibration gap, retention, transfer). It moves beyond traditional accuracy and trust metrics to capture critical failure modes and operationalize accountability through observed behavior, supporting the development of comparable benchmarks and cumulative research on human-AI readiness.

Implications

The proposed framework has significant implications for advancing safer and more accountable human-AI collaboration. It encourages a shift in evaluation practices from isolated models to integrated human-AI teams, and from short-term performance to long-term readiness, calibration, and governance. By defining a clear measurement and benchmarking agenda, it aims to foster cumulative science in human-AI interaction, leading to more robust evaluation protocols and shared measurement standards. This shift is crucial for addressing real-world deployment challenges and ensuring that AI systems truly enhance human decision-making.