Problem

The practical deployment of active learning (AL) methods for real-world object detection faces significant hurdles, primarily due to high computational costs and unreliable evaluation outcomes. Developing and comparing new AL approaches typically necessitates training numerous detectors across multiple iterations, which is extremely resource-intensive. For instance, training a single object detector on large autonomous driving datasets can consume hundreds of GPU hours. Furthermore, the performance rankings of AL methods often vary considerably across different validation sets, compromising the reliability of evaluations, especially in critical applications like autonomous driving where safety is paramount. This inconsistency makes it challenging to confidently select and deploy effective AL strategies, as a method performing well on one subset might fail to generalize to others or to the true target domain.

Method

This work introduces Object-based Set Similarity (OSS), a novel metric that unifies AL training and evaluation strategies by quantifying the informativeness of training sets and the representativeness of evaluation sets. OSS effectively predicts the performance of AL methods without the need for computationally expensive detector retraining. It achieves this by measuring the similarity between a training set and the target domain (approximated by the validation set) using object-level features. The core insight is that informativeness and representativeness are linked to how closely a dataset aligns with the target distribution. The OSS metric is built upon three key components:

Firstly, Object-Centric Analysis focuses similarity calculations on object crops rather than entire images. This prioritizes relevant detection features, mirrors how detectors are evaluated (via mAP per class), and reduces computation. Secondly, a Multi-Modal Feature Representation is extracted from these object crops for each class. These features include aspect ratio (for shape), the mean of 2D discrete cosine transform (DCT) coefficients (for texture patterns), and the mean of the flattened 3D color histogram (for color distributions), providing a comprehensive visual description. Lastly, Class-Balanced Evaluation addresses inherent class imbalances in real-world data. It calculates a class-weighted multivariate Jensen-Shannon Divergence (JSD) over the empirical feature distributions, combined with a smoothed class count ratio, ensuring that each class contributes proportionally to the similarity measure. This comprehensive approach allows OSS to be model-agnostic, efficient, and directly applicable to existing AL pipelines.

Results

The proposed OSS metric was validated on three autonomous driving datasets (KITTI, BDD100K, CODA) using uncertainty-based AL methods and two detector architectures (EfficientDet, YOLOv3). Experiments demonstrated a strong positive linear correlation between OSS values and mean Average Precision (mAP) performance across different AL methods and iterations. This confirms that OSS can reliably predict the informativeness of an AL training set before costly detector training, leading to significant computational savings. For example, eliminating ineffective methods early saved up to 3,224 GPU hours per method on the BDD dataset. Furthermore, OSS improved the reliability and consistency of AL evaluations. By using OSS to identify representative subsets of validation data, the agreement in AL method rankings between different evaluation sets (measured by Kendall's tau) substantially increased. This was achieved without additional labeling or extensive computation, allowing for more robust deployment decisions even with domain shifts. The case study on uncertainty-based AL methods revealed that calibration and class balancing are crucial for uncertainty to contribute effectively to informativeness, as these strategies naturally enhance the similarity between the selected training set and the target domain. Specifically, class-balanced and calibrated entropy-based sampling consistently outperformed standard uncertainty methods, yielding up to 2% mAP improvement.

Implications

The introduction of Object-based Set Similarity (OSS) provides a practical framework that significantly streamlines the development and deployment of active learning methods in real-world object detection applications. By offering a detector-agnostic and computationally efficient way to quantify training set informativeness and evaluation set representativeness, OSS enables practitioners to eliminate ineffective AL methods early, thereby drastically reducing development costs and GPU hours. Moreover, it enhances the reliability and generalizability of AL evaluations, ensuring more consistent performance rankings even in the presence of domain shift, which is critical for safety-critical systems like autonomous driving. The findings also underscore the importance of uncertainty calibration and class balancing in improving the informativeness of selected samples in uncertainty-based AL. Ultimately, this work contributes to the broader adoption of active learning by making its development process more efficient and its evaluation more robust and trustworthy.