Problem

The advancement of autonomous vehicles (AVs) critically depends on the availability of extensive, diverse, and precisely annotated datasets. However, many existing public datasets do not accurately reflect specific driving conditions, such as those found in Poland, which differ significantly in infrastructure, signage, weather, and driver behavior. Creating such tailored datasets, particularly for multimodal data like LiDAR point clouds and camera images, faces a major hurdle: manual annotation. This process is inherently labor-intensive, costly, and difficult to keep consistent and accurate across various sensors and annotators, often becoming the most time-consuming and expensive phase in dataset production.

Method

This report details the design and implementation of a semi-automated data annotation pipeline developed for the DARTS project, which aims to create a large-scale, multimodal dataset tailored for Polish driving conditions. The system adopts a human-in-the-loop approach, integrating AI models with human verification to enhance efficiency. Its architecture is coordinated by a central DARTS system, connecting human annotators, administrative tools, and the external Segments.ai platform, with workflows orchestrated by Apache Airflow.

Key modules within the pipeline include: the Annotation Generator, which uses a fine-tuned DSVT model (based on OpenPCDet) for automated 3D object detection from LiDAR data; the Anonymization Module, utilizing RetinaFace for face detection and a YOLO model for license plates, applying Gaussian blur for privacy compliance; the Data Preprocessing Module, converting raw sensor data into a standardized DARTS format (an extension of nuScenes) and performing integrity checks; the Database Module, a PostgreSQL-based system serving as the central source of truth for metadata and annotations; the Multiple Object Tracking (MOT) Module, adapted to ensure object continuity across frames, refining positions, and detecting stationary objects; and the Segments Toolkit Module, facilitating seamless integration and synchronization with the Segments.ai annotation platform. The DARTS Utils Module provides general utilities, including core model classes, annotation difference analysis, Rerun visualization integration, and performance metrics calculation.

The workflow begins with raw data recording, followed by preprocessing and integrity checks via Airflow. Automated preliminary annotations are generated by AI models and uploaded, along with processed data, to Segments.ai for human review. After human verification, correction, and quality control, the validated annotations are imported back into the DARTS PostgreSQL database, completing the annotation cycle for inclusion in the DARTS dataset.

Results

To accurately assess the practical utility of automated annotation in human-in-the-loop systems, the paper introduces a novel metric: the Correction Acceleration Ratio (CAR). Unlike traditional perception metrics such as Average Precision (AP), CAR directly quantifies the time savings achieved through pre-annotation by accounting for various error types (false positives, false negatives, positional, and classification errors) and their estimated average correction times. A manual study using modified nuScenes data was conducted to estimate these correction times, revealing that false negatives typically require the longest correction (around 23 seconds), while false positives and classification errors are the fastest (1-2 seconds).

Experiments evaluating state-of-the-art 3D object detectors, including DSVT (which was adopted for the pipeline), on public datasets like Zenseact, nuScenes, Waymo, and KITTI, demonstrated significant efficiency gains. The results showed that integrating these detectors substantially reduces manual annotation time. Notably, on the Zenseact dataset—which closely resembles DARTS project data—the highest observed CAR value was 0.93. This indicates a potential reduction of up to 93% in manual annotation effort compared to a fully manual process, underscoring the substantial practical benefit of the semi-automated pipeline.

Implications

The developed semi-automated data annotation pipeline offers a critical solution to the challenges of creating large-scale, high-quality, and cost-effective datasets for autonomous vehicle research. By substantially reducing the time and expense associated with manual annotation while ensuring consistent and accurate labels, the system directly supports projects like DARTS in building essential datasets tailored to specific regional driving conditions, such as those in Poland.

This innovation strengthens the technological foundation for autonomous vehicle development within Poland, fostering national competencies in intelligent transportation systems. Furthermore, the introduction of the Correction Acceleration Ratio (CAR) provides a more practical and intuitive metric for evaluating the real-world efficiency gains of AI models in annotation workflows, moving beyond traditional accuracy metrics to focus on direct productivity impacts in human-in-the-loop systems.