Based upon the autonomous driving dataset Argoverse 1.1, the authors build their dataset with high-frame-rate annotations for streaming evaluation that they name Argoverse-HD (High-frame-rate Detection). Despite being created for streaming evaluation, Argoverse-HD can also be used for study on image/video object detection, multi-object tracking, and forecasting.

Motivation

Embodied perception entails an autonomous agent’s capacity to perceive and respond to its environment effectively. An essential factor influencing the agent’s responsiveness is its reaction time. In practical scenarios like self-driving vehicles or augmented/virtual reality (AR/VR), achieving reaction times comparable to humans, typically around 200 milliseconds (ms) for visual stimuli, is crucial. In these contexts, the deployment of low-latency algorithms is imperative to ensure safe operation or provide a truly immersive experience. Historically, the computer vision community has not prioritized algorithmic latency, leading to the development of a diverse range of techniques for robotic vision, often explored in offline settings. However, vision-for-online-perception introduces distinct latency demands; by the time an algorithm completes processing a frame, say, after 200ms, the surrounding environment has already changed. Consequently, perception must inherently incorporate predictive elements to anticipate future events—an aspect mirrored in human vision, such as when a baseball player predicts the trajectory of a fast ball. Thus, the realm of streaming perception holds significance for researchers across the broader computer vision domain.

Latency is inevitable in a real-world perception system. The system takes a snapshot of the world at t1 (the car is at location A), and when the algorithm finishes processing this observation, the surrounding world has already changed at t2 (the car is now at location B, and thus there is a mismatch between prediction A and ground truth B).

Dataset description

The authors extend the publicly available video dataset Argoverse 1.1 with their annotations for streaming evaluation, which was named Argoverse-HD (High-framerate Detection). It contains diverse urban outdoor scenes from two US cities. The authors select Argoverse for its embodied setting (autonomous driving) and its high frame rate sensor data (30 FPS). We focus on the task of 2D object detection for our streaming evaluation. While Argoverse has multiple sensors, they only use the center RGB camera for simplicity. The authors collect annotations since the dataset does not provide dense 2D annotations. For the annotations, the authors follow MS COCO class definitions and format. They use only a subset of 8 classes (from 80 MS COCO classes) that are directly relevant to autonomous driving: person, bicycle, car, motorcycle, bus, truck, traffic light, and stop sign. This definition allows them to evaluate off-the-shelf models trained on MS COCO. No training is involved in the following experiments unless otherwise specified. All numbers are computed on the validation set, which contains 24 videos ranging from 15–30 seconds each (the total number of frames is 15k).

Comparison between Argoverse-HD and MS COCO. Top shows an example image from Argoverse-HD, overlaid with our dense 2D annotation (at 30 FPS). Bottom presents results of Mask R-CNN (ResNet 50) evaluated on the two datasets.