In the RGB-D Peoples Dataset the authors present a 3D people detection and tracking approach using RGB-D data. It combines a novel multi-cue person detector for RGB-D data with an on-line detector that learns individual target models. The two detectors are integrated into a decisional framework with a multi-hypothesis tracker that controls on-line learning through a track interpretation feedback. For on-line learning, they take a boosting approach using three types of RGB-D features and a confidence maximization search in 3D space. The approach is general in that it neither relies on background learning nor a ground plane assumption.

Motivation

Detection and tracking of individuals play pivotal roles in various robotic systems, interactive platforms, and intelligent vehicles. Cameras and range finders are popular sensors employed for this purpose, each offering unique advantages and drawbacks. However, with the emergence of affordable and increasingly reliable RGB-D sensors capable of providing both image and range data, the distinction between these modalities may become less relevant.

In the realm of robotics, researchers have tackled the challenge of people detection and tracking using range data. Early approaches focused on 2D data, employing ad-hoc classifiers to identify moving local minima in scans. However, the detection and tracking of people in 3D range data present a relatively new and underexplored area.

In contrast, the computer vision community has extensively studied the detection, tracking, and modeling of humans primarily using image data. Unlike range-based systems, the rich information provided by images facilitates the learning of target appearance models. Consequently, visual tracking systems often achieve satisfactory results with relatively simple methods, such as independent particle filters with nearest-neighbor data association.

The authors of this work contribute to advancing the state of the art in several key aspects. Firstly, they address the novel challenge of detecting and tracking people in RGB-D data. Their approach combines a pre-existing person detector with an online-learned person detector and a multi-hypothesis tracker (MHT) capable of estimating the motion state of multiple individuals in 3D space.

Additionally, the authors introduce a novel framework for integrating the two detectors and the tracker, incorporating track interpretation feedback to control learning. This framework enables the system to compensate for gaps in detections from the pre-existing detector, handle target occlusions, and prevent drift of the online detector.

Detected people in RGB-D data from dense depth data (right) and color image data (left).

Dataset description

The authors rely on a novel RGB-D person detector called Combo-HOD (Combined Histograms of Oriented Depths and Gradients). The method takes inspiration from Histogram of Oriented Gradients (HOG) and combines the HOG detector in the color image with a novel approach in the depth image called Histograms of Oriented Depths (HOD). Since RGB-D data contains both color and depth information, the Combo-HOD detector combines the two sensory cues. HOD descriptors are computed in the depth image and HOG descriptors are computed in the color image. They are fused on the level of detections via a weighted mean of the probabilities obtained by a sigmoid fitted to the SVM outputs. HOD includes a depth-informed scale-space search in which the used scales in an image are first collected and then tested for compatibility with the respective depth. This test is made particularly efficient by the use of integral tensors, an extension of integral images over several scales. This strategy dramatically reduces the number of descriptors computed in the image at improved detection rates. The output of the detector in each step is the positions and size of all targets in 3D space and the center and size of the bounding boxes in the depth images.

This dataset contains 3000+ RGB-D frames acquired in a university hall from three vertically mounted Kinect sensors. The data contains mostly upright walking and standing persons seen from different orientations and with different levels of occlusions.

The setup consisting from three vertically mounted Kinect sensors offering a joint field of view of 130◦ × 50◦ and supplying RGB-D data with a resolution of 1440 × 640 pixels at 30 Hz. They are mounted at 1.2 m height.

In order to assess and contrast various detector approaches, the authors curated a comprehensive indoor dataset capturing unscripted human behavior. This dataset was gathered in the lobby of a bustling university cafeteria during lunch hours. To prevent bias in the a priori detector towards the visual characteristics of the cafeteria lobby, additional background data was collected from a visually distinct university building. This was particularly important given that the data was acquired from stationary sensors. The dataset was meticulously annotated, including bounding boxes in 2D depth image space, the visibility status of subjects (fully visible/partially occluded), and ground truth data association for the tracks. In total, 3021 instances of individuals across 1133 frames and 31 tracks were labeled. The sensory setup comprised three vertically mounted Kinect sensors, collectively providing an expanded field of view measuring 130° × 50°. Careful measures were taken to calibrate the intrinsic and extrinsic parameters of the setup, ensuring synchronized acquisition of the three images at the desired frame rate.