The authors constructed the first publicly available dense GMOT-40: Generic Multiple Object Tracking Dataset, dubbed GMOT-40, which contains 40 carefully annotated sequences evenly distributed among 10 object categories. Each sequence features multiple objects belonging to the same category, with an average of approximately 22 objects per frame. These sequences present various challenges inherent to real-world tracking scenarios, including heavy blur, occlusion, and other complicating factors.

Motivation

The field of Multiple Object Tracking (MOT) has been extensively explored within the computer vision community due to its diverse applications spanning robotics, surveillance, autonomous driving, and cell tracking, among others. Recent advancements in MOT owe much to the refinement of key components such as detection, single object tracking, and association. Additionally, the proliferation of MOT benchmarks has played a significant role in driving progress in the field. However, despite these advancements, previous research in MOT has predominantly focused on specific object categories such as pedestrians, cars, or cells, relying heavily on pre-trained models tailored to these objects. This approach raises questions about the generalizability of existing MOT algorithms to unseen objects, which limits the applicability of MOT to new domains with sparse data for training object detectors. In contrast, Generic Multiple Object Tracking (GMOT) seeks to address these challenges by operating without prior knowledge of the objects being tracked. GMOT holds promise for a wide array of applications including video editing, animal behavior analysis, and vision-based object counting. Despite its broad potential, GMOT remains relatively underexplored, with only limited early investigations conducted thus far. Upon comparing the progress in GMOT with that in MOT, it becomes evident that there is a notable absence of GMOT benchmarks and effective deep learning-based GMOT baselines. This gap underscores the need for further research and development in GMOT, particularly in establishing standardized benchmarks and leveraging deep learning methodologies. It’s important to note that GMOT, as defined by the authors, involves tracking multiple objects belonging to a generic object class. Addressing the above issues, the authors contribute to the study of GMOT in three aspects: dataset, baseline, and evaluation.

Dataset description

The authors have developed GMOT-40, the first publicly available dense dataset for the comprehensive study of Generic Multiple Object Tracking (GMOT). GMOT-40 comprises 40 meticulously curated sequences spanning ten distinct categories, such as insects and balloons, with four sequences allocated per category. Each sequence features multiple objects belonging to the same category, with an average of approximately 22 objects per frame. To ensure accuracy and reliability, all sequences undergo manual annotation with meticulous validation and correction processes. These sequences present various challenges inherent to real-world tracking scenarios, including heavy blur, occlusion, and other complicating factors. To facilitate rigorous evaluation, a specific tracking protocol is adopted to assess different aspects of tracking algorithms. In response to the unique requirements of GMOT, the authors have devised a series of baseline tracking algorithms tailored for one-shot GMOT. These baselines are structured around a two-stage process: the first stage involves one-shot detection, adapted from the GlobalTrack algorithm, while the second stage employs target association techniques drawn from several standard Multiple Object Tracking (MOT) algorithms. In each baseline, the one-shot detection algorithm functions as the public detector. Subsequently, the authors conduct comprehensive evaluations on GMOT-40, involving classic tracking algorithms with necessary adjustments for the GMOT context. The findings of these evaluations underscore the significant potential for improvement in addressing the challenges posed by GMOT, indicating ample opportunities for further advancements in the field.

One-shot generic multiple object tracking (GMOT). (a): The input of one shot generic MOT is a single bounding box to indicate a target template in the first frame. (b): The target template is used to discover and propose all other target candidates of the same category, which is different than model-based MOT where a pretrained detector (typically class-specific) is required. (c): MOT then can be performed on the proposed candidates in either an online or offline manner. Yellow rectangles are zoomed-in local views of targets.

To tackle the issue of generalization observed in prior Multiple Object Tracking (MOT) research, GMOT-40 has been meticulously curated to feature a more comprehensive and diverse dataset. Unlike many previous datasets, which typically encompassed fewer than three categories, GMOT-40 comprises 40 sequences spanning 10 distinct categories. This substantial increase in category diversity is intended to provide a more robust testing ground for MOT algorithms. Within each category, GMOT-40 offers four sequences, carefully crafted to incorporate a wide range of variations and challenges. For instance, the “person” category within GMOT-40 encompasses not only typical instances of individuals as found in datasets like PASCAL-VOC but also introduces a novel category of individuals wearing “wingsuits.” Similarly, the “insect” category presents instances of both “ant” and “bee,” which are not commonly encountered in datasets such as MS-COCO or PASCAL-VOC. This deliberate inclusion of unseen object types within familiar categories serves to enhance the dataset’s capacity for testing the generalization capabilities of MOT algorithms across a broader spectrum of scenarios and object types.

Samples from each category of GMOT-40.

During the selection of sequences for GMOT-40, the authors have placed particular emphasis on including a diverse range of real-world challenges commonly encountered in tracking scenarios. These challenges encompass factors such as occlusion, targets entering or exiting the scene, rapid motion, and motion blur. Furthermore, the density of targets varies across sequences, spanning from 3 to 100 targets per frame, with an average density of approximately 26 targets per frame. These deliberate choices ensure that GMOT-40 encompasses a broad spectrum of scenarios, offering researchers a comprehensive dataset for evaluating tracking algorithms. To ensure the highest quality annotations, each frame within the sequences undergoes meticulous manual annotation to guarantee accuracy. This annotation process involves initial labeling followed by thorough validation and revision to rectify any discrepancies or errors. While additional sequences could further enhance the dataset’s utility, the significant manual effort required for annotation may delay the timely release of the dataset. It is worth noting that GMOT-40 represents a significant advancement over previously available GMOT datasets, offering comprehensive improvements and promising to catalyze further advancements in GMOT research in the future.

Comparison of densely annotated data used in GMOT studies.

Data collection

The authors began by identifying 10 object categories likely to exhibit dense and crowded behavior. When choosing video sequences, they stipulated that at least 80% of the frames in a sequence should feature more than 10 targets. While most targets within the same category shared a similar appearance, some exhibited variations in appearance, mirroring real-world scenarios more accurately. They set a minimum sequence length of 100 frames. Once the classes and criteria were established, the authors commenced a search on YouTube for potential candidate videos. Initially, around 1000 sequences were identified as candidates. Following careful examination, 40 sequences were selected for their higher quality and greater challenge. However, it’s important to note that these 40 sequences were not yet ready for annotation. Some sequences contained significant segments unrelated to the task. For instance, in the balloon category, there were sections at the beginning and end focusing on stage performances or festival crowds, which needed to be removed. Consequently, the videos were meticulously edited, and only the best clips with a minimum length of 100 frames were chosen. Ultimately, GMOT-40 boasts an average of 50.65 trajectories per sequence. The entire dataset comprises 9,643 frames, with each sequence averaging 240 frames in length. Approximately 85.28% of the frames feature more than 10 targets. The frames per second (FPS) range from 24 to 30, and the resolution varies from 480p to 1080p.

Data annotation

The annotation format in GMOT follows that of MOT15, with one key distinction: there is no “out-of-view” value, meaning that all bounding boxes in the ground truth file are considered for evaluation protocols. Moreover, only targets within the same category are annotated. For instance, in the “stock” category, only the wolf would be annotated if the initial bounding box indicates that it is the object of interest. Additionally, targets within the same category are treated equally, such as red and white balloons. Ensuring a high-quality GMOT dataset involves manual labeling, double-checking, and error correction. To achieve this, a team of experts, including Ph.D. students, is involved in the annotation process. Each video is initially assigned to a labeler to determine the group of interest. An expert then reviews the target group to ensure it meets the requirements. Once approved, the labeler proceeds with the annotation. The completed annotation undergoes another round of review and potential revision by the experts.

GMOT-40 encompasses a broader range of scenarios and attributes compared to previous GMOT datasets utilized in research. For instance, classes such as person, ball, and insect exhibit characteristics like motion blur and fast motion. Additionally, the appearance of objects in the boat category is significantly influenced by the viewpoint. Moreover, challenges such as low resolution and camera motion are present in the “ball” and livestock categories, respectively. The abbreviation of attributes have the following meaning: CM – camera motion; ROT – target rotation; DEF – target deforms in the tracking; VC – significant viewpoint change that affects the appearance of target; MB – target is blurred due to camera or target motion; FM – fast motion of the targets with displacements larger than the bounding box; LR – target bounding box is smaller than 1024 pixel for at least 30% of the targets in the whole sequences. Although some of the attributes above are present in previous studies of GMOT, yet GMOT-40 is the most comprehensive one, since it is collected from various natural scenes. These miscellaneous attributes of GMOT-40 can help the community to evaluate their trackers from multiple aspects.

Number of sequences for different attributes in the GMOT-40.

Note: the authors did not provide information in the dataset about the belonging of the specified attributes to a specific sequence.