The authors propose a novel multi-object tracking (MOT) dataset of marine organisms in the wild named BrackishMOT: The Brackish Multi-Object Tracking Dataset. It describes the task of obtaining the trajectories of all objects in the scene. The dataset focused on tracking schools of small fish, which is a notoriously difficult MOT task. BrackishMOT consists of 98 sequences and six different classes captured in the wild. Additionally, the authors propose a framework for creating synthetic sequences in order to expand the dataset. The framework consists of animated fish models and realistic underwater environments.

Motivation

For millennia, humans have depended on oceans as a reliable food source. However, marine ecosystems are now rapidly declining due to human activities. This poses a significant challenge for coastal communities worldwide, which heavily rely on fish as their primary food source, and for overall biodiversity. The seriousness of this issue is underscored by the United Nations’ inclusion of Life Below Water as the fourteenth Sustainable Development Goal. The growing emphasis on monitoring ocean health has placed immense pressure on marine researchers to collect data at an unprecedented rate. However, traditional methods of gathering marine data are often labor-intensive, intrusive, and not easily scalable, as they involve manually catching and measuring organisms. Therefore, it’s imperative to develop assistive solutions that optimize and scale data collection in marine environments. Over the past decade, there has been a remarkable advancement in computer vision solutions, driven by the use of powerful graphics processing units (GPUs) and the widespread adoption of deep learning algorithms. At the same time, underwater cameras have become substantially more efficient and affordable. This presents an opportunity for marine researchers to leverage both cameras and computer vision to enhance data collection. However, this integration faces a significant obstacle: the lack of marine datasets for training and evaluating computer vision models tailored to underwater environments. Tracking is a core component in marine research and can be used for multiple purposes such as counting or conducting behavioral analysis. Manually annotated datasets are critical and necessary for evaluating the performance of trackers on data from the wild. However, they are not scalable and do not necessarily generalize well to environments that are not included in the dataset.

Dataset description

The authors explore the potential of using synthetic underwater sequences to train multi-object trackers. They present a framework designed to generate synthetic sequences resembling the BrackishMOT environment. The study delves into an analysis of the impact of key factors specifically, turbidity, floating particles, and background on tracker performance.

The authors present and publish a bounding box annotated underwater multi-object tracking dataset captured in the wild named BrackishMOT, together with a synthetic framework for generating more data.

The authors suggest expanding the Brackish Dataset to encompass a Multiple Object Tracking (MOT) task. Consequently, they furnish a fresh array of ground truth annotations for each sequence, following the MOTChallenge annotation format. Furthermore, they introduce nine additional sequences concentrating on the small fish category, thereby augmenting the total sequences for the MOT task within the Brackish Dataset to 98, hereby referred to as BrackishMOT. The inclusion of the small fish category is particularly pertinent for the MOT task, as it encompasses species known for their social and schooling behaviors.

Image samples from the Brackish Dataset. In a majority of the sequences containing the small fish class, there are multiple specimens forming a school of fish.

The dataset is imbalanced with few occurrences of the shrimp, fish, and jellyfish classes. Furthermore, as the number of bounding boxes and frame occurrences are equal the authors can decipher that these three classes occur in the sequences as single objects. As the small fish class is the only class that exhibits erratic motion and appears in groups it is deemed the most interesting class with respect to MOT.

Plots describing the composition of the brackishMOT dataset with respect to motion and class distribution. For both plots, the data is from all the sequences.

Synthetic data framework

The authors constructed their synthetic framework within the Unity game engine, leveraging its built-in rendering pipeline. The framework revolves around three primary components: generating lifelike fish meshes, simulating fish behavior, and crafting a visually convincing underwater setting. They offer customizable options for each component, allowing users to tailor the synthetic environment to resemble the BrackishMOT data. Importantly, the framework is designed to be highly adaptable, enabling easy extension to accommodate other species, behavior models, and environments.

Fish model. The model chosen for this project was sourced from a database of fish images and photogrammetry 3D reconstructions. It was specifically selected for its visual similarity to a stickleback, a common member of the small fish class frequently found schooling in the sequences of the Brackish dataset, as noted by the authors. The 3D input model underwent decimation to reduce its complexity to 11,000 vertices. To retain finer details of the mesh, a normal map was generated from a high-resolution texture. Subsequently, the model was rigged using Blender’s bone system to facilitate fluid animations of both the body and tail. The number of fish spawned in each sequence is randomly determined within a range of 4 to 50, reflecting the diversity observed in the BrackishMOT sequences. Additionally, the initial pose, scale, and appearance including texture albedo and glossiness of each fish vary among the sequences.

Illustration of the stickle back fish model used in the framework. (a) Initial high resolution model and (b) decimated and rigged model for Unity.

Behavior model. To emulate realistic fish schooling behavior, the authors employ a boid-based approach. Each fish in the simulation takes into account the position and direction of all nearby fish. The velocity and heading of each fish are determined by four key factors: separation (s), cohesion (k), alignment (m), and leader (l). Separation ensures that each fish avoids collisions with its neighbors. Cohesion acts as a force that pulls fish towards the center of the group. Alignment encourages individual fish to match the velocity of their neighbors. The leader factor indicates the direction towards which a designated leader is heading. For each fish, the leader is selected as the neighbor whose heading vector most closely aligns with the fish’s own heading vector.

The surrounding environment. To explore how environmental changes affect tracker performance, the authors devised a synthetic environment based on three variables: turbidity, background, and distractors. Turbidity represents microscopic particles suspended in water, akin to a fog that becomes denser with increasing distance from the camera. This phenomenon significantly impacts visibility in the BrackishMOT sequences. The authors simulated turbidity using a custom Unity material with adjustable transparency, coupled with post-processing effects like depth of field and color grading. The material’s color ranges from grey to green, mirroring the turbidity observed in BrackishMOT sequences. Both color and intensity vary across different generated sequences. For background scenes, the authors utilized videos from the Brackish dataset devoid of fish to enhance realism. They varied background sequences through saturation, color, and blur adjustments. In the absence of a specific background, a monotone color matching the turbidity hue is employed. Additionally, distractors, representing floating particles, were introduced. In shallow waters with strong currents like those in BrackishMOT sequences, suspended plant materials and sediments often obscure the camera view. To simulate this, distractors spheres of varying size, transparency, and color are implemented. The color spectrum for distractors aligns with turbidity and monotone backgrounds. Their quantity varies across sequences, with each distractor randomly positioned and moved between frames. These environmental variables collectively enrich the synthetic scene with complexity akin to real-world scenarios. The combination of these variables yields eight distinct synthetic environments. Each generated video sequence lasts 10 seconds and comprises 150 frames at a frame rate of 15 FPS, closely resembling BrackishMOT sequences. The authors produced 50 sequences for each environment variation. Importantly, the synthetic framework’s flexibility allows for adjustments to fit diverse underwater environments. For instance, altering the video background or introducing new models can significantly alter sequence visuals.

Visualisation of different conditions of the synthetic environment. (a) Plain background, no turbidity, no distractors (Synth). (b) Video background, no turbidity, no distractors (SynthB). (c) Plain background, turbidity, no distractors (SynthT). (d) Plain background, no turbidity, with distractors (SynthD). (e) Plain background, with turbidityand distractors (SynthDT). (f) Video background with turbidity, but without distractors (SynthBT). (g) Video background with distractor, but no turbidity (SynthBD). (h) Video background with turbidity and distractors (SynthBTD).