The authors introduce the OFFSED: Off-Road Semantic Segmentation Dataset, comprising 203 images with full-image semantic segmentation masks. These masks categorically assign one of 19 classes to each pixel. The selection of images was done in a way that captures the whole range of environments and human poses. In addition to pixel labels, a few selected countable classes also come with instance identifiers. This allows for the use of the dataset in instance and panoptic segmentation tasks.

Note, similar OFFSED: Off-Road Semantic Segmentation Dataset dataset is also available on the DatasetNinja.com:

• OPEDD: Off-Road Pedestrian Detection Dataset

Motivation

In the past decade, significant strides in neural network technology have led to notable advancements in automated driver assistance systems. The ability of self-driving vehicles to navigate their surroundings reliably and autonomously relies on their capacity to infer semantic information about the environment. While a considerable portion of research focuses on private passenger cars and cargo trucks, which typically operate in common environments like paved roads, highways, and cities, it’s crucial to address the unique challenges faced by industrial vehicles such as tractors and excavators. These industrial vehicles constitute a substantial portion of the global motorized vehicle fleet and operate in distinctly different settings.

Current state-of-the-art approaches in semantic full image segmentation predominantly rely on convolutional neural networks. These networks take a monocular RGB image as input and assign a class label to every pixel in the image. Training these networks requires large image datasets with ground truth semantic labels for each pixel. However, existing datasets, primarily focused on private passenger cars and cargo trucks, predominantly feature urban environments or highway roads. These settings are unsuitable for training assistance systems on industrial vehicles and working machines, which operate in semantically and visually distinct surroundings. For instance, the passable ground for a passenger car differs significantly from that of an excavator, encountering different objects than cars in urban environments.

To address this gap, the authors present a OFFSED: Off-Road Semantic Segmentation Dataset, that introduces an off-road variant to the collection of semantic segmentation datasets. This dataset captures off-road environments, including meadows, woods, construction sites, farmland, and paddocks. Pedestrians in various poses, some highly unconventional in the Advanced Driver Assistance Systems (ADAS) context, such as crouching, lying down, or handstands, are depicted to provide extremes for comprehensive training and testing scenarios. Dataset contains 203 images by incorporating full-image semantic segmentation masks, assigning one of 19 classes to each pixel. The selection of images was thoughtfully curated to encompass a diverse range of environments and human poses featured in the original dataset. In addition to pixel labels, certain countable classes also include instance identifiers for enhanced precision.

Full image segmentation masks (right) and their corresponding images (left). Construction sites are common environments for mobile working machines, but are underrepresented in most deep learning datasets.

Dataset creation

The images in this dataset are a subset of the data from authors previous work, OPEDD. It consists of 203 images that were captured in five different environments, meadows, woods, construction sites, farm-land and paddocks. The images were extracted from stereo video sequences recorded with a ZED camera in lossless compression. Manually annotating images for semantic segmentation is a labour intensive and costly task. The ground truth semantic segmentation annotations were created manually with the help of the annotation tool CVAT. In this annotation tool, annotators delineate polygons for each object and assign labels based on a predefined set of 19 classes. Notably, these polygons are endowed with a z-layer level, establishing a depth ordering that streamlines the annotation process by drawing borders between polygons only once, thus optimizing time. Subsequently, full-image pixel masks are generated from these polygons, taking into account the polygon depth.

The selection of classes was carefully curated to mirror potential environments for mobile working machines. Emphasis was placed on specific classes such as drivable and nondrivable surfaces like dirt or pavement/concrete. These classes, crucial for determining the suitability of terrain for mobile working machines, are inherently linked to the ego-vehicle and are susceptible to semantic and visual ambiguity. For instance, a dirt heap traversable by a larger machine may pose an obstacle for a smaller one. Throughout the annotation process, considerations were guided by the context of ADAS for mobile working machines, influencing decisions regarding distinctions between drivable and non-drivable dirt, with a focus on addressing potential challenges arising from semantic and visual nuances.

Note: the authors of the dataset did not provide a way to compare instance identifier with the corresponding image.