The TAS500: Semantic Segmentation Dataset For Autonomous Driving in Unstructured Environments is a unique dataset designed specifically for semantic segmentation in autonomous driving within unstructured environments. TAS500 provides detailed classifications of vegetation and terrain, enabling effective learning of drivable surfaces and natural obstacles in outdoor scenes. The authors highlight a significant gap in the research of autonomous driving for unstructured environments due to the scarcity of semantically labeled datasets, especially when compared to urban environments.

Research in autonomous driving for unstructured environments suffers from a lack of semantically labeled datasets compared to its urban counterpart. Urban and unstructured outdoor environments are challenging due to the varying lighting and weather conditions during a day and across seasons. TAS500 offers fine-grained vegetation and terrain classes to learn drivable surfaces and natural obstacles in outdoor scenes effectively. The authors evaluate the performance of modern semantic segmentation models with an additional focus on their efficiency. The authors experiments demonstrate the advantages of fine-grained semantic classes to improve the overall prediction accuracy, especially along the class boundaries.

Semantic scene understanding is a key capability for autonomous robot navigation in real-world environments, but current research in autonomous driving focuses mainly on urban, suburban, and highway scenes. These scenes are considered as structured environments. In terms of their scene statistics, structured environments often provide more explicit object boundaries and contain objects with strong structural regularities. The authors are interested in robot navigation in unstructured environments, such as paths through forests and along fields. Here the occurrence of many semi-transparent classes such as tree foliage and the subtle color difference between vegetation types pose a challenge during image processing.

Apart from a robust detection of passable and impassable vegetation, the domain of robot navigation requires a real-time capability from the authors semantic segmentation model to be able to react to changes in the environment. The authors studied how a more fine-grained labeling policy affects the overall performance of the semantic segmentation in unstructured environments. That is, they want to distinguish between drivable (grass) and non-drivable vegetation (bush and tree trunk), and detect different kinds of drivable surfaces (asphalt, gravel, soil, sand).

Data acquisition

The authors’ goal is to improve understanding of the semantic scene in structured environments led to the development of TAS500 dataset. It contains more than
500 RGB images with pixel-level ground truth annotations for driving in unstructured outdoor environments. The data was collected using the autonomous vehicle
MuCAR-3. The authors vision system is mounted on a multifocal active/reactive camera platform called MarVEye. A camera sensor provides color images at 2.0 MP resolution. They recorded data with a frame rate of 10 Hz and cut off most of the sky and ego vehicle hood from all images. The final images have a resolution of 620 × 2026 px. Their label rate amounts to around 0.1 Hz, and they consequently provide a pixel-wise semantic mask for every hundredth recorded image. The authors provide fine-grained annotations at pixel level, they split up categories such as nature and flat to further subdivide vegetation as well as drivable surfaces. They annotated images from five test drives(Unfortunately, the authors did not provide information about the sequences).

Number of labeled frames and corresponding duration for the five annotated sequences in TAS500.

Annotation Process

The authors pixel-level semantic masks were labeled with a self-developed application, which works similar to the image annotation tool LabelMe, but their tool is also capable of annotating 3D point clouds. They use closed polygonal chains to label objects and structures in camera images; exactly one class label was assigned for every pixel. The annotation speed was increased by reusing the object boundaries and totaled approximately 45 min per frame, including a quality control process. The authors labeling policy defines 44 class labels that are categorized into nine groups (tagged as supercategory): animal construction, human, object, sky, terrain, vegetation, vehicle, and void. Moreover, they want to ensure compatibility with existing datasets. They therefore include most classes from the Cityscapes dataset but exclude rare classes. Thus they only use the classes car and bus from the vehicle category. They map infrequent classes such as rider to the closest corresponding category (here: person), while too specific class distinctions such as the classes street light and pole are consolidated into one super-class (here: pole). Through this process they arrive at 23 classes. The labeling policy for TAS500 was designed for driving in unstructured environments. Common objects were subdivided if they might require different planning behavior from an autonomous vehicle. Specifically vegetation classes can be split into drivable (e.g. high grass) and non-drivable vegetation (e.g. bush). The authors fine-grained vegetation annotations allow the distinction between passable (e.g. tree crown) and impassible (e.g. tree trunk) classes within the non-drivable vegetation subcategory. Passable vegetation covers any type of vegetation that an autonomous vehicle could graze during path planning.

Number of fine-grained pixels (y-axis) per class and their associated category (x-axis).