Self-Supervised 3D Semantic Occupancy Prediction from Multi-View 2D Surround Images

authored by: S. Abualhanud, E. Erahan, M. Mehltretter
Abstract: An accurate 3D representation of the geometry and semantics of an environment builds the basis for a large variety of downstream tasks and is essential for autonomous driving related tasks such as path planning and obstacle avoidance. The focus of this work is put on 3D semantic occupancy prediction, i.e., the reconstruction of a scene as a voxel grid where each voxel is assigned both an occupancy and a semantic label. We present a Convolutional Neural Network-based method that utilizes multiple color images from a surround-view setup with minimal overlap, together with the associated interior and exterior camera parameters as input, to reconstruct the observed environment as a 3D semantic occupancy map. To account for the ill-posed nature of reconstructing a 3D representation from monocular 2D images, the image information is integrated over time: Under the assumption that the camera setup is moving, images from consecutive time steps are used to form a multi-view stereo setup. In exhaustive experiments, we investigate the challenges presented by dynamic objects and the possibilities of training the proposed method with either 3D or 2D reference data. Latter being motivated by the comparably higher costs of generating and annotating 3D ground truth data. Moreover, we present and investigate a novel self-supervised training scheme that does not require any geometric reference data, but only relies on sparse semantic ground truth. An evaluation on the Occ3D dataset, including a comparison against current state-of-the-art self-supervised methods from the literature, demonstrates the potential of our self-supervised variant.
Organisation(s): Institute of Photogrammetry and GeoInformation (IPI)
External Organisation(s): Robert Bosch GmbH
Type: Article
Journal: PFG - Journal of Photogrammetry, Remote Sensing and Geoinformation Science
Volume: 92
Pages: 483-498
No. of pages: 16
ISSN: 2512-2789
Publication date: 10.2024
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: Geography, Planning and Development, Instrumentation, Earth and Planetary Sciences (miscellaneous)
Electronic version(s): https://doi.org/10.1007/s41064-024-00308-9 (Access: Open)

BibTeX

@article{2ef15dc9ef6c4e34a46c0661b9bdc1e3,
title = "Self-Supervised 3D Semantic Occupancy Prediction from Multi-View 2D Surround Images",
abstract = "An accurate 3D representation of the geometry and semantics of an environment builds the basis for a large variety of downstream tasks and is essential for autonomous driving related tasks such as path planning and obstacle avoidance. The focus of this work is put on 3D semantic occupancy prediction, i.e., the reconstruction of a scene as a voxel grid where each voxel is assigned both an occupancy and a semantic label. We present a Convolutional Neural Network-based method that utilizes multiple color images from a surround-view setup with minimal overlap, together with the associated interior and exterior camera parameters as input, to reconstruct the observed environment as a 3D semantic occupancy map. To account for the ill-posed nature of reconstructing a 3D representation from monocular 2D images, the image information is integrated over time: Under the assumption that the camera setup is moving, images from consecutive time steps are used to form a multi-view stereo setup. In exhaustive experiments, we investigate the challenges presented by dynamic objects and the possibilities of training the proposed method with either 3D or 2D reference data. Latter being motivated by the comparably higher costs of generating and annotating 3D ground truth data. Moreover, we present and investigate a novel self-supervised training scheme that does not require any geometric reference data, but only relies on sparse semantic ground truth. An evaluation on the Occ3D dataset, including a comparison against current state-of-the-art self-supervised methods from the literature, demonstrates the potential of our self-supervised variant.",
keywords = "3D Occupancy Prediction, 3D Perception, NeRF, Semantic Scene Completion",
author = "S. Abualhanud and E. Erahan and M. Mehltretter",
note = "Publisher Copyright: {\textcopyright} The Author(s) 2024.",
year = "2024",
month = oct,
doi = "10.1007/s41064-024-00308-9",
language = "English",
volume = "92",
pages = "483--498",
journal = "PFG - Journal of Photogrammetry, Remote Sensing and Geoinformation Science",
issn = "2512-2789",
publisher = "Springer International Publishing AG",
number = "5",
}