Robust Shape Fitting for 3D Scene Abstraction

authored by: Florian Kluger, Eric Brachmann, Michael Ying Yang, Bodo Rosenhahn
Abstract: Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.
Organisation(s): Institute of Information Processing
External Organisation(s): Niantic Inc.
University of Bath
Type: Article
Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume: 46
Pages: 6306-6325
No. of pages: 20
ISSN: 0162-8828
Publication date: 09.2024
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: Software, Computer Vision and Pattern Recognition, Computational Theory and Mathematics, Artificial Intelligence, Applied Mathematics
Electronic version(s): https://doi.org/10.48550/arXiv.2403.10452 (Access: Open)
https://doi.org/10.1109/TPAMI.2024.3379014 (Access: Closed)

BibTeX

@article{efa0adbf6c6548cda932c632436b0a2d,
title = "Robust Shape Fitting for 3D Scene Abstraction",
abstract = "Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.",
keywords = "cuboid fitting, Estimation, Image reconstruction, minimal solver, multi-model fitting, Scene abstraction, Shape, shape decomposition, Solid modeling, Surface reconstruction, Three-dimensional displays, Training",
author = "Florian Kluger and Eric Brachmann and Yang, {Michael Ying} and Bodo Rosenhahn",
note = "Publisher Copyright: {\textcopyright} 1979-2012 IEEE.",
year = "2024",
month = sep,
doi = "10.48550/arXiv.2403.10452",
language = "English",
volume = "46",
pages = "6306--6325",
journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",
issn = "0162-8828",
publisher = "IEEE Computer Society",
number = "9",
}