SIGGRAPH Asia 2024 Course on
Automatic 3D modeling and exploration of indoor structures from panoramic imagery
Giovanni Pintore, CRS4 & National Research Center in High-Performance Computing, Big Data and Quantum Computing, Italy
Marco Agus, HBKU, Qatar
Enrico Gobbetti, CRS4 & National Research Center in High-Performance Computing, Big Data and Quantum Computing, Italy
Time and venue
- Date and time
- Wednesday, 4 December 2024 | 2:00pm - 5:45pm JST
- Location
- G610, G Block, Level 6, Tokyo International Forum, Tokyo, Japan
- Link
- SIGGRAPH Asia 2024 Web page
Overview
Summary. Surround-view panoramic imaging provides the quickest and most complete per-image coverage and is supported by a wide variety of professional and consumer capture devices. Research on inferring 3D indoor models from 360-degree images has been thriving in recent years and has led to very effective solutions. Given the complexity and variability of interior environments, and the need to cope with noisy and incomplete captured data, many open research problems remain. In this course, we provide an up-to-date integrative view of the field. After introducing a characterization of input sources, we define the structure of output models, the priors exploited to bridge the gap between imperfect input and desired output, and the main characteristics of geometry reasoning and data-driven approaches. We then identify and discuss the main sub-problems in indoor reconstruction from panoramas and review and analyze state-of-the-art solutions for indoor capture, room modeling, integrated model computation, visual representation generation, and immersive exploration. Relevant examples of implemented pipelines will be described, with a major focus on deep-learning solutions. We finally point out relevant research issues and analyze research trends.
Background and intended audience. The course is at the intermediate level. Basic computer vision and deep learning backgrounds are prerequisites. The target audience includes graduate students, researchers in 3D modeling and scene understanding, and practitioners in the relevant application fields. Researchers will find a structured overview of the field, which organizes the various problems and existing solutions, classifies the existing literature, and indicates challenging open problems. Domain experts will, in turn, find a presentation of the areas where automated methods are already mature enough to be ported into practice, as well as an analysis of the kind of indoor environments that still pose major challenges.
Organizers and Lecturers
Giovanni Pintore
CRS4, Italy
National Research Center in High-Performance Computing, Big Data and Quantum Computing, Italy
(AUTHOR+LECTURER)
Giovanni Pintore is a senior researcher at the CRS4 research center in Italy. In his career he has coordinated and managed several international research and industrial projects in various fields, from space exploration to security management in urban environments. His research, widely published in major journals and conferences, spans many areas of computer graphics and computer vision, including deep learning architectures, geometry reasoning, panoramic scene understanding, multiresolution representations of large and complex 3D models, 3D multi-view reconstruction, and new generation mobile graphics. He regularly serves the scientific community through participation in conference committees and executive boards. His primary research focus is now in 3D reconstruction and immersive exploration of structured indoor scenes from omnidirectional images, on whose topic he has recently published papers at SIGGRAPH Asia, ISMAR, ECCV, CVPR, and CGF, and given courses at SIGGRAPH, SIGGRAPH Asia, 3DV, and CVPR in recent years.
Marco Agus is currently Associate Professor at Hamad Bin Khalifa University (HBKU) - Qatar Foundation in Doha, Qatar. He was previously research engineer at King Abdullah University of Science and Technology (KAUST), in Jeddah, Saudi Arabia and research scientist at Center of Research, Development and Advanced Studies (CRS4), in Cagliari, Italy. He obtained M.Sc. and Ph.D. from University of Cagliari, Italy. His research interests span different domains in visual computing, from haptics and visual rendering for medical applications, to real time exploration of massive models, to machine learning methods for electron microscopy biology data and indoor environments. He published more than 50 peer-reviewed papers on these topics. He taught courses at several important visual computing venues, including CVPR, 3DV, ACM SIGGRAPH and Eurographics, and he regularly acts as committee member, reviewer, chair and associate editor for top journals and conferences in the visual computing domain.
Enrico Gobbetti
CRS4, Italy
National Research Center in High-Performance Computing, Big Data and Quantum Computing, Italy
(AUTHOR+ORGANIZER)
Enrico Gobbetti is the director of Visual and Data-intensive Computing (ViDiC) at the Center for Advanced Studies, Research, and Development in Sardinia (CRS4), Italy. He holds an Engineering degree (1989) and a Ph.D. degree (1993) in Computer Science from the Swiss Federal Institute of Technology in Lausanne (EPFL), as well as Full Professor Habilitations in Computer Science and Information Processing from the Italian Ministry of University and Research. Prior to joining CRS4, he held positions at EPFL (Switzerland), UMBC (USA), and NASA/CESDIS (USA). At CRS4, Enrico develops and manages a research program in visual and data-intensive computing supported through institutional, industrial and government grants, including many national and international collaborative projects. His research spans many areas of visual and data-intensive computing and is widely published in major journals and conferences. The primary focus is the creation of innovative solutions for the acquisition, creation, processing, distribution and exploration of complex and/or massive datasets and real-world objects and environments. He regularly serves the scientific community through participation in editorial boards, conference committees, working groups and steering boards, as well as through the organization and chairing of conferences. He is a Fellow of the Eurographics Association.
Schedule and slides
| Time | Duration | Lecturer | Topic | Sub-topics | Link |
|---|---|---|---|---|---|
| 2:00pm-2:10pm JST | 10' | Giovanni Pintore | Opening | Course motivation and outline; Presenters introduction; course overview | [Slides] |
| 2:10pm-2:35pm JST | 25' | Marco Agus | Indoor capture, modeling, and exploration basics | Definitions and applications; Tasks and model, Data capture; Panoramic cameras; Artifacts; Reconstruction priors; Open research data | [Slides] |
| 2:35pm-3:20pm JST | 45' | Giovanni Pintore | Room modeling | Bounding surfaces; Exploiting priors; Deep learning solutions; Examples of data-driven pipelines for depth and layout recovery | [Slides] |
| 3:20pm-3:35pm JST | 15' | - | BREAK | - | - |
| 3:35pm-4:20pm JST | 45' | Giovanni Pintore | Integrated indoor model computation | Multi-rooms; Multi-view; Segmentation and localization; Examples of data-driven pipelines for 3D floorplan recovery | [Slides] |
| 4:20pm-5:20pm JST | 60' | Marco Agus | Visual representation generation and exploration | Appearance; Immersive panoramic exploration from single panoramas; Example of integration within interactive XR applications | [Slides] |
| 5:20pm-5:45pm JST | 25' | ALL | Wrap-up, discussion, and Q&A | Summary of techniques and assessment of capabilities; Open problems; Open discussion | [Slides] |
Additional material
The course builds on our survey [PMG2020a], and on our tutorial presented at SIGGRAPH 2020 [PMG2020b]. While those articles and presentations covered many kinds of visual and non-visual input sources, we focus here on panoramic imagery. The topic has attracted a large interest in the community, as demonstrated by the growing number of papers in the latest computer vision conferences (see references in [PMG2020a] and [PAG2024]), and by the events that are regularly featured to discuss the latest advances (e.g., workshop on Holistic Structures for 3D Vision (ICCV, ECCV), OmniCV: Omnidirectional Computer Vision in Research and Industry (CVPR), ScanNet Indoor Scene Understanding Challenge (CVPR)). Our course fills a gap in the visual computing. A first version, more centered on computer vision aspects, has been presented at CVPR 2023 [PAG2023].
A copy of the slides for this course will be made available close to the conference date through the links in the "Schedule" table. The tutorial notes are available in the ACM Digital Library [PAG2024].
- [PMG2020a]
- Giovanni Pintore, Claudio Mura, Fabio Ganovelli, Lizeth Fuentes-Perez, Renato Pajarola, and Enrico Gobbetti. State-of-the-art in Automatic 3D Reconstruction of Structured Indoor Environments. Computer Graphics Forum, 39(2): 667-699, 2020. DOI: 10.1111/cgf.14021
- [PMG2020b]
- Giovanni Pintore, Claudio Mura, Fabio Ganovelli, Lizeth Fuentes-Perez, Renato Pajarola, and Enrico Gobbetti. Automatic 3D Reconstruction of Structured Indoor Environments. In SIGGRAPH 2020 Courses. Pages 10:1-10:218, August 2020. DOI: 10.1145/3388769.3407469
- [PAG2023]
- Giovanni Pintore, Marco Agus, and Enrico Gobbetti. Automatic 3D modeling of indoor structures from panoramic imagery. In CVPR Tutorials, 2023.
- [PAG2024]
- Giovanni Pintore, Marco Agus, and Enrico Gobbetti. Automatic 3D modeling and exploration of indoor structures from panoramic imagery. In SIGGRAPH Asia 2024 Courses (SA Courses '24). DOI: 10.1145/3388769.3407469
BibTeX
@inproceedings{PAG:2024:CTA,
author = {Giovanni Pintore and Marco Agus and Enrico Gobbetti},
title = {Auomatic 3D modeling and exploration of indoor structures from panoramic imagery},
booktitle = {Proc. SIGGRAPH Asia Courses (SA Courses '24)},
year = {2024}.
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3388769.3407469},
url = {\url{https://www.crs4.it/vic/sigasia2024-course-pano}},
}
Acknowledgments
GP and EG acknowledge the contribution of the Italian National Research Center in High-Performance Computing, Big Data and Quantum Computing. MA, GP, and EG received funding from NPRP-Standard (NPRP-S) 14th Cycle grant 0403-210132 AIN2 from the Qatar National Research Fund (a member of Qatar Foundation). The findings herein reflect the work and are solely the responsibility of the authors.