Generative and Implicit Methods for 3D Point Cloud Processing

Abstract

3D point clouds are a popular form of data representation with many applications in computer vision, computer graphics, and robotics. As the output of range sensing devices, point clouds have gained popularity with the current interest in self-driving vehicles. More formally, point clouds are an unordered set of irregular points collected from the surface of an object. Each point consists of a Cartesian coordinate, along with additional information such as an RGB color value and surface normal estimate. However, deep learning methods fall short in the processing of 3D point clouds due to the irregular and permutation-invariant nature of the data.

In this dissertation, we design novel types of neural networks that leverage raw 3D point clouds for data creation and reconstruction. First, we investigate dense colored point cloud generation and present an understanding of shape color correlation with a progressive conditional generative adversarial network (PCGAN). PCGAN learns to create a 3D data distribution by producing colored point clouds with subtle details at a range of resolutions. Next, we reconstruct open surfaces with inner details by extracting surface points from an unsigned distance field with an implicit point voxel network (IPVNet). In IPVNet, we show that by combining features from different 3D representations such as point clouds and voxels, deep learning models can reduce both inaccuracies and the number of outliers in the reconstruction. Finally, we discuss reconstructing a 3D surface from a single image by learning an implicit function through a spatial transformer (LIST). Within the LIST framework, we introduce an innovative spatial transformer that creates the ability to accurately retrieve intricate details from a single image without the need for any additional rendering information.

Overall, we provide a comprehensive investigation of generative and implicit point cloud processing techniques. We establish novel deep-learning frameworks to facilitate the 3D reconstruction and generation tasks. Additionally, we make our source code and other resources publicly available for the benefit of the research community.