In recent years, a variety of approaches have developed deep neural network-based architectures for 3D shape reconstruction and synthesis with wide-ranging applications to computer-aided design, fabrication, architecture, art, and entertainment. While these methods can capture diverse macro-level appearances, they rarely model shape structure or topology explicitly, relying instead on the representational power of the network to generate plausible-looking shapes. In my work, I introduce realistic 3D shape reconstruction and generation methods that accurately models complex topological and geometrical details, and support interpretable control of shape structure and geometry.
(1) I propose ANISE - a new part-aware shape reconstruction method based on neural implicit functions. Specifically, given a partial shape observation (image or point cloud), it reconstructs shapes as a combination of parts, each with its own geometric representations. I formulate shape reconstruction in two different manners: as a union of part implicit functions, or by retrieving parts in a reference database and assembling them into a final shape. This approach allows modification of the final result either by moving parts or swapping them using part latent codes.
(2) I introduce the GEM3D - a two-step 3D shape generation model that first generates medial skeletal abstraction that captures and then infers and assembles a collection of locally-supported neural implicit functions, conditioned on generated skeletal abstraction. This skeleton-based latent grid is more structure-aware compared to other irregular latent grid approaches, providing more interpretable support for latent codes in 3D space, while remaining capable of representing complex, fine-grained topological structures. It also allows for editing of the resulting surface through manipulation of the generated skeleton.
(Future and ongoing work) Current method relies on a simplified form of point-based skeletons with fixed resolution. Generalizing this method to create more expressive representations, such as skeletal diagrams, may further enhance the topology information encoded in the model. Despite the improvements in the topology of reconstructed shapes, the current approach does not guarantee certain topology characteristics; the surfaces might still mismatch a target topology (e.g., a certain genus) or contain undesired, noisy shape components. Finally, an interesting future avenue would be to enable interactive skeleton editing, exploration, and shape synthesis from the edits.
Advisor: Evangelos Kalogerakis