PhD Dissertation Proposal: Shifan Zhu, Geometry-Guided Spatial Understanding: From Motion Estimation to Photorealistic Scene Construction

Content

Speaker

Shifan Zhu

Abstract

Geometric constraints provide a foundation for understanding spatial structure and relationships. In visual perception, the geometry of image formation relates camera motion and three-dimensional scene structure to recorded observations. Camera-motion estimation and photorealistic scene construction address complementary aspects of this relationship: geometric correspondences constrain motion estimation, while estimated camera trajectories provide constraints for learning scene geometry and appearance. Methods such as three-dimensional Gaussian splatting (3DGS) can produce photorealistic renderings at viewpoints adequately supported by those observations. However, rendering quality can degrade at camera positions and viewing directions beyond the recorded coverage, where newly visible surfaces may be insufficiently constrained by the available images. The central research question is how geometric constraints and imperfect priors can advance spatial understanding through more accurate motion estimation and photorealistic scene synthesis that remains consistent across views.

The completed research demonstrates how geometric constraints can reduce ambiguity in visual perception and support spatial understanding. The proposed work extends this geometric foundation to photorealistic three-dimensional scene construction. Recovered camera trajectories and an initial textured mesh will provide scene-specific geometry and observed appearance. Vision-language models will combine these priors with learned geometric priors to infer scene structure insufficiently determined by the recordings. The resulting mesh will serve as a geometric prior for learning appearance from the recorded images.

Learning appearance from this generated mesh is challenging because its inferred geometry may differ substantially from the physical environment. Missing, misplaced, or invented structures can make rendered views differ from recorded images at the same camera view, complicating the use of those images to supervise appearance learning. The core proposed contribution is a differentiable method for learning photorealistic appearance under substantial geometric mismatch. The method will learn how mesh surfaces correspond to recorded image content, combining geometric priors with observed appearance. Its intended outputs are photorealistic images or video at camera positions and viewing directions insufficiently covered by the recordings, with consistency across views.

Evaluation will assess photorealism, consistency across views, and usefulness for robot learning, with navigation as a representative application. The project will target the public release of 1,000 photorealistic scenes as simulation assets with separate meshes for individual objects, designed to support navigation and manipulation. This real-to-sim pipeline aims to make diverse recordings, including online videos, useful for robot learning and reduce the visual sim-to-real gap.

Advisor

Donghyun Kim