Abstract: Neural image synthesis approaches have become increasingly popular over the last years due to their ability to generate photorealistic images useful for several applications (digital entertainment, art, mixed reality, synthetic dataset creation, to name a few). Despite the progress over the last years, current approaches lack two important aspects: (a) they often ignore the underlying 3D geometry of the shape/scene in the image, and as a result, they frequently lose coherency and details. (b) they fail to capture long-range interactions in the image, and as a result, they fail to generate scenes with complex dependencies between their different objects or parts.
My thesis proposes novel solutions to the above problems. First, I propose a method that generates artistic images with the guidance of input 3D shapes. In contrast to previous methods, the use of a geometric representation of 3D shape enables the synthesis of more precise stylized drawings with fewer artifacts. My method outputs the synthesized images in a vector representation, enabling richer downstream analysis or editing in interactive applications. I also show that the method produces substantially better results than existing image-based methods, in terms of predicting artists' drawings, and in user evaluation of results.
Second, I propose a neural transformer architecture that captures long-range interactions and context for image synthesis at high resolutions, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that was not possible to generate reliably with previous ConvNets and other transformer approaches. The key idea of the architecture is to sparsify the transformer's attention matrix at high resolutions, guided by dense attention extracted at lower image resolution. I present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of the method, and its superiority compared to the state-of-the-art.
Advisor: Evangelos Kalogerakis