Deng, Zhigang2023-07-20May 20232023-04-27Portions of this document appear in: Deng, Qixin, Luming Ma, Aobo Jin, Huikun Bi, Binh Huy Le, and Zhigang Deng. "Plausible 3d face wrinkle generation using variational autoencoders." IEEE Transactions on Visualization and Computer Graphics 28, no. 9 (2021): 3113-3125; and in: Deng, Qixin, Binh H. Le, Aobo Jin, and Zhigang Deng. "End-to-End 3D Face Reconstruction with Expressions and Specular Albedos from Single In-the-wild Images." In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4694-4703. 2022.https://hdl.handle.net/10657/15037Realistic 3D facial modeling and reconstruction have been increasingly used in many graphics, animation, and virtual reality applications. Currently many existing face models are not able to present rich details while deforming, which means lack of wrinkles while face shows different expressions. Also, to create a realistic face model for an individual is also needs complex setup and sophisticated works from experienced artists. The goal of this dissertation is to achieve an end-to-end system to augment coarse-scale 3D face models, and to reconstruct realistic face from in-the-wild images. I propose an end-to-end method to automatically augment coarse-scale 3D faces with synthesized fine scale geometric wrinkles. I define the wrinkle as the displacement value along the vertex normal direction, and save it as displacement map. The distribution of wrinkles has some spatial characteristics, and deep convolutional neural network (DCNN) is pretty good at learning spacial information across image-format data. I labeled the wrinkle data with its identity and expression vectors. By formulating the wrinkle generation problem as a supervised generation task, I implicitly model the continuous space of face wrinkles via a compact generative model, such that plausible face wrinkles can be generated through effective sampling and interpolation in the space. Then I introduce a complete pipeline to transfer the synthesized wrinkles between faces with different shapes and topologies. The method can augment an exist 3D face model with more fine-scale details, but to create a realistic human face model is not yet solved. Properly modeling complex lighting effects in reality, including specular lighting, shadows, and occlusions, from a single in-the-wild face image is still considered as a widely open research challenge. To reconstruct an realistic face model from an unconstrained image, I propose a CNN based framework to regress the face model from a single image in the wild. I designed novel hybrid loss functions to disentangle face shape identities, expressions, poses, albedos, and lighting. The outputted face model includes dense 3D shape, head pose, expression, diffuse albedo, specular albedo, and the corresponding lighting conditions.application/pdfengThe author of this work is the copyright owner. UH Libraries and the Texas Digital Library have their permission to store and provide access to this work. UH Libraries has secured permission to reproduce any and all previously published materials contained in the work. Further transmission, reproduction, or presentation of this work is prohibited except with permission of the author(s).3D face reconstructionDeep learning algorithmsSpecular albedoFacial expressionsFace modelingWrinkle synthesisDeep generative modelsVariational autoencoders2023-07-20Thesisborn digital