At SIGGRAPH 2024, which will be held 28 July–1 August in Denver, novel research — as part of the Technical Papers program — will demonstrate key advances that enable detailed and lifelike animations, once considered the realm of science fiction. This relationship between AI and computer graphics continues to evolve, signaling a future where the digital and real worlds seamlessly converge.
"The impact of generative AI on 3D graphics and interactive techniques is profound, poised to democratize content creation and lower barriers to entry in creative industries. Generative AI can facilitate real-time, personalized experiences in gaming, architecture, and industrial design, revolutionizing fields through procedural content generation and rapid prototyping," Qixuan (QX) Zhang, project lead of research that will debut at SIGGRAPH 2024, says. QX and his collaborators will present CLAY, a large-scale 3D native generative model that accurately captures complex geometric features and generates realistic textures in a variety of objects.
"We anticipate a future where generative AI enhances productivity and creativity across professional domains, making advanced 3D content creation and interaction a part of everyday life for a global audience," QX adds.
Following SIGGRAPH's 50th conference milestone in 2023, the conference this year will showcase exciting new research fueling the future — advances developed from a core place of imagination and creativity.
As a preview of the Technical Papers program, here is a sampling of three novel methods and their unique approaches incorporating generative AI to the vast field.
Dreaming Up High-Quality, Realistic Images via Text Prompts
Creating high-quality appearances of objects is a critical task in computer graphics because they can significantly improve the realism of rendering in a variety of applications — movies, games, and AR/VR. An international team of researchers from Zhejiang University, Tencent Games-China, and Texas A&M University, have developed DreamMat, a novel text-to-appearance method that generates images consistent with their geometry and environment light. Central to the team's framework is the geometry- and light-aware diffusion model that can avoid common problems of "baking in" shading effects that create unrealistic looking objects in a virtual environment.
The researchers set out to make this specific artistic task easier and faster with a tool for efficiently creating object appearances such that even novice users can do so with just simple text prompts. In this work, they fine tuned a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lighting in the material distillation, DreamMat can generate high-quality PBR (physically based rendering) materials that are not only consistent with the given geometry but also free from any unwanted and unrealistic shading effects.
"DreamMat has significantly lowered the barrier to creating PBR materials for untextured meshes (only using text prompts), enabling the generation of high-quality appearance that can be seamlessly integrated into games and films — a feat previously unattainable with earlier generative techniques," Xiaogang Jin, lead author and a professor at the State Key Lab of CAD & CG at Zhejiang University, China, says. "The most exciting thing is witnessing the potential for genuine application of AI-generated 3D assets beyond mere display."
DreamMat's automatic creation of PBR materials can be imported directly into modern graphics engines such as Blender, GameMaker, or Unity for efficient rendering. In addition to Jin, the team comprises Yuqing Zhang and Zhiyu Xie at Zhejiang University; Yuan Liu from the University of Hong Kong and Tencent Games; Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, and Cheng Lin at Tencent Games; and Wenping Wang at Texas A&M University, and they are slated to showcase their work at SIGGRAPH 2024. For more about DreamMat and a video, visit their page.
Transforming Realistic 3D Assets, Quickly and Efficiently
In just recent years, the world has witnessed rapid advancements in generative AI across various modalities — text, images, audio, and video. With the public release of ChatGPT and the like, a much wider audience now has a direct connection and firsthand experience with how AI works. But despite significant research achievements in the 3D computer graphics field, industrial adoption has lagged.
In a collaboration between ShanghaiTech University and Deemos Tech-China, researchers set out to better understand the limitations game developers and CG teams are facing with existing 3D generation approaches. What they learned is that the traditional "reconstruction" methods fall short of industry needs. True 3D generative AI "must go beyond mere reconstruction; it must comprehend user requirements, support multitasking, and offer customization options akin to the vibrant 2D generative AI community," QX, a member of the ShanghaiTech-Deemos team that will be presenting their research at SIGGRAPH 2024, says.
The researchers have developed CLAY, a native 3D generative model with 1.5 billion parameters that accurately captures complex geometric features and generative realistic textures. Trained on a comprehensive 3D dataset, CLAY's uniqueness, say the researchers, lies in its alignment with the specific needs of the 3D industry — usability being its primary focus and design ability to cater directly to practical applications.
Within a minute, CLAY can generate detailed 3D assets with PBR materials adaptable to various conditions, from text, image to 3D. Their approach denoises 3D data, and the researchers say that when compared to previous similar methods, CLAY generates detailed geometries in seconds while preserving essential geometric features, such as flat surfaces and structural integrity, of multiple objects.
"Looking ahead, we envision generative AI technologies in the 3D field liberating artists from repetitive tasks, allowing them to focus on creativity and innovation," QX adds. "We aim to extend 3D generation technologies to everyday applications such as virtual try-ons for online shopping, enhanced creative in the metaverse, and novel gameplay experiences in gaming."
Working with QX on the CLAY development are collaborators Longwen Zhang, Ziyu Wang, Qiwei Qiu, and Haoran Jiang from ShanghaiTech and Deemos; Anqi Pang, Lan Xu, and Jingyi Yu at ShanghaiTech; and Wei Yang at Huazhong University of Science and Technology. The team's paper and video can be accessed
here .
