Clip-based 3D style transfer

Motivation

One of the many reasons I started dabbling with modern computer graphics is the trend in the evolution of computer hardware. The shift from early-generation consoles to gaming GPUs, and next-generation tech has gradually changed how we perceive reality. We are slowly transitioning into a world where the line between digital reality blurs with the real world. The master builders and the visionaries of this hybrid world will be the creatives who can visualize and realize their ideas. Enabling them would allow us to plant seeds that lead to a multitude of inspirations and positive transformations. For someone deeply entrenched in computer science, theoretically and practically, I want to be the spark that can trigger the wildfire of change.

About Internship

During my internship, my mentor (Jonathan Granskog) and I endeavoured to explore the way for 3D asset creation. If you see the entire pipeline of 3D media, there are numerous possibilities for enhancement. We wanted to tackle asset creation. This led to our original idea to go from concept art to 3D mesh. But isn’t that the scope being overblown to the maximum? That is like converting a quick rough sketch into a full-blown complex representation.

Even in early civilization, humans sketched reality to communicate ideas. Their art was simple yet captured the essence of our outer world. In our modern times, we can conjure any world using already available computer graphics techniques. With the advent of generative AI, we have never been closer to going from idea to reality. Since our final goal is to get an image for us to see, we need a form that allows us to see the final picture.

Using simplicity as the motivation, we narrowed the scope from this grand goal into something achievable. Thus, we explored alternative methods for style transfer that can capture artistic elements such as brush strokes, paint textures and so on. Our research culminated in a publication in the Eurographics journal.

To view more cool demos visit here

To cite:

 @inproceedings {10.2312:egs.20231006,
 booktitle = {Eurographics 2023 - Short Papers},
 editor = {Babaei, Vahid and Skouras, Melina},
 title = {{CLIP-based Neural Neighbor Style Transfer for 3D Assets}},
 author = {Mishra, Shailesh and Granskog, Jonathan},
 year = {2023},
 publisher = {The Eurographics Association},
 ISSN = {1017-4656},
 ISBN = {978-3-03868-209-7},
 DOI = {10.2312/egs.20231006}
 }