Real-Time Neural Appearance Models

Tizian Zeltner ^* Fabrice Rousselle ^* Andrea Weidlich ^* Petrik Clarberg ^* Jan Novák ^* Benedikt Bitterli ^* Alex Evans Tomáš Davidovič Simon Kallweit Aaron Lefohn

^* Equal contribution. Order determined by a rock-paper-scissors tournament.

In Transactions on Graphics (Presented at SIGGRAPH 2024)

Close-up renderings of a teapot asset with our neural BRDF. Our model learns the intricate details of the complex multi-layered material behavior of the ceramic, fingerprints, smudges, and dust which are responsible for the realism of the object while being faster to evaluate than traditional non-neural models of a similar complexity. The system we present allows us to include such high-fidelity objects in real-time renderers in a scalable way.

Abstract

We present a complete system for real-time rendering of scenes with complex appearance previously reserved for offline use. This is achieved with a combination of algorithmic and system level innovations.
Our appearance model utilizes learned hierarchical textures that are interpreted using neural decoders, which produce reflectance values and importance-sampled directions. To best utilize the modeling capacity of the decoders, we equip the decoders with two graphics priors. The first prior—transformation of directions into learned shading frames—facilitates accurate reconstruction of mesoscale effects. The second prior—a microfacet sampling distribution—allows the neural decoder to perform importance sampling efficiently. The resulting appearance model supports anisotropic sampling and level-of-detail rendering, and allows baking deeply layered material graphs into a compact unified neural representation.
By exposing hardware accelerated tensor operations to ray tracing shaders, we show that it is possible to inline and execute the neural decoders efficiently inside a real-time path tracer. We analyze scalability with increasing number of neural materials and propose to improve performance using code optimized for coherent and divergent execution. Our neural material shaders can be over an order of magnitude faster than non-neural layered materials. This opens up the door for using film-quality visuals in real-time applications such as games and live previews.

NVIDIA project page

Video

Downloads

Publication

Supplemental

BibTeX Reference


        @article{Zeltner2024RealtimeNeural,
    author = {Zeltner, Tizian and Rousselle, Fabrice and Weidlich, Andrea and Clarberg, Petrik and Nov\'{a}k, Jan and Bitterli, Benedikt and Evans, Alex and Davidovi\v{c}, Tom\'{a}\v{s} and Kallweit, Simon and Lefohn, Aaron},
    title = {Real-time Neural Appearance Models},
    year = {2024},
    issue_date = {June 2024},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    volume = {43},
    number = {3},
    doi = {10.1145/3659577},
    journal = {ACM Trans. Graph.},
    month = {jun},
    articleno = {33},
    numpages = {17}
}