Stochastic Texture Filtering

M. Fajardo, B. Wronski, M. Salvi, M. Pharr, arXiv preprint

arXiv Link

2D texture maps and 3D voxel arrays are widely used to add rich detail to the surfaces and volumes of rendered scenes, and filtered texture lookups are integral to producing high-quality imagery. We show that filtering textures after evaluating lighting, rather than before BSDF evaluation as is current practice, gives a more accurate solution to the rendering equation. These benefits are not merely theoretical but are apparent in common cases. We further show that stochastically sampling texture filters is crucial for enabling this approach, which has not been possible previously except in limited cases. Stochastic texture filtering offers additional benefits, including efficient implementation of high-quality texture filters and efficient filtering of textures stored in compressed and sparse data structures, including neural representations. We demonstrate applications in both real-time and offline rendering and show that the additional stochastic error is minimal. Furthermore, this error is handled well by either spatiotemporal denoising or moderate pixel sampling rates.

Random-Access Neural Compression of Material Textures

K. Vaidyanathan*, M. Salvi*, B. Wronski*, T. Akenine‑Möller, P. Ebelin, A. Lefohn, Accepted to ACM Siggraph 2023 (^*Equal contributors)

Project website

We propose a novel neural compression technique specifically designed for material textures. We unlock two more levels of detail, i.e., 16X more texels, using low bitrate compression, with image quality that is better than advanced image compression techniques, such as AVIF and JPEG XL. Our method allows on-demand, real-time decompression with random access similar to block texture compression on GPUs, enabling compression on disk and memory. The key idea behind our approach is compressing multiple material textures and their mipmap chains together using a small neural network, that is optimized for each material, to decompress them. We use a custom training implementation to achieve practical compression speeds, whose performance surpasses that of general frameworks, like PyTorch, by an order of magnitude.

Fast and High-quality Image Denoising via Malleable Convolutions

Y. Jiang, B. Wronski, B. Mildenhall, J. T. Barron, Z. Wang, T. Xue, Accepted to ECCV 2022

Project website, arXiv preprint

To achieve spatial-varying processing without significant overhead, we present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. The weights of MalleConv are dynamically produced by an efficient predictor network capable of generating content-dependent outputs at specific spatial locations. Unlike previous works, MalleConv generates a much smaller set of spatially-varying kernels from input, which enlarges the network’s receptive field and significantly reduces computational and memory costs. These kernels are then applied to a full-resolution feature map through an efficient slice-and-conv operator with minimum memory overhead.

Procedural Kernel Networks

B. Wronski, arXiv preprint / technical report

https://arxiv.org/abs/2112.09318

In this work, we introduce Procedural Kernel Networks (PKNs), a family of machine learning models which generate parameters of image filter kernels or other traditional algorithms. A lightweight CNN processes the input image at a lower resolution, which yields a significant speedup compared to other kernel-based machine learning methods and allows for new applications. The architecture is learned end-to-end and is especially well suited for a wide range of low-level image processing tasks, where it improves the performance of many traditional algorithms. We also describe how this framework unifies some previous work applying machine learning for common image restoration tasks.

Image Stylization: From Predefined to Personalized, IET Research Journal 2020

I.Garcia-Dorado, P. Getreuer, B. Wronski, P. Milanfar, IET Research Journals 2020 special edition.

https://arxiv.org/abs/2002.10945 Arxiv preprint

We present a framework for interactive design of new image stylizations using a wide range of predefined filter blocks. Both novel and off-the-shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters.

Handheld Multi-Frame Super-Resolution, ACM Siggraph 2019 Technical Paper

B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.K. Liang, M. Levoy, and P. Milanfar, * to appear in ACM Transactions on Graphics, Vol. 38, No. 4, Article 28, July 2019 (SIGGRAPH 2019)

https://arxiv.org/abs/1905.03277

We present a multi-frame super-resolution algorithm that supplants the need for demosaicing in a camera pipeline by merging a burst of raw images. In the above figure we show a comparison to a method that merges frames containing the same-color channels together first, and is then followed by demosaicing (top). By contrast, our method (bottom) creates the full RGB directly from a burst of raw images. This burst was captured with a hand-held mobile phone and processed on the device. Note in the third (red) inset that the demosaiced result exhibits aliasing (Moiré), while our result takes advantage of this aliasing, which changes on every frame in the burst, to produce a merged result in which the aliasing is gone but the cloth texture becomes visible.

Volumetric fog: Unified, compute shader based solution to atmospheric scattering, ACM Siggraph 2014

Bartlomiej Wronski, ACM Siggraph 2014

PPTX Version – 83MB (with movies)

PDF Version with presenter notes – 6MB

This talk presents “Volumetric Fog”, a novel technique developed by Ubisoft Montreal for Assassin’s Creed 4: Black Flag for next-gen consoles and PCs.

The technique addresses problem of calculating in unified, coherent and optimal way various atmospheric effects related to the atmospheric scattering:

• Fog, smoke and haze with varying participating media density
• „God rays”
• Light-shafts
• Volumetric lighting and shadows

Developed technique supports varying density of participating media, multiple light sources, is compatible with both deferred and forward shading and is faster than existing ray marching approaches.

Assassin’s Creed 4: Road to Next-gen Graphics, GDC 2014

Bartlomiej Wronski, GDC 2014

Presentation

PDF Version – 4.11MB

PDF Version with presenter notes – 9.63MB

PPTX Version without the movies – 7MB

PPTX Version with the movies – 164MB

Movies separate download

Global Illumination – time of day cycle – 17 MB

Volumetric fog – animated plants shadows – 21 MB

Volumetric fog – local lights support – 39 MB

Screenspace reflections – on / off video – 79MB

This talk will describe the novel techniques and easy-to-integrate effects of Assassin’s Creed IV that contribute to the next-gen look. It will provide attendees with basic information about porting various GPU effects to next-gen consoles. The talk is divided into four parts. First, there will be a description of deferred normalized irradiance probes – a GI technique that is based on FarCry 3 deferred radiance transfer volumes. The next part will describe volumetric fog – a novel technique that simulates various light scattering phenomena through the use of compute shaders and light accumulation in volumetric textures. The third part will include information about atmospheric and material effects such as GPU-simulated rain particles and raymarched screenspace reflections that can easily contribute to the next-gen look of the game without many content or pipeline changes. Finally, there will be a brief description of AMD Southern Islands architecture and practical lessons learned while developing/porting/optimizing those GPU effects to PlayStation 4 and Xbox One.

Assassin’s Creed 4: Lighting, weather and atmospherics, Digital Dragons 2014

Bartlomiej Wronski, Digital Dragons 2014

PPTX Version, 226MB – but worth it (tons of videos!)

PPTX Version with extremely compressed videos, 47MB

PDF Version with sparse notes, 6MB

PDF Version, no notes, 7MB

Other publications

GPU Pro 6: Advanced Rendering Techniques

A K Peters/CRC Press
Published September 11, 2015
Reference – 586 Pages – 279 Color Illustrations
ISBN 9781482264616 – CAT# K24427

Two articles:

Deferred Normalized Irradiance Probes, John Huelin, Benjamin Rouveyrol, and Bartlomiej Wronski

Volumetric Fog and Lighting, Bartlomiej Wronski