Fast and High-quality Image Denoising via Malleable Convolutions

Y. Jiang, B. Wronski, B. Mildenhall, J. T. Barron, Z. Wang, T. Xue, paper under review

Project website, arXiv preprint

To achieve spatial-varying processing without significant overhead, we present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. The weights of MalleConv are dynamically produced by an efficient predictor network capable of generating content-dependent outputs at specific spatial locations. Unlike previous works, MalleConv generates a much smaller set of spatially-varying kernels from input, which enlarges the network’s receptive field and significantly reduces computational and memory costs. These kernels are then applied to a full-resolution feature map through an efficient slice-and-conv operator with minimum memory overhead.

Procedural Kernel Networks

B. Wronski, arXiv preprint / technical report

https://arxiv.org/abs/2112.09318

In this work, we introduce Procedural Kernel Networks (PKNs), a family of machine learning models which generate parameters of image filter kernels or other traditional algorithms. A lightweight CNN processes the input image at a lower resolution, which yields a significant speedup compared to other kernel-based machine learning methods and allows for new applications. The architecture is learned end-to-end and is especially well suited for a wide range of low-level image processing tasks, where it improves the performance of many traditional algorithms. We also describe how this framework unifies some previous work applying machine learning for common image restoration tasks.

Image Stylization: From Predefined to Personalized, IET Research Journal 2020

I.Garcia-Dorado, P. Getreuer, B. Wronski, P. Milanfar, IET Research Journals 2020 special edition.

https://arxiv.org/abs/2002.10945 Arxiv preprint

We present a framework for interactive design of new image stylizations using a wide range of predefined filter blocks. Both novel and off-the-shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters.

Handheld Multi-Frame Super-Resolution, ACM Siggraph 2019 Technical Paper

B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.K. Liang, M. Levoy, and P. Milanfar, * to appear in ACM Transactions on Graphics, Vol. 38, No. 4, Article 28, July 2019 (SIGGRAPH 2019)

https://arxiv.org/abs/1905.03277

We present a multi-frame super-resolution algorithm that supplants the need for demosaicing in a camera pipeline by merging a burst of raw images. In the above figure we show a comparison to a method that merges frames containing the same-color channels together first, and is then followed by demosaicing (top). By contrast, our method (bottom) creates the full RGB directly from a burst of raw images. This burst was captured with a hand-held mobile phone and processed on the device. Note in the third (red) inset that the demosaiced result exhibits aliasing (Moiré), while our result takes advantage of this aliasing, which changes on every frame in the burst, to produce a merged result in which the aliasing is gone but the cloth texture becomes visible.

Volumetric fog: Unified, compute shader based solution to atmospheric scattering, ACM Siggraph 2014

Bartlomiej Wronski, ACM Siggraph 2014

PPTX Version – 83MB (with movies)

PDF Version with presenter notes – 6MB

This talk presents “Volumetric Fog”, a novel technique developed by Ubisoft Montreal for Assassin’s Creed 4: Black Flag for next-gen consoles and PCs.

The technique addresses problem of calculating in unified, coherent and optimal way various atmospheric effects related to the atmospheric scattering:

• Fog, smoke and haze with varying participating media density
• „God rays”
• Light-shafts
• Volumetric lighting and shadows

Developed technique supports varying density of participating media, multiple light sources, is compatible with both deferred and forward shading and is faster than existing ray marching approaches.

Assassin’s Creed 4: Road to Next-gen Graphics, GDC 2014

Bartlomiej Wronski, GDC 2014

Presentation

PDF Version – 4.11MB

PDF Version with presenter notes – 9.63MB

PPTX Version without the movies – 7MB

PPTX Version with the movies – 164MB

Movies separate download

Global Illumination – time of day cycle – 17 MB

Volumetric fog – animated plants shadows – 21 MB

Volumetric fog – local lights support – 39 MB

Screenspace reflections – on / off video – 79MB

This talk will describe the novel techniques and easy-to-integrate effects of Assassin’s Creed IV that contribute to the next-gen look. It will provide attendees with basic information about porting various GPU effects to next-gen consoles. The talk is divided into four parts. First, there will be a description of deferred normalized irradiance probes – a GI technique that is based on FarCry 3 deferred radiance transfer volumes. The next part will describe volumetric fog – a novel technique that simulates various light scattering phenomena through the use of compute shaders and light accumulation in volumetric textures. The third part will include information about atmospheric and material effects such as GPU-simulated rain particles and raymarched screenspace reflections that can easily contribute to the next-gen look of the game without many content or pipeline changes. Finally, there will be a brief description of AMD Southern Islands architecture and practical lessons learned while developing/porting/optimizing those GPU effects to PlayStation 4 and Xbox One.

Assassin’s Creed 4: Lighting, weather and atmospherics, Digital Dragons 2014

Bartlomiej Wronski, Digital Dragons 2014

PPTX Version, 226MB – but worth it (tons of videos!)

PPTX Version with extremely compressed videos, 47MB

PDF Version with sparse notes, 6MB

PDF Version, no notes, 7MB

Other publications

GPU Pro 6: Advanced Rendering Techniques

A K Peters/CRC Press
Published September 11, 2015
Reference – 586 Pages – 279 Color Illustrations
ISBN 9781482264616 – CAT# K24427

Two articles:

Deferred Normalized Irradiance Probes, John Huelin, Benjamin Rouveyrol, and Bartlomiej Wronski

Volumetric Fog and Lighting, Bartlomiej Wronski