Digital Compositing for Film and Video

Steve Wright is a visual effects pioneer and a 20-year veteran of visual effects compositing on over 70 feature films and many broadcast television commercials. With extensive production experience and a knack for the math and science of visual effects he is a world-recognized expert on visual effects compositing. Since 2005 he has been a master trainer in compositing visual effects, providing staff training to over 25 visual effects studios around the world including Pixar Animation Studios, Disney Feature Animation, Troublemaker Studios, New Deal Studios, and Reliance MediaWorks, along with many others. He has also trained over 1,000 artists in compositing. Visit Steve’s training website at www.fxecademy.com

About the Author

Acknowledgements

Preface

Chapter 1 - Getting Started

1.1 How this Book is Organized

1.2 Web Content

1.3 What’s New in the 4th Edition

1.4 Gold Mines

1.5 Tool Conventions

1.5.1 The Slice Tool

1.5.2 Flowgraphs

1.5.3 Color Lookup Tables (LUTs)

1.5.4 Nuke

1.6 Data Conventions

1.6.1 Floating Point Data

1.6.1.1 Banding

1.6.1.2 Clipping

1.6.2 Linear Light Space

1.6.3 HDR Images

1.6.4 Stops

PART I MAKING A GREAT COMPOSITE

Chapter 2 - Pulling Keys

2.1 Lumakeys

2.1.1 How Lumakeys Work

2.1.2 Making Your Own Luminance Image

2.1.2.1 Variations on the Luminance Equations

2.1.2.2 Non-luminance Monochrome Images

2.1.3 Making Your Own Lumakeyer

2.2 Chromakeys

2.2.1 How Chroma Keys Work

2.2.2 Making Your Own Chroma Keyer

2.2.3 Making a 3D Chroma Keyer

2.3 Difference Mattes

2.3.1 How Difference Mattes Work

2.3.2 Making Your Own Difference Matte

2.3.2.1 Making the Difference Image

2.3.2.2 Making the Difference Matte

2.4 Bump Mattes

2.5 Color Difference Keys

2.6 The "Blur and Grow" Technique

2.7 Rotoscoping

2.7.1 Control Point Coherency

2.7.2 Shape Breakdown

2.7.2.1 Hierarchical Articulation

2.7.2.2 Organization

2.7.3 Bezier or B-spline?

2.7.4 Keyframe Strategies

2.7.4.1 On 2’s

2.7.4.2 Binary Multiples

2.7.4.3 Bifurcation

2.7.4.4 Motion Extremes

2.7.5 Motion Blur

2.7.5.1 Spline Placement

2.7.5.2 Edge Decontamination

2.7.6 Inspection

Chapter 3 - Working with Keyers

3.1 Keyers

3.2 How Keyers Work

3.2.1 Calculating the Color Difference Matte

3.2.1.1 The Theory

3.2.1.2 Pulling the Raw Matte

3.2.1.3 A Simplified Example

3.2.1.4 A Slightly More Realistic Case

3.2.1.5 And Now, the Real World

3.2.1.6 Matte Edge Penetration

3.2.2 Scaling the Raw Matte

3.3 The After Effects Keyer

3.3.1 Step-by-step Procedure

3.3.2 Flowgraph of the After Effects Keyer

3.4 Typical Greenscreen Problems

3.4.1 Overexposed

3.4.2 Underexposed

3.4.3 Impure Greenscreens

3.4.4 Uneven Lighting

3.5 Preprocessing the Greenscreen

3.5.1 Denoise and Degrain

3.5.2 Screen Leveling

3.5.3 Local Suppression

3.5.4 Channel Clamping

3.5.5 Channel Shifting

3.5.6 Screen Correction

3.5.6.1 Step-by-step Procedure

3.5.6.2 Pictographic Flow Chart

3.5.6.3 Flowgraph of the Screen Correction Procedure

3.5.6.4 How to Create a Clean Greenscreen

Chapter 4 - Refining Mattes

4.1 Gamma Slamming

4.2 Garbage Mattes

4.2.1 Pre-matting

4.2.2 Post-matting

4.3 Filtering the Matte

4.3.1 Noise Suppression with a Median Filter

4.3.2 Softer Edges

4.3.3 Controlling the Blur Operation

4.3.3.1 The Blur Radius

4.3.3.2 The Blur Percentage

4.3.3.3 Masking the Blur

4.4 Adjusting the Matte Size

4.4.1 Eroding a Matte with Blur and Scale

4.4.2 Dilating a Matte with Blur and Scale

4.4.3 Blurring Out

4.4.4 Sculpting Edges

4.5 Edge Masks

Chapter 5 - Spill Suppression

5.1 Sources of Spill

5.2 The Despill Operation

5.3 Despill Algorithms

5.3.1 Green Limited by Red

5.3.1.1 Implementing the Algorithm

5.3.1.2 The Spillmap

5.3.2 Green Limited by the Average of Red and Blue

5.3.3 An Adjustable Despill

5.3.4 What About Blue Spill?

5.3.5 Refining the Despill

5.3.5.1 Channel shifting

5.3.5.2 Spillmap Scaling

5.3.5.3 Mixing Despills

5.3.5.4 Matting Despills Together

5.4 The Unspill Operation

5.4.1 How to Set It Up

5.4.2 Grading to the Backing Color

5.5 Despill Artifacts

5.5.1 Finding the Artifacts

5.5.2 Hue Shifts

5.5.3 Dark Edges

5.5.4 Fixing Despill Artifacts

5.6 Edge Grading

5.7 Edge Extension

Chapter 6 - the Composite

6.1 Premultiply vs. Unpremultiply

6.1.1 Premultiply

6.1.2 Unpremultiply

6.1.3 The Double Premultiply

6.2 The Composite

6.2.1 The Over Composite

6.2.2 The KeyMix Composite

6.2.3 The AddMix Composite

6.2.3.1 How It Works

6.2.3.2 How to Build It

6.2.3.3 How to Use It

6.2.4 The Processed Foreground Method

6.2.4.1 The Workflow

6.2.4.2 What to Watch Out For

6.3 Compositing With a Keyer

6.3.1 Soft Comp/Hard Comp

6.3.2 "Cut and Paste" Keyer Compositing

6.4 Compositing Outside the Keyer

6.4.1 The Single Key

6.4.2 The Uberkey

6.4.3 Soft Key/Hard Key

6.4.4 The Additive Keyer

6.5 Stereo Compositing

6.5.1 Anaglyph

6.5.2 Stereopsis

6.5.3 Stereoscopy

6.5.4 The Stereo Conversion Process

6.5.5 Depth Grading

6.5.5.1 Scene Transition

6.5.5.2 The Dashboard Effect

6.5.5.3 Window Violation

6.5.5.4 Miniaturization

6.5.5.5 Divergence

6.5.6 Stereo Compositing

6.5.6.1 Dual View Display

6.5.6.2 Split and Join Views

6.5.6.3 Disparity Maps

Chapter 7 - Compositing CGI

7.1 Multi-pass CGI Compositing

7.1.1 Process Verification for Your Renderer

7.1.2 Render Passes

7.1.3 Lighting Passes

7.1.3.1 Render Passes Workflow

7.1.3.2 Beauty Pass Workflow

7.1.4 AOVs

7.1.5 ID Passes

7.1.6 Normals Relighting

7.2 EXR File Format

7.2.1 Film Scans

7.2.2 Linear Lightspace

7.2.3 Arbitrary Image Channels

7.3 HDR Images

7.4 Deep Compositing

7.4.1 Deep Images

7.4.2 The Layering Complexity Problem

7.4.3 The Depth Compositing Edge Problem

7.4.4 The Re-rendering Problem

7.4.5 Deep Compositing with Live Action

Chapter 8 - 3D Compositing

8.1 A Short Course in 3D

8.1.1 the 3D Coordinate System

8.1.2 Vertices

8.1.3 Meshes

8.1.4 Surface Normals

8.1.5 UV Coordinates

8.1.6 Map Projection

8.1.7 UV Projection

8.1.8 3D Geometry

8.1.9 Geometric Transformations

8.1.10 Geometric Deformations

8.1.10.1 Image Displacement

8.1.10.2 Noise Displacement

8.1.10.3 Deformation Lattice

8.1.11 Point Clouds

8.1.12 Lights

8.1.13 Shaders

8.1.14 Reflection Mapping

8.1.15 Ray Tracing

8.1.16 Image-based Lighting

8.1.17 Cameras

8.2 3D Compositing

8.2.1 3D compositing from 2D images

8.2.2 Pan and Tile

8.2.3 Camera Projection

8.2.4 Multiplane Shots

8.2.5 Set Extension

8.2.6 3D Backgrounds

8.3 Alembic Geometry

8.3.1 The Simple Case

8.3.2 Scenegraphs

8.3.3 Advantages Over FBX

8.4 Camera Tracking

8.4.1 Step 1 - Feature Tracking

8.4.2 Step 2 - The Solve

8.4.3 Step 3 – Build the Scene

8.4.4 Placing the Geometry

8.4.5 A Large Outdoor Scene

PART II THE QUEST FOR REALISM

Chapter 9 - Color Correction

9.1 The Behavior of Light

9.1.1 The Inverse Square Law

9.1.2 Diffuse Reflections

9.1.3 Specular Reflections

9.1.4 Bounce Light

9.1.5 Scattering

9.2 Gamma

9.2.1 The Math

9.2.2 Why Do We Need Gamma?

9.3 The Affect of Color Operations

9.3.1 Lift

9.3.2 Gamma

9.3.3 Gain

9.3.4 Offset

9.3.5 Saturation

9.3.6 Color Grading vs. Color Correcting

9.3.7 Increasing Contrast with the "S" Curve

9.3.8 Histograms

9.3.9 Channel Swapping

9.3.10 Premultiply vs. Unpremultiply - Again

9.4 Matching the Light Space

9.4.1 Brightness and Contrast

9.4.1.1 Matching the Black and White Points

9.4.1.2 Matching the Midtones

9.4.1.3 Gamma Slamming

9.4.2 Matching Color

9.4.2.1 Grayscale Balancing

9.4.2.2 Flesh Tones

9.4.2.3 The "Constant Green" Method of Color Correction

9.4.2.4 Daylight

9.4.2.5 Specular Highlights

9.4.3 Lighting Direction

9.4.4 Quality of Light Sources

9.4.4.1 Creating Softer Lighting

9.4.4.2 Creating Harsher Lighting

9.4.5 Non-linear Gradients for Color Correction

9.4.6 The DI Process

9.4.7 A Checklist

Chapter 10 - Sweetening the Comp

10.1 Layer Integration

10.2 Interactive Lighting

10.3 Edge Blending

10.4 Light Wrap

10.5 Creating Shadows

10.5.1 Edge Characteristics

10.5.2 Density

10.5.3 Color

10.5.4 Faux Shadows

10.5.5 Shadow Warping

10.5.6 Contact Shadows

10.6 Atmospheric Haze

10.7 Adding a Glow

10.8 Grain Management

10.8.1 Grain Characteristics

10.8.2 Regraining Techniques

10.8.2.1 Regrain Tool

10.8.2.2 Lifted Grain

10.8.2.3 Grain Rescue

10.8.3 Grain Management Workflows

10.8.3.1 Live Over Live

10.8.3.2 Live Over CGI

10.8.3.3 CGI Over Live

10.8.3.4 CGI Over CGI

10.8.3.5 Still Photos

10.9 Managing Clipping

Chapter 11 - Camera Effects

11.1 Lens Effects

11.1.1 Lens Distortion

11.1.2 Depth of Field

11.1.3 Vignetting

11.1.4 Lens Defects

11.1.4.1 Spherical Aberration

11.1.4.2 Astigmatism

11.1.4.3 Chromatic Aberration

11.1.5 Glows and Flares

11.1.5.1 Lens Flare

11.1.5.2 Lens Filter Flare

11.1.5.3 Diffraction Glows

11.1.5.4 Veiling Glare

11.1.6 Grain

11.2 Lens Distortion Workflows

11.2.1 CGI Over Live Action

11.2.2 Live Action Over CGI

11.2.3 CGI Over CGI

11.2.4 Live Action Over Live Action

11.3 Matching the Focus

11.3.1 Using a Blur for Defocus

11.3.2 How to Simulate a Defocus

11.3.3 Sharpening

11.3.3.1 Sharpening Operations

11.3.3.2 Unsharp Masks

11.3.3.3 Making Your Own Unsharp Mask

11.4 Rolling shutter

PART III THINGS YOU SHOULD KNOW

Chapter 12 - Digital Color

12.1 Color Spaces

12.1.1 Primary Chromaticities

12.1.2 Units of Measure

12.1.3 Transfer Function

12.1.4 Gamut

12.1.5 HSV and HSL

12.1.6 Log and Linear

12.2 Working in Linear

12.2.1 What Exactly Is Linear?

12.2.2 Color Operations

12.2.3 Transformations and Filtering

12.2.4 CGI

12.3 Metadata

12.4 OpenColorIO

12.5 ACES Color Management

12.5.1 The ACES Workflow

12.5.2 The ACES Gamut

12.5.3 What About Video Productions?

Chapter 13 - Image Blending

13.1 Image Blending in Linear Light Space

13.1.1 Image-blending Operations

13.1.2 Compositing Operations

13.1.3 Matching the Look of sRGB in Linear

13.1.3.1 All sRGB Color Space

13.1.3.2 sRGB Within Linear

13.2 Alpha Compositing Operations

13.3 Image-blending Operations

13.3.1 The Screen Operation

13.3.1.1 Adjusting the Appearance

13.3.2 The Weighted Screen Operation

13.3.3 Multiply

13.3.3.1 Adjusting the Appearance

13.3.4 Maximum

13.3.5 Minimum

13.3.6 Absolute Difference

13.4 Adobe Photoshop Blending Modes

13.4.1 Simple Blending Modes

13.4.2 Complex Blending Modes

13.5 Slot Gags

13.6 Retiming Clips

13.6.1 Constant Speed Changes

13.6.2 Variable Speed Changes

13.6.3 Interpolation Methods

13.6.3.1 Nearest Neighbor

13.6.3.2 Frame Average

13.6.3.3 Motion Estimation

13.7 VR Stitching

13.7.1 Workflow Overview

13.7.2 Removing Lens Distortion

13.7.3 Building a Matching Computer Rig

13.7.4 Projecting Onto the Panosphere

13.7.5 The Stitching Process

13.7.6 Coping with Parallax

13.7.7 Exposure Correction

13.7.8 Visual Effects

Chapter 14 - Transforms and Tracking

14.1 Geometric Transforms

14.1.1 2D Transforms

14.1.1.1 Translation

14.1.1.2 Rotation

14.1.1.2.1 Pivot Points

14.1.1.3 Resize vs. Scale

14.1.1.4 Skew

14.1.1.5 Corner Pinning

14.1.2 Managing Motion Blur

14.1.2.1 Transform Motion Blur

14.1.2.2 Motion UV Motion Blur

14.1.2.3 Speed Changes

14.1.3 3D Transforms

14.1.4 Filtering

14.1.4.1 The Effects of Filtering

14.1.4.2 Twinkling Starfields

14.1.4.3 Choosing a Filter

14.1.5 Lining Up Images

14.1.5.1 Offset Mask Lineup Display

14.1.5.2 Edge-detection Lineup Display

14.1.5.3 The Pivot Point Lineup Procedure

14.2 Image Displacement

14.3 Warps and Morphs

14.3.1 Mesh Warps

14.3.2 Spline Warps

14.3.3 Morphs

14.3.4 Tips, Tricks, and Techniques

14.4 Point Tracking

14.4.1 The Tracking Operation

14.4.1.1 Selecting Good Tracking Targets

14.4.1.2 Bad Tracking Targets

14.4.1.3 Tracker Enable/Disable

14.4.1.4 Offset Tracking

14.4.1.5 Keep Shape and Follow Shape

14.4.1.6 Pre-processing the Clip

14.4.1.7 Coping with Grain

14.4.1.8 Tracking Workflow

14.4.1.9 Cleaning up Tracking Data

14.4.1.10 The Stability Test

14.4.1.11 Reasons for Failure

14.4.2 Match-Move

14.4.2.1 2D Transforms

14.4.2.2 Corner Pinning

14.4.3 Stabilizing

14.4.3.1 The Repo Problem

14.4.3.2 Motion Smoothing

14.4.3.3 Stabilizing For Rotoscoping

14.5 Planar Tracking

14.5.1 The Planar Grid

14.5.2 Drift Correction

14.5.3 Exporting Data

14.5.4 Roto Assist

Chapter 15 - Digital Images

15.1 HD Video

15.1.1 Frame Formats

15.1.2 Anamorphic video

15.1.3 Scan Modes

15.1.4 Working with Interlaced Video

15.1.4.1 De-interlacing

15.1.4.2 Scan Line Interpolation

15.1.4.3 Field Averaging

15.1.5 Color Subsampling

15.1.6 Keying with 4:2:2 Video

15.1.7 Frame Rates

15.1.7.1 24, 25, 30, 60fps

15.1.7.2 23.98, 29.97, 59.94fps

15.1.8 Timecodes

15.1.9 Video File Formats

15.1.10 Telecine

15.1.10.1 The 3:2 Pull-down

15.1.10.2 The 3:2 Pull-up

15.2 Digital Cinema Images

15.2.1 Digital Camera Advantages

15.2.2 The Bayer Array

15.2.3 Sensor Crop

15.2.4 HFR – High Frame Rate

15.2.5 The DCI

15.3 Film Scans

15.3.1 Grain

15.3.2 The "Safe-to" Window

15.3.3 Apertures

15.3.4 Aspect Ratios

15.3.5 Film Formats

15.3.5.1 Full Aperture

15.3.5.2 Academy Aperture

15.3.5.3 Super 35 Formats

15.3.5.4 Cinemascope

15.3.5.5 Working with Cscope

15.3.5.6 "3-perf" Film

15.3.5.7 VistaVision

15.3.5.8 65mm/70mm

15.3.5.9 IMAX

15.4 Log Images

15.4.1 What Are Log Images?

15.4.2 Why We Need Log Images

15.4.2.1 Human Vision

15.4.2.2 Data Compression

15.4.2.3 Working with Log Images

15.5 Light Field Cinematography

15.5.1 How It Works

15.5.2 The Impact on Visual Effects

15.5.2.1 Deep Images

15.5.2.2 Arbitrary Depth of Field

15.5.2.3 Depth Maps

15.5.2.4 Stereo Through a Single Lens

15.5.2.5 Volumetric Optical Flow

15.5.2.6 Position Pass

15.5.2.7 Point Clouds

15.5.2.8 Mattes

15.5.2.9 Normals and Normal Relighting

15.5.2.10 Camera Tracking

15.5.3 When, and How Much?

Glossary

Index