The VES Handbook of Visual Effects

Jeffrey A. Okun, VES, is an award winning Visual Effects Supervisor, having worked on a large number of feature films, commercials and television shows. He started in the inudstry in 1976, and he is currently a member and Fellow of the VES and The Academy of Motion Pictures, Arts and Sciences as well as the American Society of Cinematographers (ASC), the Television Academy and the Editors Guild. He was the chair of the VES for 9 years and 1st vice chair for 16 years, and chair of the L.A. Section for 2 years. He created the VES awards along with Tim McGovern and Kim Lavery, VES. Okun created a visual effects tracking and bidding software in 1992 that is still in wide use within the industry today, as well as the revolutionary visual effects techniques dubbed the "PeriWinkle Effect" (an underwater blue screen technique) and the "Pencil Effect" (accurately predicts the final visual effects count and budget). Susan Zwerman, VES has been a member of the VES since 1998. She is a highly respected Visual Effects Producer who has been producing visual effects for more than 25 years. Zwerman is also a well-known seminar leader and author. As Chair of the DGA’s UPM/AD VFX Digital Technology Committee, Susan emphasizes the importance of the visual effects teams to DGA members at visual effects seminars organized under her guidance. In 2013, Susan received the Frank Capra Achievement Award in recognition of career achievement and service to the industry and the Directors Guild of America. She is also a member of the Academy of Motion Picture Arts and Sciences, the Producers Guild of America and a Fellow of the VES.

Coming Soon …

Chapter 1: Introduction

First, Some Ground Rules

Now, the Introduction

Why Use Visual Effects

Chapter 2: Pre-Production /Preparation

Overview

Breaking Down a Script – Budgeting

Ballpark Budget

More Detailed Budgets

Bidding

Plate Photography

Temp Screenings

Reviewing Bids

Contracts

Rebidding during Shooting

Rebidding in Post

Monitoring the Budget and Schedule

Keeping the Budget Down

Working with the Director and Producer

Demo Reel

The Meeting

Moving Forward

Production Departments

Production Design

Camera

Working with the Cinematographer

Special Effects

Stunts

Wardrobe

Makeup

Production

Visual Effects

Editorial

Locations

Production Meeting

Designing Visual Effects Shots

Guidelines for Directors

Storyboards

Previs

Objective of the Shot

Concept Art

Continuity

Photorealism

Original Concepts

Budget

Reality and Magic

Camera Angles

Framing

Scale

Detail

Speed

Scaled Images

Depth of Field

Sequence of Shots

Camera Motion

Less Is More

Action Pacing

CG Characters

Creatures and Character Design

Powers of 10 Shots

Visual Effects Techniques

Technique Considerations

Additional Suggestions for Determining Techniques

Development of Previs Techniques

History and Background

What is Previs and Other Forms of Visualization?

What is Previs?

Different Types of Visualization and When to Use Them

Visualization: The New Essential

The Application of Previs: Who Benefits and How?

The Benefits of Previs

Project Types

Post-Visualization

What is Post-Visualization?

Why Use Postvis?

Who Does Postvis?

Cautions and Suggestions for Good Practice

The Perils of Previs!

Passing the Work On

The Role of the VFX Supervisor in Previs

Previs: Advanced Techniques

Visualization Usefulness

VR as a Tool

Visualization in Engine

Render in Engine

Visualization in Real Time

AR as a Tool

Camera Angle Projection

Introduction

Drawing What the Lens Sees

Techvis

What Is Techvis?

Who is Techvis for?

Virtual Production

What is Virtual Production?

How is Virtual Production Used?

Chapter 3: Acquisition / Shooting

Working on Set

Common Types of Special Effects

What are Special Effects?

A Brief History of Special Effects

The Special Effects Coordinator

Working with the Visual Effects

Visual Effects in Service to SFX

Special Effects Design and Planning

Storyboards and Previs

The Elements: Rain, Wind, and Snow and Ice

Smoke, Fire, and Pyrotechnics

Mechanical Effects

Flying Wire Rigs and Stunts

Safety

Greenscreen and Bluescreen Photograph

Best Practices and Otherwise

Overview

Function of the Backing

Negative Scanning and Digital Conversion

Backing Uniformity and Screen Correction

The Alpha Channel

The Processed Foreground

The Composite

Recommended Specifications and Practices

How to Expose a Green Screen Shot, and Why

Setting Screen Brightness

Choosing the Backing Color

Floor Shots, Virtual Sets

Foreground Lighting

Controlling Spill Light

Lighting Virtual Sets

Tracking Markers

On-Set Preview

Cameras for Blue Screen or Green Screen Photography

Underwater Photography

On-Set Data Acquisition

Camera Report

Tracking Markers

Props for the Actors

Cyberscanning

Digital Photos

Lidar/Laser Scanning

Lens Distortion Charts

HDRI and Chrome Balls

Lidar Scanning and Acquisition

On-Set 3D Scanning Systems

Types of Technology

Lidar

Photogrammetry

Prop Scanners

Lighting Data

Gathering Lighting Data

Beware of False Savings!

Using Conventional Still Cameras

Reference Shooting Considerations

Clean Plates

Shooting the Clean Plate

Locked-Off Camera

Moving Camera

Other Issues

Post-Process

Alternates without Clean Plates

Other Uses for Clean Plates

Monster Sticks

On-Set Animation Capture: Witness Cam (IBMC)

Wireless Non-Video Motion Capture

Factors Affecting Witness Cameras

Dealing with the Data in Post-Production

Camera Tracking for Real-Time Visualization

Camera Tracking Pre-Production

The Camera Department

Prior to Shooting

Current Tracking Systems in Use

Triangulation As a Method of Recording Camera Data

Camera/Subject Positional Information

Basics: The Toolkit

Basics: Nodal Point

Photographic Reference

How to Proceed

Shooting Video as a Reference

Rules, Setup, and Testing

Do a Complete Test Shot!

Why Run Through Example or Test Shots?

Digital Cinematography

Digital Definitions

High Dynamic Range (HDR)

Lens Metadata

Look Management

The Recording System

Lens Mapping for VFX

VFX Photography

The Camera Array

Designing an Array Shot

Technicians

Shoot Day

Special Techniques

Post

The Future

Filming Live-Action Plates to be Used in VFX

Camera Position (Station Point)

Angle of View

Lighting Considerations

Camera Tilt

Background Quality

Moving Plates

Scouting the Camera Positions

A Case Study

Camera Cars

Camera Car Safety Issues

Purpose-Built Crane Cars

Vibration and Camera Stabilization

Road Speed

Precautions

Panoramic Rigs

On the Water

Air to Air

Cable Systems

Shooting Elements for Compositing

What Is an Element?

Stock Footage

Types of Elements

Generic versus Shot-Specific Elements

Determining Element Needs

Cheating

Backgrounds

Black Backgrounds

Line-Up

Camera Format Considerations

Assorted Methods for Shooting Elements

High-Speed Photography and Filming Elements

Cameras

Technicians

Director of Photography

Lighting

Application

Locking Down the Camera

Video Assist

Post

Supervising Motion Control

What is Motion Control?

Performance Choreography

Multiple-Pass Photography

Scaling

Import and Export of Camera Move Data

The Data

Types of Motion Control Systems

Motion Control Software

Camera Types

Sync and Phase

Dealing with Production

Acquisition of Motion / Still Photographic Textures for Mapping onto CG

Panoramic Backgrounds

Tiled Stills

Motion Tiling and Synchronous Plates

Practical Considerations

Stills for Textures and Lighting

Stop-Motion

Evolution of Stop-Motion Photography

The Time Required to Shoot in Stop-Motion

Preparation before Shooting

Setting up a Shooting Space for Stop-Motion

Use of Motion Control in Stop-Motion

Useful Caveats

Evolution of a Shot

Use of Stop-Motion in Visual Effects

Chapter 4: Performance and Motion Capture

What is Motion Capture?

Is Motion Capture Right for a Project?

The Mocap Look

Technical Specifications

Entry Point

Budget

Which Technology is Right for a Project?

Gauging a Project’s Needs and Constraints

Passive Retroreflective Optical

Active Optical

Inertial

Structured Light

Dense Stereo Reconstruction

Bend Sensors

Preparing for Motion Capture

Actors

Motion Capture Suits

Marker Placement – Body

Marker Placement – Face

Rigging for Motion Capture

Shot List

Technology Considerations

Hardware

The Strobe

Markers

Lenses

Filter

Image Sensor

Onboard Processor

Inputs/Outputs

Setup

Software

Acquisition

Calibration

Post-Processing

Reconstruction

Labeling

Gap Filling

Cleaning

Solving Motion Capture

Facial Motion Capture

Facial Actor Survey

Actor Survey – Hardware

Reference Data

Statistical Data

Facial Rigging

Facial Acquisition

Audio

Facial Motion Capture Solving

Real-Time Motion Capture

Real-Time Uses

Real-Time Limitations

Alternate Technologies

Motion Capture Resources

Virtual Production

World Building

Previsualization

On-Set Visualization

Virtual Cinematography

Chapter 5: Stereoscopic 3D

How 3D Works

Accommodation and Convergence

Interaxial Separation

Toe-in Versus Horizontal Image Translation

Parallax or Depth Budget

Positive and Negative Parallax

Floating Windows

Fix It in Post

Stereoscopic Design

The Emerging Grammar of 3D

Creative Use of Depth

Previsualization

Avoiding Painful 3D

The Aesthetic of Scale

Cutting for 3D

Designing for Multiple Release Formats

Immersion-Based versus Convergence-Based Stereo

Native Stereo

Pre-Production

On-Set

Stereography in the Visual Effects Process

Stereography for Finishing

HFR as a Solution for Better 3D Movies

VFX Elements and Stereo

Introduction – How VFX Elements are Used

Native Stereo Content

Mono Capture – Packaged Script and Element Deliveries

Mono Capture – Hybrid Approach for Stereo Delivery

Mono Capture – Full CG Approach for Stereo Delivery

Creating Depth – Layout and Stereography

Stereo Camera – Depth Wedges

Stereo Compositing

Requesting a Full CG Stereo Render Mid-Production

VFX Production Tasks

2D to 3D Conversion

Depth Creation Preparation

Visual Analysis of 2D Depth Cues

Pre-Production and Previs for Conversion

Source and Target Perspective

Shared Shots/Shared Workflows

Main Stages of 2D-to-3D Conversion

Major 2D-to-3D Conversion Workflows

Special Cases

Re-projection Mapping Workflow

Pixel Displacement or Pixel Shifting

Other 2D-to-3D Conversion Workflows

Is "Real" Always Right?

2D-to-3D Conversion Management

Stereoscopic Visual Effects

Prepping for the Third Dimension

Shooting the Third Dimension

Visual Effects in the Third Dimension

Photographed Elements

Accuracy and Attention to Detail

Artistic Skill Level

Data Management

Stereoscopic Digital Intermediate Workflow

Stereoscopic 3D Process Milestone

Viewing 3D Dailies

Projection Screens for Stereoscopic Viewing

3D Editorial Processes

Data Workflow

Applying the 3D Grade

3D Stereo Deliverables

3D Home Video Deliverables

Stereoscopic Window

The Stereoscopic Window

Placement of the Window in Relation to the 3D Scene

Window Violations

Window Placement Logic

How to Create a Stereoscopic Window

Producing Movies in Three Dimensions

Development – Getting the Green Light

Production – What to Look Out For

Chapter 6: Post-Production / Image Manipulation

Resolution and Image Format Considerations

Formats

Transport

Resolution

Academy Color Encoding System (ACES)

ACES Components

ACES Benefits

ACES Color Space Encoding

Viewing ACES

Preparations for Using ACES

Image Compression/File Formats for Post Production

Image Encoding

Still Image Compression

Other Lossless Compression Methods

File Formats

Color Management

The Three Guidelines

Digital Color Image Encodings and Digital Cameras

Color Management at the Desktop

Bringing Color Management to Film Workflows

Digital Intermediate

Shot Element Pulls and Delivery to VFX

Introduction

The Lab

"Production" Is to "Lab" as …

The Merge

The Handoff

VFX Editorial

The Select

The Pull

VFX Editorial

Editing within a Shot: The Art of Precompositing (Precomp)

How It Came to Be

Modern Day Tracking and Disseminating of Information

As the Shot Changes

Wrapping It Up

Editorial Workflow in Feature Animation

Introduction

Editorial Crew Staffing and Structure

Editorial Involvement with Feature Animation Production Stages

Communication with Artists

Starting

Working with Teams

Working Globally

Reference and Perspective

Shot Production

Communicating with Artists in Other Departments

Completion

Compositing of Live-Action Elements

Modern Digital Compositing

Scene Tracking

Rotoscoping and Paint

Rotoscoping

Digital Painting and Plate Reconstruction

Matte Paintings/Creative Environments

Matte Paintings: Art of the Digital Realm

What Is a Matte Painting?

Matte Painting Pioneers and History

Visualizing the Matte Painting Shot in Pre-Production

On-Set Supervision for Matte Painting Shots

Basic Skills and Tricks of the Trade

Miniatures and Computer-Generated Sets

Finding the Best Frame

Re-Projected Photo Survey

The Need for Creative Compositing

3D Matte Painting

Chapter 7: Digital Element Creation

Digital Modeling

Overview: The Importance of Modeling

Types of Modeling

Model Data Types

Development of Models

Modeling for a Production Pipeline

Engineering Aspects for Polygons

Engineering Aspects for NURBS

Rigging and Animation Rigging

Rigging: What is It?

Animation Rigging

Deformation Rigging

Texturing And Surfacing

The Importance of Texture Painting

Hard Surface Models

Creature Models

Types of Geometry: Their Problems and Benefits

Prepping the Model to Be Painted

Texture Creation

Various Other Map-Driven Effects

Texture Painting in Production

Model Editing

Digital Hair / Fur

Hair Generation Process

General Issues and Solutions

Digital Feathers

Morphology of Real Feathers

Modeling Digital Feathers

Similarities between Hair and Feathers

Differences between Hair and Feathers

General Geometry Instancing

Asset Creation

World Building

Shot Considerations

Dynamics and Simulation

How is a Simulation Created?

When is Simulation Appropriate?

Tricks and Cheats

Important Considerations

Planning and Preparation

Software Solutions: A Broad Overview of Current Options

Particles

What are Particle Systems?

The Next Step

The Birth of Particles

Creating Effects

Rigid-Body Dynamics

How Rigid-Body Dynamics are Created

Potential Problems

Other Issues

Tricks for Getting It Done

Digital Lighting

Light in Reality and in Computer Graphics

Case Study of Reality Compared with Simple CG Simulation

Visual Sophistication through Texture Mapping

Physically Derived Shading Models

Beneath the Surface

Goals of Lighting in Visual Effects

Work Flow for Successful Creative Digital Lighting

The Technologies of Lights in Computer Graphics

Direct Lighting: Source to Surface to Camera

Reflections

Photographed Reflections

Shadows

Image-Based Lighting

Rendering Occlusion

Ambient Occlusion

Reflection Occlusion

Creating Light Sources from Environment Maps

Physically Based Rendering

Physically Plausible Rendering

Volumetric Lighting Effects

Shader Basics

What are Shaders?

Shading Models

Bump and Displacement

Map-Based Shaders

Procedural Shaders

Shader Design

Anti-aliasing Considerations

2D Compositing

2D File Formats

Image Quality: Color Bit Depth and Concatenation

Log vs. Linear

Low Dynamic Range and High Dynamic Range Images

Mattes and Pre-Multiplied Alpha

Working with Rendered CG Elements

Integration Techniques

2D Compositing

Z-Depth Compositing

Adding Depth of Field

Adding Motion Blur

Relighting

3D Compositing

Working with 3D Data in a Compositor

Pan and Tile

Camera Projections

Set Extensions

Coverage Mapping

3D Mattes

3D Retouch and Cleanup

Adjusting Camera Moves

Particles

Deep Compositing

Crowd Generation and Simulation Techniques

Live-Action Replication

Sprites

Computer-Generated Crowds

Modeling for Replication

Variation

Mesh Density

Animation Cycles for Replication

Motion Capture

Keyframe Animation

Dynamic Motion Synthesis

Behaviors and Crowd Control

CG Prosthetics and Actor Enhancements

On-Set Tracking and Capture Considerations

Eye Enhancements

3D Techniques

2D Techniques

2D Techniques

Silhouette Changes

Re-Projection

3D Products, Systems, and Software

Digital Element Creation Process

3D Graphics Software

3D Tracking

Special Effects

Rendering

Texturing

Chapter 8: Interactive Games

Overview

How the Gaming Industry and Film/TV Industries are different

Game Engines and Real-Time Rendering

Runtime Component

Disciplines and Job Titles

Game Design

Engineering

Production

Test

Art

The Art Director

Concept Art

Environment Artists

Texture Artist

Characters

Hard Modeling

Props

Lighting

Baked vs. Dynamic Lighting

Shadows

Effects

System Effects

Environmental FX

Breakables

Destruction

Tech Artist

Animation

UI

Real Time Shaders and Materials

Pre-Rendered Cinematics vs. Real Time Visuals

Optimization and Runtime Budgets

Performance Analysis and Profiling

CPU vs. GPU bound

Technical Terminology

User Calibration

Latency

"Game Mode" on Televisions

HDR10

PBR – Physically Based Rendering

FBX

Mip Mapping

Filtering

Texel

Screen Space Ambient Occlusion

Level of Detail

Vertex Shaders and Fragment Shaders

AR/VR

Future of Gaming

On-Demand Rendering, Cloud Distribution and Ray Tracing

Chapter 9: Complete Animation

What Is An Animation Project?

Full Animation versus Visual Effects

Difference Between Visual Effects and Animation

Production Pipelines

Production

A Survey and History of Animation Techniques

Traditional Animation

Stop-Motion

Computer Graphic Technology

Considerations for a Full CG-Animated Feature Pipeline

CG Feature Animation Pipeline

Managing an Animated Film

Film Management and Personal Style

Building Brain Trusts

Building the Core Creative Team

Writing and Visual Development

Working with a Studio

Facilities and Environment

Managing the Event

The Production Process: An Animator’s Perspective

Working on CG-Animated Content in Live-Action Features

Planning the Process

Production

Character and Environment Interaction

Chapter 10: General Workflow Considerations

Virtual Studio Technology

Analysis of A Production Workflow

From Workflow to Pipeline

Service Bureau versus In-House Requirements

Design of a Production Workflow

From Analysis to Design

Deploying a Production Workflow

From Design to Implementation

Infrastructure

Tracking Assets

What is Task and Asset Tracking?

Commercial Task and Asset Tracking Systems

Building Task and Asset Tracking Systems

Scene Assembly

3D Scene Assembly

2D Scene Assembly (Compositing)

Working Across Multiple Facilities

Images

Models

Texturing

Animation

Compositing

R&D

Chapter 11: VR / AR (Virtual / Augmented Reality)

A Note from the Editors

Prelude to Virtual Reality & Augmented Reality

Pre-Production for VR/AR

Production for VR/AR

Post-Production for VR/AR

Editorial Post-Production for VR/AR

How to Direct the Viewer?

The Post-Process

Nonlinear Editorial, Timelines and Edits

World Lock and Forced Perspective

Types of Head-Mounted and Handheld Displays

What are the HMD and the "VR Presence"?

Electronical Designs of HMDs and their Respective Performances

Optical Designs of HMDs and their Respective Usages

Image Quality Factors in VR HMD Displays

Hemispheres and Domes

Game Mechanics Are What it is All About

Overcoming Doubt and Preconceived Notions

Dome Projections

The Future of Domes

VR/AR Tracking Displays

Uses of VR

Enterprise

Narrative Storytelling

Future of VR and AR

This is Just the Beginning

Acknowledgments

Appendix A: Charts and Formulas

Appendix B: Credits / Titles to Be Submitted in Accordance with VES Guidelines

Appendix C: Glossary

Index