The VES Handbook of Visual Effects

COMING SOON - John Knoll

CHAPTER 1: INTRODUCTION - Michael Fink, VES

What is a Visual Effect?

What is a Special Effect?

Art and Technology

CHAPTER 2: PRE-PRODUCTION/PREPARATION

Overview - Scott Squires, VES

Breaking Down a Script - Budgeting - Scott Squires, VES

Ballpark Budget

More Detailed Budgets

Bidding

Plate Photography

Temp Screenings

Reviewing Bids

Contracts

Rebidding during Shooting

Rebidding in Post

Monitoring the Budget and Schedule

Keeping the Budget Down

Examples of Script and Shot Breakdowns - Susan Zwerman, VES, Jeffrey A. Okun, VES

Breakdowns

Production Breakdowns

Generating a VFX Breakdown: The First Pass

Working with the Director and Producer - Scott Squires, VES

Demo Reel

The Meeting

Moving Forward

Production Departments - Scott Squires, VES

Production Design

Camera

Working with the Cinematographer - Bill Taylor, ASC, VES

Special Effects - Scott Squires, VES

Stunts

Wardrobe

Makeup

Production

Visual Effects

Editorial

Locations

Production Meeting

Designing Visual Effects Shots - Scott Squires, VES

Guidelines for Directors

Storyboards

Previs

Objective of the Shot

Concept Art

Continuity

Photorealism

Original Concepts

Budget

Reality and Magic

Camera Angles

Framing

Scale

Detail

Speed

Scaled Images

Depth of Field

Sequence of Shots

Camera Motion

Less is More

Action Pacing

CG Characters

Creatures and Character Design

Powers of 10 Shots

Visual Effects Techniques - Scott Squires, VES

Technical Considerations

Additional Suggestions for Determining Techniques

Development of Previs Techniques - Mat Beck, Stephanie Argy

History and Background

New Potent Techniques

Artificial Intelligence (AI)

What is Previs and Other Forms of Visualization? - Hamilton Lewis

What is Previs?

Different Types of Visualization and When to Use Them

Visualization: The New Essential

The Application of Previs: Who Benefits and How? - J. Todd Constantine

The Benefits of Previs

Project Types

Post Visualization - Scott Hankel

What is Post-Visualization?

Why Use Postvis?

Who Does Postvis?

Cautions and Suggestions for Good Practice - Mat Beck

The Perils of Previs!

Passing the Work On

The Role of the VFX Supervisor in Previs

Previsualization: Advanced Techniques - Christopher Ferriter

Introduction

Benefits of Visualization

Advanced Techniques

Camera Angle Projection - Steven D. Katz

Introduction by Bill Taylor, ASC, VES

Drawing What the Lens Sees

Techvis - Christopher Ferriter

What is Techvis?

How is Techvis used?

Advancements in Techvis

Emerging Techniques and Trends

Essential Data Captured in Techvis

Real Time Engines and Their Previs Applications in VFX - Ryan McCoy

Introduction

Pre-Production Applications

Production Applications

Pipeline Advantages

Real Time Engines and Their Application in Animation - Ryan McCoy

Traditional 3D Pipeline

Real-Time Pipeline

CHAPTER 3: ACQUISITION / SHOOTING

Working on Set - Scott Squires, VES

Common Types of Special Effects - Gene Rizzardi

What are Special Effects?

A Brief History of Special Effects

The Special Effects Supervisor

Working with the Visual Effects

Visual Effects in Service to SFX

Special Effects Design and Planning

Storyboards and Previs

The Elements: Rain, Wind, and Snow and Ice

Smoke, Fire, and Pyrotechnics

Mechanical Effects

Flying Wire Rigs and Stunts

Safety

Greenscreen and Bluescreen Photography - Bill Taylor, ASC, VES

Best Practices and Otherwise

Overview

Function of the Backing

Negative Scanning and Digital Conversion

Backing Uniformity and Screen Correction

The Alpha Channel

The Processed Foreground

The Composite

Recommended Specifications and Practices

How to Expose a Greenscreen Shot, and Why

Setting Screen Brightness

Choosing the Backing Color

Floor Shots, Virtual Sets

Foreground Lighting

Controlling Spill Light

Lighting Virtual Sets

Tracking Markers

On-Set Preview

Cameras for Bluescreen or Greenscreen Photography

Underwater Photography

On-Set Data Acquisition - Karen Goulekas

Camera Report

Tracking Markers

Props for the Actors

Cyberscanning

Digital Photos

LiDAR/Laser Scanning

Lens Distortion Charts

HDRI and Chrome Balls

LiDAR Scanning and Acquisition - Alan Lasky

On-Set 3D Scanning Systems - Joshua DeHerrera

Types of Technology

LiDAR

Photogrammetry

Prop Scanners

Lighting Data - Charles Clavadetscher, Jeffrey A. Okun, VES

Gathering Lighting Data

Using Conventional Still Cameras

Reference Shooting Considerations

Beware of False Savings!

Clean Plates - Scott Squires, VES

Shooting the Clean Plate

Locked-Off Camera

Moving Camera

Other Issues

Post-Process

Alternates without Clean Plates

Other Uses for Clean Plates

Monster Sticks - Scott Squires, VES

On-Set Animation Capture: Witness Cam (IBMC) - Joel Hynek

Wireless Non-Video Motion Capture

Factors Affecting Witness Cameras

Dealing with the Data in Post-Production

Camera Tracking for Real-Time Visualization - Joe Lewis

Introduction

Camera Department: Pre-Production and Cooperation

Camera Tracking Strategies in an LED Volume

Emerging, Innovation and Mature Technology

Triangulation As a Method of Recording Camera Data - Stuart Robertson

Camera/Subject Positional Information

Basics: The Toolkit

Basics: Nodal Point

Photographic Reference - Charles Clavadetscher

How to Proceed

Shooting Video as a Reference

Rules, Setup, and Testing - Charles Clavadetscher

Do a Complete Test Shot!

Why Run Through Example or Test Shots?

Digital Cinematography - David Stump, ASC, Marty Ollstein, David Reisner, W. Thomas Wolf

Digital Definitions

High Dynamic Range (HDR)

Camera / Lens Metadata

Camera Tracking

Look Management

The Recording System

VFX Photography - Eric Swenson

The Camera Array

Designing an Array Shot

Technicians

Shoot Day

Special Techniques

Post

The Future

Filming Live-Action Plates to be Used in VFX - Bill Taylor, ASC, VES

Camera Position (Station Point)

Angle of View

Lighting Considerations

Camera Tilt

Background Quality

Moving Plates

Scouting the Camera Positions

A Case Study

Camera Cars

Camera Car Safety Issues

Purpose-Built Crane Cars

Vibration and Camera Stabilization

Road Speed

Precautions

Panoramic Rigs

On the Water

Air to Air

Cable Systems

Shooting Elements for Compositing - Mark H. Weingartner

What Is an Element?

Stock Footage

Types of Elements

Generic versus Shot-Specific Elements

Determining Element Needs

Cheating

Backgrounds

Black Backgrounds

Line-Up

Camera Format Considerations

Assorted Methods for Shooting Elements

High-Speed Cinematography and Filming Elements - Jim Matlosz

Cameras

Technicians

Director of Photography

Lighting

Moving the Camera

Video Assist

Post

Supervising Motion Control - Mark Weingartner

What is Motion Control?

Performance Choreography

Multiple-Pass Photography

Scaling

Import and Export of Camera Move Data

The Data

Types of Motion Control Systems

Motion Control Software

Camera Types

Sync and Phase

Dealing with Production

Acquisition of Motion / Still Photographic Textures for Mapping onto CG - Mark Weingartner

Panoramic Backgrounds

Tiled Stills

Motion Tiling and Synchronous Plates

Practical Considerations

Stills for Textures and Lighting

Stop-Motion - Pete Kozachik, ASC

Evolution of Stop-Motion Photography

The Time Required to Shoot in Stop-Motion

Preparation before Shooting

Setting up a Shooting Space for Stop-Motion

Use of Motion Control in Stop-Motion

Useful Caveats

Evolution of a Shot

Use of Stop-Motion in Visual Effects

CHAPTER 4: PERFORMANCE AND MOTION CAPTURE

Defining Motion Capture / Performance Capture - Demian Gordon, Nathan Camp, John Root

How Motion Capture is Used Across the Industry - Demian Gordon, Nathan Camp, John Root

Is Motion Capture Right for a Project? - Demian Gordon, Nathan Camp, John Root, Tracy McSherry

The Mocap Look

Visual Storytelling and Aesthetic

Production Planning

Performance

Technology and Approach

Aligning Needs vs. Wants

Optical Motion Capture

Motion Capture for Virtual Production

Motion Capture for Animation

Production Planning Scenarios for Motion Capture - Demian Gordon, Nathan Camp, Rhea Aylin Fischer,

and John Root

Subject Matter of the Shoot

Performance and Actor Considerations

Real-Time Presentation

Subjects Being Captured

Volume Size and Build

Syncing Hardware and Software

Rigging and Retargeting for Motion Capture

Shot List

Client or Vendor Delivery

Optical Motion Capture - Nathan Camp, Demian Gordon , John Root

Technical and Approach

Entry Point

Budget

Hardware

Software

Inertial Motion Capture - Rhea Aylin Fischer, Ted O'Brien ,John Root, and Demian Gordon

Use Cases

Technology and Approach

Entry Point

Budget

Hardware

Software

Motion Capture for Virtual Production - Nathan Camp, Ted O'Brien

Use Cases

Technology and Approach

Entry Point

Budget

Hardware

Software

Hand Capture Demian Gordon , Rhea Aylin Fischer, Tracy McSherry. John Root,

Use Cases

Advantages and Disadvantages

Technology and Approach

Budget

Facial Motion Capture - John Root, Demian Gordon, Gareth Edwards, and Oleg Alexander

Use Cases

Markerset

Lighting

Facial Hair

Advantages and Disadvantages

Entry Point

Budget

Facial Actor Survey

Actor Survey - Hardware

Applying AI

Volumetric Capture / Considerations for Production - Wardrobe, Hair, Makeup, and Blocking - Christina Heller, Skylar Sweetman, Mari Young

CHAPTER 5: VIRTUAL PRODUCTION AND AI TOOLS IN VISUAL EFFECTS

Virtual Production - Addison Bath

What is Virtual Production

How is Virtual Production Used?

Note from the Editors: - Jeffrey A. Okun, VES, Susan Zwerman, VES, Susan Thurmond O'Neal

Capturing Surfaces with Photogrammetry to Create Assets - Luis Cataldi

Introduction

Core Principles of Photogrammetry

Applied Principles of Image Capture for Photogrammetry

Capture Methods and Workflows

Final Thoughts

Photogrammetry and LiDAR Sets for Virtual Production - Tripp Topping

Optimization

The Use of AI in Visual Effects - Lucien Harriot

Introduction

Overview of Generative Artificial Intelligence (GenAI)

Applications of GenAI in Visual Effects

Impact on Visual Effects Processes

Limitations, Challenges, and Ethical Considerations

Future Developments in GenAI

Embracing GenAI as a Tool for Innovation

Summary of Techniques for NeRFs and 3D Gaussian Splatting - Mike Seymour

Introduction

Overview of NeRFs and Gaussian Splatting

Neural Radiance Fields (NeRF)

3D Gaussian Splatting

NeRFs and 3DGS in Combination with Other AI

Limitations, Challenges, and Ethical Considerations

Summary of AI Techniques Used in Visual Effects - Steve Wright

A Brief History

AI for Compositing

Image Enhancement

Visual Effects

Other Features

Adapting to a World with AI

CHAPTER 6: POST-PRODUCTION / IMAGE MANIPULATION

Resolution and Image Format Considerations - Scott Squires

Formats

Transport

Resolution

Academy Color Encoding System (ACES) - Bill Baggelaar, Annie Chang, Steve Tobenkin

Overview

Using ACES in Visual Effects Workflows

ACES Color Space Encoding

Compatibility

Image Compression/File Formats for Post-Production - Florian Kainz, Piotr Stanczyk

Image Encoding

Still Image Compression

Other Lossless Compression Methods

File Formats

Color Management - Joseph Goldstone

The Three Guidelines

Digital Color Image Encodings and Digital Cameras

Color Management at the Desktop

OpenColorIO

Digital Intermediate - Tashi Trieu

Introduction to Digital Intermediates

The Colorist

Tools of the Trade

Emerging Technologies

Grading with Mattes

Communicating Color Through the Visual Effects Pipeline

Visual Effects Plate Grading

The Evolution of Color Grading

Shot/Element Pulls and Delivery to VFX - David H. Tanaka

Introduction

The Lab

"Production" Is to "Lab" as

The Merge

The Handoff

VFX Editorial

The Select

The Pull

VFX Editorial - Greg Hyman, David H. Tanaka

Editing within a Shot: The Art of Precompositing (Precomp)

How It Came To Be

Modern Day Tracking and Disseminating of Information

As the Shot Changes

Wrapping It Up

Editorial Workflow in Feature Animation - David H. Tanaka

Introduction

Editorial Crew Staffing and Structure

Editorial Involvement with Feature Animation Production Stages

Communication With Artists - Eric Durst

Starting

Working with Teams

Working Globally

Reference and Perspective

Shot Production

Communicating with Artists in Other Departments

Completion

Compositing of Live-Action Elements - Steve Wright

Overview

Green/Blue Screen Capture

Pre-Processing the Green Screen

Garbage Mattes

Keying Strategies

Motion Blur and Depth of Field

Spill Suppression

Grain Management

Rotoscoping

Post-Composite Processing

3D Compositing

Inspecting the Comp

Rotoscoping and Paint - Marshall Krasser, Sandy Houston

Rotoscoping

Digital Painting and Plate Reconstruction

Deep Compositing - Steve Wright

What is a Deep Image?

Complex Depth Interactions

The Depth Compositing Edge Problem

The Re-rendering Problem

Deep Compositing and Live Action

Matte Painting/Creative Environments - Craig Barron, VES, Deak Ferrand, John Knoll, and Robert Stromberg

Matte Paintings: Art of the Digital Realm

What Is a Matte Painting?

Matte Painting - Pioneers and History

Visualizing the Matte Painting Shot in Pre-Production

On-Set Supervision for Matte Painting Shots

Basic Skills and Tricks of the Trade

Miniatures and Computer-Generated Sets

Finding the Best Frame

Re-Projected Photo Survey

The Need for Creative Compositing

Digital Environments - Robert Stromberg

Reimagining Artistic Roles in Film: The Rise of the "Environment and Creation Artists"

Virtual Production and Game Engine Technology

Future of the Environment Creation Artist and AI

The Power of Curiosity in Creativity and Innovation

CHAPTER 7 - DIGITAL ELEMENT CREATION

Digital Modeling - Kevin Hudson

Overview: The Importance of Modeling

Types of Modeling

Model Data Types

Development of Models

Modeling for a Production Pipeline

Engineering Aspects for Polygons

Engineering Aspects for NURBS

Rigging and Animation Rigging - Steve Preeg

Rigging: What is It?

Animation Rigging

Deformation Rigging

What Makes a Good Rig for Animation - Felix Balbas

Introduction

User and Uses

Technical Aspects

Technology Restrictions

New Technologies

Texturing and Surfacing - Ron Woodall

The Importance of Texture Painting

Hard Surface Models

Creature Models

Types of Geometry: Their Problems and Benefits

Prepping the Model to Be Painted

Texture Creation

Various Other Map-Driven Effects

Texture Painting in Production

Model Editing

Digital Hair/Fur - Armin Bruderlin, Francois Chardavoine

Hair Generation Process

General Issues and Solutions

Digital Feathers - Armin Bruderlin, Francois Chardavoine

Morphology of Real Feathers

Modeling Digital Feathers

Similarities between Hair and Feathers

Differences between Hair and Feathers

General Geometry Instancing - Armin Bruderlin, Francois Chardavoine

Asset Creation

World Building

Shot Considerations

Dynamics and Simulation - Craig Zerouni, Judith Crow

How is a Simulation Created?

When is Simulation Appropriate?

Tricks and Cheats

Important Considerations

Planning and Preparation

Software Solutions: A Broad Overview of Current Options

Particles - Craig Zerouni

What are Particle Systems?

The Next Step

The Birth of Particles

Creating Effects

Rigid Body Dynamics - Craig Zerouni

How Rigid-Body Dynamics are Created

Potential Problems

Other Issues

Tricks for Getting It Done

Digital Lighting - Andrew Whitehurst

Light in Reality and in Computer Graphics

Case Study of Reality Compared with Simple CG Simulation

Visual Sophistication through Texture Mapping

Physically Derived Shading Models

Beneath the Surface

Goals of Lighting in Visual Effects

Work Flow for Successful Creative Digital Lighting

The Technologies of Lights in Computer Graphics

Direct Lighting: Source to Surface to Camera

Reflections

Photographed Reflections

Shadows

Image-Based Lighting

Rendering Occlusion

Ambient Occlusion

Reflection Occlusion

Creating Light Sources from Environment Maps

Physically Based Rendering

Physically Plausible Rendering

Volumetric Lighting Effects

Shader Basics - Derek Spears

What are Shaders?

Shading Models

Bump and Displacement

Map-Based Shaders

Procedural Shaders

Shader Design

Anti-aliasing Considerations

2D Compositing - Matt Leonard

2D File Formats

Image Quality: Color Bit Depth and Concatenation

Log vs. Linear

Low Dynamic Range and High Dynamic Range Images

Mattes and Pre-Multiplied Alpha

Working with Rendered CG Elements

Integration Techniques

2.5D Compositing - Matt Leonard

Z-Depth Compositing

Adding Depth of Field

Adding Motion Blur

Relighting

3D Compositing - Matt Leonard

Working with 3D Data in a Compositor

Pan and Tile

Camera Projections

Set Extensions

Coverage Mapping

Cubical Camera Setup

3D Mattes

3D Retouch and Cleanup

Adjusting Camera Moves

Particles

Deep Compositing

Crowd Generation and Simulation Techniques - John Knoll

Live-Action Replication

Sprites

Computer-Generated Crowds

Modeling for Replication

Variation

Mesh Density

Animation Cycles for Replication

Motion Capture

Keyframe Animation

Dynamic Motion Synthesis

Behaviors and Crowd Control

CG Prosthetic and Actor Enhancements - John Knoll

On-Set Tracking and Capture Considerations

Eye Enhancements

3D Techniques

2D Techniques

2.5D Techniques

Silhouette Changes

Re-Projection

3D Products, Systems and Software - Richard Kidd

Digital Element Creation Process

3D Graphics Software

3D Tracking

Special Effects

Rendering

Texturing

CHAPTER 8 - INTERACTIVE GAMES

Overview - David Johnson, VES

How the Gaming Industry and Film/TV Industries are different - David Johnson, VES

Publishers and Developers v. Film/Television Studios and VFX Facilities - David Johnson, VES

Game and TV/Film Converging with Virtual Production and Cinematics - David Johnson, VES

Game Engines and Real-Time Rendering - David Johnson, VES

Runtime Component

Unreal vs. Unity

Disciplines and Job Titles - David Johnson, VES

Game Design

Engineering

Production

Test

Art

The Art Director

Concept Art

Environment Artists

Texture Artist

Characters

Hard Modeling

Props

Lighting

Baked vs. Dynamic Lighting vs. Raytracing

Shadows

Effects

System Effects

Environmental FX

Breakables

Destruction

Tech Artist

Animation

UI

Real Time Shaders and Materials - David Johnson, VES

Optimization and Runtime Budgets - David Johnson, VES

Performance Analysis and Profiling - David Johnson, VES

CPU vs. GPU bound

Technical Terminology - David Johnson, VES

User Calibration

Latency

"Game Mode" on Televisions

HDR10

PBR - Physically Based Rendering

FBX

Mip Mapping

Filtering

Texel

Screen Space Ambient Occlusion

Level of Detail

Vertex Shaders and Fragment Shaders

AR/VR - David Johnson, VES

Future of Gaming - David Johnson, VES

On-Demand Rendering, Cloud Distribution and Ray Tracing

Fast Drives, Nanite and Lumen

Fluid Simulations

AI

CHAPTER 9 - COMPLETE ANIMATION

What Is An Animation Project? - Stephan Vladimir Bugaj

Full Animation vs Visual Effects

Difference Between Visual Effects and Animation - Rob Bredow

Production Pipelines

Production

Animation Techniques - Frank Gladstone

Traditional Animation

Stop-Motion

Computer Graphic Technology

Considerations for a Full CG-Animated Feature Pipeline - Stephan Vladimir Bugaj

CG Feature Animation Pipeline

Managing an Animated Film - Don Hahn

Film Management and Personal Style

Building Brain Trusts

Building the Core Creative Team

Writing and Visual Development

Working with a Studio

Facilities and Environment

Managing the Event

The Production Process: An Animator's Perspective - Lyndon Barrois

Working on CG-Animated Content in Live-Action Features

Planning the Process

Production

Character and Environment Interaction

CHAPTER 10 - VISUAL EFFECTS WORKFLOWS

Analysis of a Production Workflow - Stephan Vladimir Bugaj

From Workflow to Pipeline

Service Bureau versus In-House Requirements

Design of a Production Workflow - Dan Rosen

From Analysis to Design

Deploying a Production Workflow - Stephan Vladimir Bugaj

From Design to Implementation

Tracking Assets - Stephan Vladimir Bugaj

What is Task and Asset Tracking?

Commercial Task and Asset Tracking Systems

Building Task Asset Tracking Systems

Scene Assembly - Stephan Vladimir Bugaj

2D Scene Assembly

3D Scene Assembly (Compositing)

Working Across Multiple Facilities - Scott Squires, VES

Images

Models

Texturing

Animation

Compositing

R&D

Version Control Software - Jase Lindgren, Katie Cole

Introduction to Version Control

Version Control Uses in Virtual Production

Version Control Set-Up

CHAPTER 11 - VR / AR (VIRTUAL REALITY / AUGMENTED REALITY)

A Note from the Editor Prelude to Virtual Reality & Augmented Reality - Jeffrey A. Okun, VES

Pre-Production for VR/AR - Chris Edwards

Production for VR/AR - Grant Anderson

Post-Production for VR/AR - Dave Franks

Editorial Post-Production for VR/AR - John Grod

How to Direct the Viewer?

The Post-Process

Nonlinear Editorial, Timelines and Edits

World Lock and Forced Perspective

Types of Head-Mounted and Handheld Displays - Bernard Mendiburu

Identifying the Computer-Mediated Realities

State of the Art in VR Photonic Technologies

State of the Art in VR Hardware Technologies

State of the Art in VR Software Technologies

Using an HMD for Visual Effects Production

Choosing an HMD for Visual Effects Production

Hemispheres and Domes - John Grod

Games Mechanics Are What it is All About

Overcoming Doubt and Preconceived Notions

Dome Projections

The Future of Domes

The Evolution of Domes: The MSG Sphere - Phillip Galler

Uses of VR - John Grod

Enterprise

Narrative Storytelling

Future of VR and AR - Ben Grossmann

This is Just the Beginning

Immersive Experiences Make Use of Advanced Visual Effects Technology - Philip Galler

What are Immersive Experiences?

Visual Effects' Impact on Immersive Experiences

Content Creation for Immersive Experiences

Challenges for Delivering Content for Immersive Experiences

Wearables and the Future of Immersive Experiences

Acknowledgments

Appendix A: Charts and Formulas

Appendix B: Glossary

Index