Ever shoot an insane number of takes for a scene, plus one more for safety, only to still be disappointed with your options in post? Of course you have.
An actor might bring too much energy in one scene, not enough in the next, varied pause lengths and facial expressions … but none hitting just right to satisfy your Director vision?
New software in development from Disney could be “the final frontier in post-production manipulation” by allowing editors to synch, merge, and modulate between their favorite takes.
Share this post!
We present a method to continuously blend between multiple facial performances of an actor, which can contain different facial expressions or emotional states. As an example, given sad and angry video takes of a scene, our method empowers a movie director to specify arbitrary weighted combinations and smooth transitions between the two takes in post-production. Our contributions include (1) a robust nonlinear audio-visual synchronization technique that exploits complementary properties of audio and visual cues to automatically determine robust, dense spatio-temporal correspondences between takes, and (2) a seamless facial blending approach that provides the director full control to interpolate timing, facial expression, and local appearance, in order to generate novel performances after filming. In contrast to most previous works, our approach operates entirely in image space, avoiding the need of 3D facial reconstruction. We demonstrate that our method can synthesize visually believable performances with applications in emotion transition, performance correction, and timing control.