Art 173a - The Illusion of Life: Facial Animation – Steve DiPaola
This discussion of facial techniques is broken down into modeling and animation sections.
Polygonal – Polygonal modeling specifies exactly each 3d point, which connected to each other as polygons. This is an exacting way to get topology (points) where you need it on a face and not where you don’t.
Advantages: Specific control and placement (of topology and mapping).
Disadvantages: Hard to work with polygon data as you must work with it directly.
Patches (NURBs) Patches (or a set of splines) indirectly defines a smooth curve surface from a set of control points. A small amount of control points (called CVs in Maya) can define a complex surface. One type of spline is called NURBs which stands for Non-Uniform Rational B-Splines. This type of batch allows each control point to have its own weight that can affect the “pinch” of the curve at the point. So they are considered the most versatile of batches. They work very well for organic smooth objects so hence they are well suited for facial modeling however several issues arise.
Disadvantages:
To create a face from NURBS you must use topology (control points) evenly across the face but the face tends to have areas of high detail (eyes, mouth, ears) and areas of low detail (forehead, cheeks, …). Modeling with batches deals with this in two (not perfect) ways: either you 1) make one large patch with high density everywhere because it is needed in the high detail areas or 2) make a low density patch for the face and use higher density patches specific to facial areas ( ie. Eyes) and stitch them in to the overall face batch.
The two online tutorials demonstrate modeling heads with patches. The first shows creating a even density head gravitating the extra splines into the mouth for its needed detail and the stitching in eyes, at the end it shows how to make one (lame) patch over the whole head. See tut. The second tutorial starts by making several individual patches and stitching them all together - tut2.
Other issues are that with faces have many creases (around the eyes, mouth, ears, nose, and wrinkles) and patches present a challenge to do creases in a non-adhoc way.
Sub-Division surfaces is a fairly new modeling technique that gives you the control and flexibility of polygons with the ease of use and smoothness of patches. Tony DeRose (who wrote the paper on Sub-D surfaces and created a working version for Pixar, first used in Geri’s Game) has slides on the advantages on sub-d surface (to slide 6) here: derose1.
Sub-D surfaces gives you the detail only where you need it. See derose2. Paul Aichele discussed this on our Pixar trip with Geri’s head. “Rather then fighting the curvature with NURBs”, he was able to put the topology where the curvature of the sculpture called for it. See Geri-Model. Sub-D surfaces also gives you better control of texturing, creases and local sharpness. Maya now has Sub-D surfaces in the unlimited version and Sub-D emulation (with polygons) in there complete version.
Facial models can be created by digitizing live humans or physical models. There are several techniques. One that does need an expensive digitizer is using fudical points to reconstruct 2D photographs into a 3D model such as I did for the Kraftwerk music video animation. See kraftwerk. Now however, automatic digitizing equipment like that from CyberWare (see Scanner) is regularly used to create high resolution 3D models of live human models complete with color data (See face). While digitizing models are very useful for many application such as Stanford’s Digital Michelangelo Project (see Michelangelo), typically the data from these systems are too high resolution and not semantically setup for facial animation.
Web based avatar companies and others are using techniques that take a photograph of a human face (sometimes from the front and side) and map it onto a pre-made 3D model that animates by going through a registration process, where key points on the photograph (corners of: the eyes, eyebrows, mouth, ..) are picked via the mouse to register the image with the model. See Biovirtual.com. Several researchers are working on automatic registering techniques but lighting conditions on a live face or photograph and other standardization issues plagues the process.
Face generation systems genetically evolve the type of face you want or have you search or surf through a theoretical space of faces as with the program I developed for The Sims and we used on our assignment (see facespace).
Still other face generation systems allow you to pick from a database of facial parts to create a head as you might with a police artist sketch kit such as the Faces system we used as an assignment. (see FACES)
Keyframing
The most widely used animation technique is key-framing, where the animator creates key poses of an articulated model and the animation system interpolates the “in between” frames from that set of key-frame data. Typically the data being keyed and interpolated are transformations (move, rotation, scale) of rigid objects such as the hierarchical parts of human body. The problem with facial animation is there really are no rigid parts that move in relation to each other to key-frame. Hence there is a myriad of facial animation techniques based on what you are actually key-framing/interpolating to get smooth animation of a flexible surface of the face.
One widely used basic technique is to create a model of a face in a rest position. Then using essentially modeling techniques, edit the points of a copy of that face to make other faces (typically with the same topology hence the copying of the rest face) in different phoneme and expression states. Then animate a facial animation sequence by morphing (point interpolation) between this set of like-minded faces.
The disadvantage with this technique is that the animator is only picking from a set of pre-made face morphs and thereby limited to the expressions possible from that set in the final animated face sequence. There are several variants of this morphing technique – most notable compound or hierarchical morph targets which allow the animator to blend several faces together with differing weights and/or only morph specific areas of the face. The online tutorial goes through the basic process – see tut. Again, all versions of this technique limit the creative process by only allowing you to pick from a pre-made set of expressions (or force you to stop the animation process to create additional morph targets).
Tools that were made and are useful for character and organic animation have been used to animate the face. Typically these techniques are not straightforward and are cumbersome because they are not very well suited for a flexible face. They include free form deformations, “bones”, point cluster techniques and others.
Fred Parke's early work in facial animation at Univ. of Utah and NY Inst of Technology lead to the first facial animation system. It used a control parameterized method where the animation becomes a process of specifying and controlling parameter set values as a function of time. Parameter systems are able to animate the facial expressions as well as the facial types. The FaceLift program (see facespace) used for The Sims uses a simplified version of a production parameterization system. Most parameter systems use Paul Ekman's FACS ( Facial Action Coding System) which describes facial muscle movement as a basis or starting point for specifying and defining the range of parameters. Ken Perlin's web Java-based system also uses a simple but very effective parameterized technique (see Perlin). Parameter systems create a very compact specification for facial animation and are therefore ideally suited for the web and games.
Keith Waters work uses a simple simulation of muscle deformation to animate the face. It uses two types of muscles: linear muscles that pull and sphincter muscles that squeeze. He uses a mass and spring technique to animate or deform the skin. The muscle control system has a one to one correspondence to know face muscles and to Ekman's FACS. An extended, fast and open source version of Waters technique with applications for games and our real-time systems is called Expressions (see Expressions).
Facial animation has also been achieved via performance systems where a live performance is digitalized and applied to the facial model rather than created by an animator. Motion capture is the most widely used performance technique. Systems typically track via one or several cameras, small point-like reflective stickers attached in strategic positions on the performers face. See two examples of such systems.
http://famous3d.com/solutions/capture/
Another form of performance animation system uses the voice only to create not only the synched lip movement, but the movement of the other parts of the face as well including the eyebrows, blinks and eye movement, and head movement (neck rotation). These systems analyze the voice to get lip syncing phoneme positions and also use pitch, volume, sentence semantics (dividing speech into sentence sections based on pauses) and other cues to approximate the animation of a faces. These systems in standalone form or combined with photogrametric techniques are being used both in linear animation systems and in real-time web-based applications. See magpie, lipsinc and biovirtual.
