The last decade of twentieth century experienced the merging of some traditionally separate forms of audio-visual art and entertainment. The boundaries between “live action” feature films, animation, and games started to disappear, and the key to the newly forming comprehensive medium was “interactivity”. Advances in computer hardware and software have introduced the Interactive Multimedia Presentation as a common base for a variety of audio-visual contents.

iMediaTek Interactive Media Technologies, Inc. (referred to as “iMediaTek” throughout this document) is founded by Ali Arya and Steve DiPaola to design and develop technologies to bring Interactive Media (iMedia) to a variety of applications and clients. Considering the importance of facial figures in human communications, iMediaTek^™ introduces its flagship product, Interactive Face Animation – Comprehensive Environment (iFACE), a framework that provides the means of creating facial animations as simple as two-dimensional cartoons to complicated realistic three-dimensional heads, all with the same user and programming interface.

Computer-generated facial animation is a rapidly growing part in a variety of applications, from computer games and movies to electronic services and education. For instance, although current computer games make limited use of facial expressions, next generation game platforms provide hardware capabilities for computations involved in having more degrees of freedom in characters. One of the main objectives of game designers is to utilize these new platforms to introduce more realistic characters who can change expressions more frequently and behave more interactively. A virtual customer service representative can be considered another application of such characters.

Some of the issues facing content and application developers in this regard are:

Problem 1 – Behaviour. Designing different facial actions and expressions usually involve a painstaking and time-consuming process where artists create the related animation using conventional 3D software and by defining key frames for the movement of each facial feature. This is one of the major difficulties of increasing the number of “moveable” features (and so the realism).

Problem 2 – Portability. Designs for one head model are not generally usable on another model. As a result, even a similar action on a new head requires the design process to be repeated.

Problem 3 – Interaction. The need for a detailed design process limits the amount of interactivity and dynamic behaviour a character can have at run-time. In other terms, the characters can not be completely autonomous.

Problem 4 – Programmability. There is a serious lack of programmable components that can be re-used in new applications to provide facial animation capabilities. Each application has to be developed by implementing such functionality from scratch.

Problem 5 – Level of Details. The animators, especially when using conventional graphics software, have to deal with all the details of a head model to perform actions. An intelligent software that is aware of head regions and their function can hide the details unless necessary, by performing group actions on all the points that are functionally related. For example, averting the gaze direction is a simple action that should involve only a single input as new direction. The rest, i.e. moving eyeball points, should be taken care of by the software. Such a feature is missing in most design and runtime environments due to the fact that they are not customized for face animation.

iFACE Solution. iMediaTek introduces iFACE that provides solution to all of the above problems in a unified face animation framework. iFACE parameter spaces allow animator and/or programmer to effectively control facial geometry, perform MPEG-4 compatible facial actions, show expressions, and display behaviours based on definable personality types. All of these are encapsulated within a Face Multimedia Object (FMO) that can be used in any application through programming interfaces. The framework includes stand-alone graphical design environments and plug-in components for 3D software programs such as Maya.

iFACE hierarchical geometry arranges parameters in different layers of abstraction to allow exposure to proper level of details. The physical points can be pixel or vertex to support 2D and 3D models, both with the same control interface. On top of them are the feature points which correspond to MPEG-4 Face Animation and Definition Parameters. Using only these parameters almost any facial action is possible. Features and component layers are at higher levels and allow grouping of functionally related parameters. The parameters and their respective API are independent of the head model, so a set of parameters can be applied to any head model, resulting in the same facial actions.

Dynamic behaviours in iFACE are possible through Knowledge, Mood, and Personality parameter spaces. They allow defining interaction rule and scripts written in Face Modeling Language (FML), expressions, and personality types. iFACE personality types are based on the state-of-the-art in psychology. They are defined as a combination of Affiliation and Dominance factors and control the way facial actions are performed (e.g. frequency of blinking, typical head movements, etc). Using these parameters, an autonomous character can be created which interacts properly in a dynamic environment.

The parameterized approach with its behavioural extensions allow the animators and runtime programmers to use “face” as a self-supporting object without the need for dealing with details, and apply the same parametric design to any head model. iFACE API on the other hand, provides a powerful flexible component-based structure to be used in any application that requires face animation. The API itself uses a layered architecture to access different functionality.

Company Overview

Name

iMediaTek^™ Interactive Media Technologies, Inc.

Incorporated and Registered in British Columbia , Canada .

Mission

iMediaTek^™ strives to provide a venue for development of intelligent socially expressive multimedia technologies to achieve the next generation interactive systems.

Goals

1- Linking Art, Science and Technology

2- Leading Industry Role

3- Advanced Research

4- State-of-the-art Design

5- Reliable and Efficient Development

6- Ethical Practice

Ownership

iMediaTek^™ is founded and owned equally by Ali Arya and Steve DiPaola.

Organizational Chart

Board of Directors is the top controlling body in iMediaTek^™. Board selects a member as Chief Executive Officer (CEO) who supervises the overall operation of the company. iMediaTek^™ consists of four Departments each if necessary divided into Units (or Projects). Departments and Units/Projects are led by Directors and Managers, respectively. Unit-independent affairs of each Department (referred to as General and coded 100) are controlled by the Director not a Unit Manager.

* Business Affairs

† Human Resources

‡ Customer Services

Team Member Profiles

Ali Arya received a B.Sc. degree in Electrical Engineering from Tehran Polytechnic, Iran , in 1990, and a Ph.D. from the Department of Electrical and Computer Engineering, University of British Columbia, Canada, in 2004. He has worked as research engineer, system analyst, and project manager in different research centers and industry-related companies, including Tehran Cybernetic Arm Project, Iran , and Honeywell-Measurex, Canada. He has been the instructor for “Computer Animation”, “Systems Software Engineering” and “Software Project Management” in UBC and SFU. Before founding iMediaTek^™, Ali spent 18 months as a post-doctoral fellow at the School of Interactive Arts and Technology, Simon Fraser University, Surrey, BC, Canada, where he worked on social user interfaces and the idea of face multimedia object. Ali is a member of IEEE Computer Society and has published a variety of scientific and technical papers particularly in the area of facial animation.

Steve DiPaola received a B.Sc. from SUNY Stony Brook and his Masters from NYIT. He is currently an Associate Professor in Simon Fraser University ’s newest school - the School of Interactive Arts and Technology, in Surrey, BC, Canada, which actively combines technology, science and the arts using online and collaborative methods. There, he directs of the iVizLab (ivizlab.sfu.ca) which conducts research on ‘Socially-based Interactive Visualization’. DiPaola has research interests in human centered design, computer graphics and animation, and interactive systems. He had published (papers and book chapters) extensively in the area of character and avatar based 3D virtual communication technologies and has given presentations worldwide. DiPaola has been a teacher and researcher at such institutions as Stanford University and the Computer Graphics Lab at New York Institute of Technology.

Market Analysis

Products and Services

Interactive Face Animation – Comprehensive Environment (iFACE) is the primary iMediaTek product. iFACE is a framework for creating facial animations of a variety of types and for different applications. It provides FMO implementation for PC, mobile and game console environments, in addition to off-line authoring tools, wrapper objects for web development, application programming interfaces, and sample face-based applications.

Face Engine

iFACE can be used by client application developers who need to have facial animation in their systems. iFACE FMO acts as a “face engine” isolating face-specific details from the rest of the application modules. iFACE uses a parameterized model and keyframe-based and script-based approaches to content description. This allows the content providers to specify the desired facial actions in terms of parameters, see them at design-time, and recreate them with the same face engine at the run-time. By supporting parameters in Geometry, Knowledge, Mood, and Personality spaces, iFACE system is able to control the static appearance and also the behaviour of the face which allows the creation of autonomous and intelligent agents. iFACE provides proper programming interfaces to allow dynamic behaviour and run-time control by an intelligent application.

Face-based Applications

iMediaTek develops user-specific face-based applications using iFACE framework. Examples can be a web agent-based customer service system and virtual tour guides.

Affective Computing

Using state-of-the-art research on parameterization of emotions and personality, affective aspects of non-facial media types such as music can also be processed and remapped to/from facial animation.

Interactive Media (iMedia)

iMediaTek proposes the concept of Interactive Media (iMedia) as the foundation of its efforts, products and services. This is not limited to face-based applications and includes our upcoming products such as performance-based animation, virtual painters and virtual worlds.

Potential Clients and Applications

Face-based systems are primary clients of iFACE (our currently available product). Such systems can be grouped into computer games, communication systems, agents, and simulations. In all these applications, use of an intelligent face engine will help by reducing the time/effort/money needed for creating the content and increasing controllability through a systematic approach.

Computer Games

Advances in computer (especially graphics) hardware have made it possible to design more and more sophisticated computer games. Next generation games will include complicated facial expressions, lip-synch, and dynamic user-interactive facial actions. The face engine and its user interfaces provides a convenient environment for artists to design the animation pieces while the runtime programmers can easily recreate those actions by accessing the face engine API.

Telecommunication and Mobile Devices

A realistic facial animation can replace “live action” video in a video-phone and considerably reduce the required bandwidth. Such animation can be derived by the audio data (speech) or a camera at the sender side can analyze the speaker’s facial actions and send the commands to recreate them to the receiver.

Implementation of iFACE on mobile platforms also provides the possibility of other face-based applications such as games. Considering current desktop-based implementation using .NET and DirectX, and availability of these technologies in newer versions of Windows CE, Mobile iFACE is a one of the most immediate objectives for future development.

Web Communication

Similar to telecommunication scenario, a web-based visual communication (one-to-one or one-to-many) can be achieved using iMediaTek face technology. For instance, COMPS is a web-based educational system that uses iFACE to represent a remote instructor.

Agents and Avatars

Any online or offline system that represents users by virtual characters or simply uses virtual agents can benefit enormously from iFACE framework that animates the character with programmable emotional states and personality traits.

Simulations

A variety of scientific studies which are based on observation of human figures can be simplified, automated, and enhanced by using simulated programmable characters. A research project to study personality perception from faces is already using iFACE for this purpose.

Competition

The increased need for facial animation in new applications has motivated a few companies to work on related software tools. Most of these products are designed to simplify the process of creating new head models and do not provide any support for facial actions and behaviours. The more competitive examples of this group provide stand-alone applications and also plug-in components for popular 3D design environments. Some also provide automatic generation of 3D models from 2D photographs. Typical products belonging to this group are:

FacialStudio by Di-o-matic, http://www.di-o-matic.com/

FaceGen by Singular Inversion, http://www.facegen.com/

FaceWorks by PuppetWorks, http://www.puppetworks.com
GenHead by Genemation, http://www.genemation.com/
FaceRobot by Softimage, http://www.softimage.com/products/face_robot/

Although powerful in their own way, these tools are not designed to resolve the problems mentioned in Executive Summary.

Another group of products provide a limited level of support for animation and facial expressions through a library of existing pieces and maybe a simple lip-sync. An example is LifeStudio by LifeMode Interactive, http://www.lifemi.com/.

Visage by Visage Technologies, http://www.visagetechnologies.com, is one of the few products that provide a programming interface in addition to graphical design environments and functionality compatible with MPEG-4 standard. It is still limited to simple facial actions, has no scripting method, or behavioural support.

Some other products use more limited techniques such as simple morphing between 2D images to create a simple animation, such as talking. CrazyTalk by Reallusion, http://www.reallusion.com/, is an example of this group. AlterEgo by Face2Face, http://www.f2f-inc.com/ , provides an environment for performance-based animation which can create good results but for limited cases when the objective is to mimic a real performer.

Existing Research Clients

Face Personality Study

Behavioural psychology researchers usually use photographs and less commonly video to perform experiments. They can benefit from an interactive environment that can create realistic animations with different features (e.g. mood and personality). This can replace actors which are hard or expensive to find with software that does not need external setup and can be easily configured. iFACE system is being used in such an application by UBC researchers which in turn provides information regarding how viewers perceive the personality of a subject based on his/her facial actions.

Using iFACE the researchers can change the personality traits (or any other aspect) of the subject and observe the reaction and perception of the viewers. For more information see iFACE web site:

http://ivizlab.sfu.ca/research/iface

Storytelling Masks

Museums of anthropology, especially in North America , display a variety of artefacts from “first nations” (Native Americans). Among the most attractive of these artefacts are masks and head figures presented on objects such as totem poles. Songs, myths, and stories relate these figures to the history of the people who made them. Computer-generated characters with those figure-forms who also tell their stories and sing their songs are appealing and informative for the viewers and also provide a new means of creativity and expression to the native artists.

Combination of iFACE design and scripting tools provide such a creative environment. In this specific case we have begun working with the Parks Department of British Columbia, Canada and the Native community to create a museum display where a virtual version of an artist appears and tells the story of his work, can virtually turn into the artwork – a native mask – and have a virtual version of the art tell it’s story. Because all of this is under computer control it is possible to create many of the perceptual and educational techniques that a live human guide/artist could achieve including narrative style, presentation, and Q&A.

Evolving Faces

With similar goals and techniques of the Storytelling Mask project, Evolving Faces attempts to use facial agents to better engage viewers into the content, but in this case the agents are used to describe complicated scientific details, as well as create agents that are also an integral part of the content, evolving their appearance to tell the story of man’s migration out of Africa based on new DNA techniques. iFACE allows a designer to create head models that correspond to various stages of human evolution, and assign different types of behaviour (e.g. coarse or fine) to them to be expressed during talking or interaction. Such characters are ideal for science booths or online learning. Adding simple or complicated artificial intelligence can improve the behavioural capability of the characters for real-time interaction. The display uses voices, change and expressive faces and maps rather than charts and text.

Screen shot from the “Human Migration” interactive program shows how humans migrated from Africa some 50-100,000 years ago with its DNA marker and facial type migration evidence. Viewers can click on a specific face/area and have it tell the story of that DNA marker or click on a migratory path, having an evolving face explain the science facts of the journey of man.

COMPS

Collaborative Online Multimedia Problem-based Simulation (COMPS) is a system being developed by a group of Canadian university researchers to provide Problem-Based Learning (PBL) tools for medical students. PBL works by introducing students to a case (problem), giving them some facts, and taking the students through cycles of discussion and hypothesizing until the disease is correctly identified.

A major part of a PBL-based approach for medical students is to interact with patients, especially listening to them describing their symptoms. Bringing patients to a classroom or examination room is hard and in some cases impossible. Using “actors” for this purpose is a common but rather expensive alternative. A Social Conversational Agent (SCA) is an ideal replacement. SCA can also be a proper for “automated instructor” or to represent a remote instructor (or patient) when transmitting real-time video is not possible but SCA can be animated based on real audio data.