Automatic Facial Feature Recognition And Facial Expression .
International Journal of Science, Engineering and Technology Research (IJSETR), Volume 3, Issue 4, April 2014Automatic Facial Feature Recognition and FacialExpression Analysis in ImagesSumalakshmi.C.H, Adithya V Abstract—The identification of facial activities from imageshave been of great interest in the field of computer vision.The facial activities are generally characterized by threelevels. The bottom level is the facial feature identification,which recognizes the important facial feature pointssurrounding facial components (i.e., mouth, eyebrow, etc.),which gives the face shape information. The middle level isthe facial action recognition, which determines the facialAction Units (AUs) defined in the Facial Action CodingSystem (FACS), identifying some significant facial activities(i.e., lid tightener, eyebrow raiser, etc.). The topmost level isthe facial expression analysis which finds out the facialexpressions that represent the human emotional states. Thefacial feature identification, AU recognition and expressionrecognition represent the facial activities in three levels, andthey are mutually dependent. However, the recent techniquesrecognize the facial activities in one or two levels, and areidentified separately, either discarding their interactions orlimiting to one way. Unlike the conventional approaches, aBayesian network model is proposed which also makes useof a feature based expression recognition model, tosimultaneously recognise all the three levels of facialactivities. The use of this feature based algorithm helps inincreasing the accuracy of the expressions recognised. In theexisting system we use the image measurements as well asthe Bayesian network in order to recognise the facial featuresas well as the Action Units. Hence there is a greatimprovement in the efficiency of feature detection and AUrecognition. But there is no separate method for therecognition of facial expression other than the Bayesiannetwork. Moreover, unlike the existing system which uses 15AUs, here 19 AUs are taken into consideration which willimprove the efficiency of the Bayesian network based model.Given the facial action model and image observations, allthree levels of facial activities are estimated simultaneouslythrough a probabilistic inference by systematicallyintegrating visual measurements with the proposed model.Index Terms— Bayesian network, expression recognition,facial action unit recognition, facial feature tracking,simultaneous feature recognition and expression analysis.Manuscript received March, 2014.Sumalakshmi.C.H, Computer Science and Engineering,KMCT College of Engineering, Calicut, India, 919946877539.,Adithya.V, Computer Science and Engineering, KMCTCollege of Engineering, Calicut, India, 918129897716.,ISSN: 2278 – 7798I.INTRODUCTIONThe recognition of facial activities in image sequence is asignificant and demanding problem. In recent years, anumber of computer vision techniques have been developedto identify facial activities in three levels. First, in the bottomlevel, facial feature identification, which usually recognizesthe prominent facial feature points surrounding facialcomponents (i.e., mouth, eyebrow, etc.), captures the detailedface shape information. The middle level is the facial actionrecognition which recognizes the facial Action Units (AUs)as defined in the Facial Action Coding System (FACS) [2].This level tracks some meaningful facial activities (i.e., lidtightener, eyebrow raiser, etc.). In the top level, facialexpression analysis is performed which attempts to recognizethe facial expressions that represent the human emotionalstates.The facial feature identification, AU recognition andexpression analysis represent the facial activities in threelevels and they are interdependent problems. For example,facial feature tracking can be used in the feature extractionstage in expression/AUs recognition, and expression/ AUsrecognition results can provide a prior distribution for facialfeature points [1]. However, most of the current methods onlyrecognize the facial activities in one or two levels, and trackthem separately, either ignoring their interactions or limitingthe interaction to one way. In this paper, in contrast to theconventional approaches, we build a probabilistic modelbased on the Bayesian Network (BN) to capture the facialinteractions at different levels. Thus the flow of informationis two-way, not only bottom-up, but also top-down. Inparticular, not only the facial feature identification cancontribute to the expression/AUs recognition, but also theexpression/AU recognition helps to further improve the facialfeature performance. Thus all three levels of facial activitiesare recovered simultaneously through a probabilisticinference by systematically combining the measurementsfrom multiple sources at different levels of abstraction [1].The proposed facial activity recognition system also uses analgorithm based method in order to find out the expression sothat the expression recognized by the proposed system isbased on two methods. Thus the result obtained by theproposed model produces a more accurate result. The systemhas two main stages: offline facial activity modelconstruction and online facial motion measurement andinference. Specifically, using training data and subjectivedomain knowledge, the facial activity model is constructedoffline. During the online recognition, we determine thefacial feature points, and the AUs. These measurements arethen used as evidence to infer the simultaneously recognizethe facial activities.All Rights Reserved 2014 IJSETR717
International Journal of Science, Engineering and Technology Research (IJSETR), Volume 3, Issue 4, April 2014The paper is divided as follows: In Sec. II, the stepsinvolved in facial activity analysis are presented; Sec. IIIdescribes the details of facial activity modeling, BayesianNetwork Based Expression Recognition(Sec.III-A);Algorithm Based Facial Expression Recognition System(Sec.III-B); The paper concludes in Sec. IV with a summary of ourwork and its future extensions.II.STEPS INVOLVEDIn this section, we are going to analyze the various stepsinvolved in the proposed facial activity recognition system.A. Facial Feature IdentificationFacial feature points contain critical information about faceshape and face shape deformation. Exact positioning andidentification of facial feature points are crucial in theapplications such as animation, computer graphics, etc. Forextracting the facial features each of the facial component issegmented separately and the facial features are extracted foreach of them. For each of the facial component, the skinregion is extracted, we then determine the connected region,out of which the facial features are extracted by applyingBezier curves. The various facial points are then marked onthe obtained facial features. In the proposed system we areusing 25 facial points which are depicted in fig.1. The facemodel is composed of 27 features which are defined incorrespondence with a set of 25 facial points.1) By using the Bayesian network to simultaneouslyidentify the facial features and recognize the facialexpressions. The facial features as well as Action units areused in developing the Bayesian Network. A BayesianNetwork is a directed acyclic graph that represents the jointprobability distribution among the variables. It is a unifiedprobabilistic framework that simultaneously represents thefacial activities. Dependencies are found by using theconditional probability. Thus by using this probabilisticBayesian network facial expressions are determined. Giventhe facial action model and image observations, all threelevels of facial activities are estimated simultaneouslythrough a probabilistic inference by systematicallyintegrating visual measurements.Table I. Some of the AUs and their interpretationsFig 1. Facial feature points used in the algorithm.B.AUs RecognitionOver the past decades, there has been extensive research onfacial expression analysis. Here we use a method whichfocuses on recognizing facial actions by observing therepresentative facial appearance changes, usually try toclassify expression or AUs independently and statically. Herewe use 19 AUs as defined in FACS, in order to recognize thevarious facial expressions in a more effective manner[4]. TheAUs are responsible for the identification of some significantfacial activities (i.e., lid tightener, eyebrow raiser, etc.).C. Expression RecognitionThe expression analysis portion identifies six prototypicalfacial expressions namely happy, sad, anger, surprise, disgustand fear. In the proposed system we are making use of twomethods in order to recognize the facial expression. The twomethods employed for finding the expressions are as follows.2) By using an algorithm based expression recognitionsystem to identify the expression. In the proposed system wehave defined 27 features whose measurement is used tocompute the algorithm for expression recognition. The facemodel is composed of 27 features which are defined incorrespondence with a set of 25 facial points[4]. Thisalgorithm based method initially classifies expressions asthose coming under ‗with teeth‘ category and those under‗without teeth‘ category. The five expressions except theexpression sad appear under ‗with teeth‘ category whereasthe without teeth category has all the six expressions.III.FACIAL ACTIVITY MODELINGA .Bayesian Network Based Expression RecognitionThe Bayesian network based expression recognition is thecore of the proposed system. Here the input image ispre-processed by improving the contrast of the picture. Wethen perform the face detection by determining the connectedskin region. In the measurement extraction phase the facialfeatures and the Action Units are determined. This is used indeveloping the Bayesian network which in turn recovers thefacial features, the Action Units and the expression718All Rights Reserved 2014 IJSETR
International Journal of Science, Engineering and Technology Research (IJSETR), Volume 3, Issue 4, April 2014simultaneously. The Simultaneous Facial ActivityRecognition using Bayesian Network is given inFig.2. The Bayesian network is a directed graphical model and is more general to capture complexrelationships among variables.Fig. 2. Flowchart of the BN facial activity recognition system.We propose to employ Bayesian network to model the spatialdependencies among all three levels of facial activities (facialfeature points, AUs and expression) as shown in Fig. 3. Fig. 3is not the final BN model, but a graphical representation ofthe causal relationships between different levels of facialactivities. The Et node in the top level represents the currentexpression; AUt represents a set of AUs; Xt denotes the facialfeature points we are going to track; MAUt and MXt are thecorresponding measurements of AUs and the facial featurepoints, respectively. The three levels are organizedhierarchically in a causal manner such that the level above isthe cause while the level below is the effect. Specifically, theglobal facial expression is the main cause to produce certainAU configurations, which in turn cause local musclemovements, and hence feature points movements. Forexample, a global facial expression (e.g., Happiness) dictatesthe AU configurations, which in turn dictates the facialmuscle movement and hence the facial feature point positions[1].We mainly focus on identifying 6 basic expressions, i.e.,happiness, surprise, sadness, fear, disgust and anger. Thedependency between each level is used to find out thesimultaneous facial activity. Though psychologists agreepresently that there are ten basic emotions, most currentresearch in facial expression recognition mainly focuses onsix major emotions, partially because they are the most basic,and culturally and ethnically independent expressions andpartially because most current facial expression databasesprovide the six emotion labels [1]. Given the measurementsequences, all three level facial activities are estimatedsimultaneously through a probabilistic inference viaBayesian network.ISSN: 2278 – 7798Fig 3. Bayesian Network facial activity model.1) Modeling the Relationships Between Facial Features andAUsIn this paper, we will track 27 facial features and recognize19 AUs, i.e., AU1, 2, 4, 5, 6, 7, 8, 9, 10, 12, 13,14,15,20, 23,24, 25, 26 and 27.The selection of AUs to be recognized ismainly based on the AUs occurrence frequency, theirimportance to characterize the six expressions, and theamount of annotation available [4].The 19 AUs we propose to recognize are all most commonlyoccurring AUs, and they are primary and crucial to describethe six basic expressions. They are the most commonly usedfacial expressions. Even though 19 AUs are used in thispaper, the proposed framework is not restricted torecognizing these AUs alone, given an adequate training dataset.All Rights Reserved 2014 IJSETR719
International Journal of Science, Engineering and Technology Research (IJSETR), Volume 3, Issue 4, April 2014Fig.4. Modeling the relationships between facial featurepoints and AUsThere should be a minimum of 15 AUs to be chosen forproper recognition of six major expressions. The movementof face components is mainly managed by the Facial ActionUnits and it also controls the movement of facial featurepoints.For instance, activating AU1 (inner brow raiser) results in araising the inner eyebrow portion; and activating AU6( cheekraiser) raises the cheek upward. At the same time, thedeformation of facial feature points reflects the action ofAUs. Therefore, we could directly connect the related AUs tothe corresponding feature points around each facialcomponent to represent the casual relationships betweenthem. Take Mouth for example, we use a continuous nodeXMouth to represent 8 facial feature points around mouth,and link AUs that control mouth movement to this node[1].However, directly connecting all related AUs to one facialcomponent would result in too many AU combinations, mostof which rarely occur in daily life. As a result, we introducean intermediate node, e.g., ―CM,‖ to model the correlationsamong AUs. Fig. 4 shows the modeling for the relationshipsbetween facial feature points and AUs for each facialcomponent. Each AU node has two discrete states whichrepresent the ―presence/absence‖ states of the AU. Theintermediate nodes (i.e., ―CB,‖ ―CE,‖ ―CN ,‖ and ―CM‖) arediscrete nodes, each mode of which represents a specificAU/AU combination related to the face components. TheConditional Probability P(Ci pa(Ci )) for each intermediatenode Ci is set manually based on the data analysis, wherepa(Ci ) represents the parents of node Ci . For instance, ―CB‖has five modes, each of which represents the presence of anAU or AU combination related to the eyebrow movement.2)Modeling the Relationships Between AUs and ExpressionIn this section, we will add Expression node at the top levelof the model [1]. Expression represents the global facemovement and it is generally believed that the six basicexpressions (happiness, sadness, anger, disgust, fear andsurprise) can be described linguistically using culture andethnically independent AUs, e.g., activatingAU6 AU12 AU25 produces happiness expression, asshown in Fig. 5(a).We group AUs according to different expressions[4].Several combination of AUs result in different expressions.Generally, grouping of AUs belonging to the same categoryincreases the degree of belief in classifying to that category,as shown in Fig. 5(a) (the combination of AU6 and ).However, combining AUs across differentcategories may result in the following situations: First, an AUcombination belonging to a different facial expression, e.g.,when AU1 occurs alone, it indicates a sadness, and whenAU5 occurs alone, it indicates a surprise, however, thecombination of AU1 and AU5 increases the probability offear as shown in Fig. 5(b); Second, shows increasedambiguity, e.g., when AU26 (jaw drop), an AU for surprise,combines with AU1, an AU for sadness, the degree ofsurprise is reduced and the ambiguity of classification may beincreased as illustrated in Fig. 5(c).Fig. 5. AU combinations. (a) AU12 AU6 (two AUs from thesame category) enhances classification to happiness. (b)AU1 AU5 (two AUs from different categories) becomes afear. (c) AU26 AU1 (two AUs from different categories)increases ambiguity between a surprise and a fear.These relationships and uncertainties are systematicallyrepresented by our final facial activity model as shown in Fig.6. At the top level of the final model, we introduce sixexpression nodes, (i.e., Surp, Sad, Ang, Hap, Dis and Fea),which have binary states to represent ―absence/presence‖ ofeach expression. We link each expression node to thecorresponding AUs. Expressions are inferred from theirrelationships with AUs and reasoning over time. In principle,a probabilistic combination of any relevant facial AUs istaken in order to determine the corresponding facialexpression.720All Rights Reserved 2014 IJSETR
International Journal of Science, Engineering and Technology Research (IJSETR), Volume 3, Issue 4, April 2014Fig. 6. Complete BN model for simultaneous facial activity recognition.B .Feature based Algorithm for facial expressionrecognitionThe value of the various facial features is taken in order todetermine the various expressions. This algorithm basedmethod classifies expressions as those coming under ‗withteeth‘ category and those under ‗without teeth‘ category. Thefive expressions except the expression sad appear under withteeth category whereas the without teeth category has all thesix expressions. The changes happening in various facialfeatures, over the different facial expressions are studied bycomparing their values on different facial images. Study isalso carried out on the images of the people without anyparticular expression. This learning helps to attain a detailedidea regarding the feature point values. Depending uponthese feature values we recognize the various expressions.For example, for the expression surprise to be active, thecondition chosen are raised eyebrows, increased eye width,the eye and eyebrow distance also increases. Likewise wehave feature conditions for each of the expressions. The valueof the features is compared against the threshold valueswhich are obtained from the experimental study. In thismethod the expression determination depends upon the facialfeature values. Some of the significant changes in the featuresfor each of the different expression is Table II.Expression Feature ChangesHappyTeeth presence, Increased lip lengthFrowned brows, Raised brows, IncreasedAngereye width, Lip tightenedFrowned brows, Mouth opened, IncreasedFeareye widthSadFrowned brows, Lowered browsFrowned brows, Lowered brows, ReducedDisgusteye widthMouth opened, Increased eye width,SurpriseRaised browsISSN: 2278 – 7798Table II .Some Of the Feature changes implemented inalgorithm based system.IV.EXPERIMENTS AND RESULTSThe experiment for evaluating the expression was conductedin 3 steps. At first we conduct the experiment using theBayesian based network model alone. We have considered animage set containing 100 images with all six expressions.These 100 images are given as input , one by one to theBayesian network based expression recognition system. Itwas noted that, out of the 100 images, 90 of them turned outto produce correct expression results, whereas 10 of themshowed variation from the actual expression condition.The same experiment was carried out by taking the featurebased expression recognition system alone. Here also thesame 100 images were given as input to the system one byone. Out of the 100 images that we supplied 86 of themproduced correct expression results, whereas 14 of themshowed deviations from the actual expression condition.From the first two steps it could be inferred that the efficiencyof the recognition
facial feature tracking can be used in the feature extraction stage in expression/AUs recognition, and expression/ AUs recognition results can provide a prior distribution for facial feature points [1]. However, most of the current methods only recognize the facial activities in one or two levels, and track
simultaneous facial feature tracking and expression recognition and integrating face tracking with video coding. However, in most of these works, the interaction between facial feature tracking and facial expression recognition is one-way, i.e., facial feature tracking results are fed to facial expression recognition. There is
Simultaneous Facial Feature Tracking and Facial Expression Recognition Yongqiang Li, Yongping Zhao, Shangfei Wang, and Qiang Ji Abstract The tracking and recognition of facial activities from images or videos attracted great attention in computer vision field. Facial activities are characterized by three levels: First, in the bottom level,
Another approach of simultaneous facial feature tracking and facial expression recognition by Li et.al [21] describes about the facial activity levels and explores the probabilistic framework i.e. Bayesian networks to all the three levels of facial involvement. In general, the facial activity analysis is done either in one level or two level.
KEYWORDS: Simultaneous Tracking and Recognition, Facial Feature Tracking, Facial Action Unit Recognition, Expression Recognition and Bayesian Network. I. INTRODUCTION The improvement of facial activities in image sequences is an important and challenging problem. Nowadays, many
recognition, facial feature tracking, simultaneous tracking and recognition. I INTRODUCTION The recovery of facial activities in image sequence is an important and challenging problem. In recent years, plenty of computer vision techniques have been developed to track or recognize facial activities in three levels. First, in the
Recognition James Pao jpao@stanford.edu Abstract—Humans share a universal and fundamental set of emotions which are exhibited through consistent facial expressions. An algorithm that performs detection, extraction, and evaluation of these facial expressions will allow for automatic recognition
simultaneous tracking and recognition of facial expressions. In contrast to the mainstream approach "tracking then recognition", this framework simultaneously retrieves the facial actions and expression using a particle filter adopting multi-class dynamics that are conditioned on the expression. 2. Face and facial action tracking
STORMBREAKER AnthonyHorowitz FUNERALVOICES WHENTHEDOORBELLringsatthreeinthemorning itsnevergoodnews AlexRiderwaswoken , ' . bythefirstchime Hiseyesflickeredopen .
