Enhancing Multimodal Affect Recognition With Multi-Task Affective .

and learned weights across multiple target variables, whichintroduces a form of model regularization [13]. Multi-taskmodels also require a significantly lower number ofparameters compared to the total number of parametersrequired by separately trained models for each individual task,while also allowing the model to inherently learn theinterwoven relationships between the target variables [14].Prior work indicates that MTL outperforms single-tasklearning in terms of predictive performance for a variety oftasks [15], [16]. However, the use of multiple tasks poseschallenges regarding the weighting of each task’s predictiveperformance during training, as different predictive tasksoften vary in nature and intended purpose. Additionally, theappropriate balance of task-specific and shared latentrepresentations within a multi-task model can vary as well andhave a noticeable impact on model performance.In this paper, we investigate the integration of temporalcontextual features from students’ affective sequences as ameans to improve models of student affect through multi-tasklearning. We hypothesize that using students’ future affectivestates as they engage with a game-based learning environmentcan be utilized as an auxiliary multi-task function to improvethe predictive performance of the affect recognition models.Additionally, we explore how to optimally combineinteraction-based and posture-based modalities by exploringpotential deep learning architectures: multi-task fullyconnected feedforward neural networks and cross-stitchneural networks [17]. Finally, we examine the benefit ofincluding a student’s prior affective states as a means ofproviding additional affect sequence information to improvethe performance of affect recognition models. Our resultsindicate that the use of MTL to model affective sequencepatterns from each student leads to improved prediction ofmultiple affective states, and the use of cross-stitch neuralnetworks further strengthens predictive accuracy.II. RELATED WORKA. Multimodal Affect RecognitionDue to their multifaceted perception of student behaviorand demonstrated improvements in predictive performance,multimodal approaches to affect recognition tasks have seena growing interest in recent years. Song et al. use capturedaudio and facial expression data to train a recurrent neuralnetwork model to detect the presence of frustration instudents [18]. Henderson et al. showed the improvedperformance of multimodal affect models through the use ofinteraction- and posture-based modalities while alsoinvestigating the impact of multimodal data fusion onpredictive accuracy [8]. Wu et al. used head pose and eyegaze data to enhance the performance of a facial expressionbased continuous affect recognition model through the use ofa guided temporal attention mechanism [19], while Ghaleb etal. integrated temporal contextual embeddings intomultimodal long short-term memory (LSTM) models trainedon audio and facial expression modalities [20].B. Affective SequencesAffective dynamics has been the subject of growinginterest. While a significant body of prior work focuses onpredicting individual occurrences of various affective states[8], [21], these approaches often ignore the predictiveinformation offered by a student’s overall affective trajectory,such as how a student transitions from one affective state toanother throughout a single learning session. The shifts in astudent’s affective states have been shown to reveal particularrecurring patterns, and thus can provide predictive value inaffect modeling. D’Mello and Graesser investigated a modelof affective dynamics that focused on a cyclic model fromengaged concentration to confusion that enhanced learningoutcomes [4]. Additionally, the authors explored analternative model that resulted in decreased learning asstudents transitioned from engaged concentration toconfusion, frustration, and boredom, respectively. Andres etal. expanded on this work by exploring the usage of shortertransitory patterns, namely two-step patterns that consisted ofonly two affective states [22]. The authors investigated thepresence of prolonged states of affect by analyzing four-steppatterns of the same affective state, focusing on thecorrelation between particular affective patterns and studentlearning outcomes. Ocumpaugh et al. focused on thefrequency of an expanded set of four-step prolongedemotions in addition to three-step patterns consisting of twoaffective states as they related to student actions in a blendedlearning system [23]. Botelho et al. investigated theperformance of two-step affective transitions in studentsengaged with an intelligent tutoring system, in addition toinvestigating the time a student spent in a single affectivestate before transitioning to another state [24]. While therehas been prior work investigating the use of sequentialmodeling (such as LSTMs) for predicting student affect [25],these approaches use the sequences of student behavioral dataas input to predict a single affect label instead of predictingan affect sequence as output. Our work addresses this issueby exploiting the temporal information in students’ affectivesequences to improve the predictive performance of the affectmodels through the use of auxiliary output multi-taskpredictions during the training phase.C. Multi-Task LearningRecent years have seen MTL applied to a variety oftasks, including computer vision [26], natural languageprocessing [27], and transfer learning [28]. While manymulti-task models consist of a series of feedforward neuralnetwork layers, alternative deep learning architectures havebeen explored as well, including sluice networks [29], deeprelationship networks [16], and cross-stitch networks [17].More recently, MTL has been investigated as an alternativeapproach to student modeling, including for predicting posttest scores based on individual questions [15], modelingstudent mastery of multiple concepts [30], and estimatingself-reported measures of interest and engagement with agame-based learning environment [31]. Prior applications ofmulti-task learning within affective computing involve theprediction of multiple states of affect by a single model [25].III. DATASETThe dataset used to investigate our multi-task, affectivedynamics-based approach consists of posture and interactiondata captured during student engagement with a game-basedlearning environment for training emergency medical skills,TC3Sim. Posture data was captured using a Microsoft Kinectsensor mounted on a tripod facing the front of each student,while the interaction data was extracted from gameplay tracedata logs. Students’ affective states during gameplay werediscreetly annotated and recorded in real time by two fieldobservers in accordance with the Baker Rodrigo OcumpaughMonitoring Protocol (BROMP) [32]. For this study, data wasobtained from a population of 119 students (83% male, 17%female) at The United States Military Academy.

A. TC3Sim Game-Based Learning EnvironmentTC3Sim is a serious game-based learning environmentthat is widely used to provide training for administeringmedical care within a 3D virtual environment. During thegame, students assume the first-person role of a medic withinvarious simulated narratives (Fig. 1). Students progressthrough the game by completing a series of scenarios that arecentered on different non-player characters (NPCs) thatsustain injuries and require medical attention. The students’characters administer care in accordance with medicalprotocol that is presented to each student prior to beginningthe gameplay session. Each student engaged with TC3Simindividually, with each gameplay session lastingapproximately one hour.IV. METHODOLOGYA. MTL with Affective SequencesTo adapt the single-task affect recognition approach toan MTL formulation, the target variables were expanded toinclude a one-hot representation of each possible affectivestate. The one-hot vector was indicative of the affective stateBi 1 that followed the current BROMP observation Bi (Fig. 2).Bi was a binary indicator of the presence of one of the fivepossible affective states. Therefore, the multi-task modelswere modeled using a label vector of size 6 (binary indicatorof a single affective state one-hot vector of size 5). Usingthe affect model for bored as an example, the multi-taskoutput vector for a positive annotated occurrence of boredfollowed by an annotation of confused would be [1, 0, 1, 0, 0,0], while a negative annotated occurrence of bored followedby a subsequent annotation of frustrated would be [0, 0, 0, 0,1, 0].BinaryBoredConfusedEngagedBiBi 1Bi 1Bi 1Frustrated SurprisedBi 1Bi 1Fig. 2. Multi-task feature vector representation.Fig. 1. TC3Sim game-based learning environment.B. BROMP ProtocolGround-truth labels of student affect were collectedusing the BROMP protocol [32]. BROMP is a codingprocedure designed to produce quantitative labels of studentaffect and behavior using field observers and allows forefficient and discreet real-time annotations of learner affectbased on holistic observations within real-world conditions.BROMP has been widely used in research on affect-sensitivelearning technologies [33]. Notably, BROMP observationsdo not rely on a single data channel (e.g., facial expression).BROMP enables annotations to be contextually informed bythe observers and includes practices for minimizingdisruptions during annotation. Since BROMP is anobservational protocol, it mitigates issues with self-reportssuch as recall, self-awareness, and self-presentation [34].Observers walk around the perimeter of the classroomand discreetly annotate observed students’ affective statesusing a hand-held device. Annotations of affect occurred in20-second intervals and were intended to be captured asdiscreetly as possible to minimize the influence of theobservers’ presence and disruption of the students’ gameplay.Prior to this study, the two observers established an inter-rateragreement exceeding 0.6 in terms of Cohen’s Kappa [35].Any observations indicating disagreement between theobservers were removed from the dataset, resulting in a finaldataset consisting of 755 labeled affective states. A total of435 of the BROMP observations were labeled as engagedconcentration (M 0.576, SD 0.239), 174 as confused (M 0.231, SD 0.185), 73 as bored (M 0.097, SD 0.161),32 as frustrated (M 0.042, SD 0.182), 29 as surprised (M 0.038, SD 0.045) and 12 as anxious (M 0.016, SD 0.089). Due to the low number of observations of anxious,this affective state is not considered in any of the followinganalyses.Because the multi-task models are predicting futureoccurrences of each affective state, it is impractical to utilizethese labels as input features as this information would not beavailable in a run-time environment. As a result, we use theselabels as auxiliary output variables for the purpose ofboosting the predictive performance of the multi-task modelsrelative to the current affective state, an approach that hasbeen previously demonstrated to improve predictiveperformance [27], [36]. This process can be employed whencertain features are unhelpful for predicting other outputvariables or are not available until after the predictions aremade, allowing the features to be used to present additionalinformation to the model during the training process only[36]. In this case, presenting the subsequent affective state tothe multi-task model allows the model to potentially observedifferential patterns in student behavior prior to transitioningto another affective state. For example, a student’s posturalbehavior while currently in a state of engaged concentrationmay fluctuate depending on if the subsequent affective stateis also engaged concentration or a different state such asconfusion. By introducing additional predictive tasks, themodel is trained to extract temporal features and patternsfrom affective sequences that can improve the model’sprediction of the current affective state. The occurrences ofeach two-step affective sequence are shown in Fig. 3.The most common affect sequences are persisting statesof engaged concentration (denoted as “Concentrating” inFig. 3), consecutive states of confusion, and alternatingbetween these two states. This result aligns with the proposedmodel by D’Mello and Graesser [4]. Other notable sequencesare students transitions between states of bored and engagedconcentration, particularly as this indicates that students areoften capable from returning to an engaged state whilepreviously being in a state of relative disengagement, abehavior previously observed by Andres et al. [22].

InputActivationMapsTask ATask B A AA BAB BBCross-StitchUnitOutputActivationMaps AA BAShared Task A BB ABShared Task BFig. 4. Visualization of a cross-stitch network for weighting sharedrepresentations between task A and task B.whether the level of connectivity within each architecture hasan observable impact on the predictive performance of theaffect models.Fig. 3. Frequency of each affective state and corresponding subsequentstate.B. Cross-Stitch NetworksAn active area of investigation in multi-task deeplearning is determining the appropriate level of layerconnectivity across each task. A multi-task model thatconsists of only fully connected layers contains the highestlevel of connectivity, as each layer propagates the same fullyshared data representation across all tasks with the exceptionof task-specific output layers. Alternatively, to avoid anyinter-task communication within the multi-task framework,separate models can be trained for each task, so that thetrained weights are unique for each output. While prior workapplying multi-task learning for student modeling utilizes fullconnectivity across tasks within the model’s hidden layers[15], [30], other work within computer vision has exploredthe benefits of “split” neural architectures, or architecturesthat maintain a degree of separation between tasks within apre-determined subset of the model’s hidden layers.Cross-stitch networks were proposed by Misra et al. as ageneralizable approach to implementing “split” architecturesby implementing parameterized linear combinations betweena network’s hidden layers that can learn optimal weightingsbetween shared and task-specific latent representations [17].This approach allows feature representations to be combinedwithin certain hidden layers and shared across tasks whilealso maintaining separation between task-specificrepresentations. For example, in the case of modeling twotasks (A and B), a learned weight matrix is used toparameterize the linear combinations of multiple tasks ( AB, BA) as well as activations from a single task ( AA, BB) (Fig.4). A value of 0.5 for indicates that the representations areequally shared, with a value of 0 or 1 indicating that therepresentations are completely separate. Specifically,Equation 1 shows how the shared representation x̂ iscalculated at row i and column j by a cross-stitch unit thattakes an input activation map, x:", α!!# ", & (α%!𝑥%𝑥%!%", α!% 𝑥!α%% * #𝑥 ", &(1)%The values of the weight matrix are adjusted duringbackpropagation, with the partial derivatives easilycalculatable as the cross-stitch units are modeled with linearcombinations. We evaluate cross-stitch networks alongside amulti-task variant of fully connected feedforward neuralnetworks in our modeling of student affect to investigateC. Posture-Based Feature EngineeringThe features representing the posture data captured fromthe Kinect sensor are generated from three tracked vertices:top skull, center shoulder, and head. These vertices areselected based on prior literature that has investigated theeffectiveness of the posture modality within affectrecognition tasks [37]. Each posture-based feature iscalculated based on the postural position and movement ofeach student that occurs within the 20-second observationalwindow prior to each BROMP observation. Eighteen distinctfeatures are generated for each vertex, including the mostrecent observed distance, minimum and maximum observeddistance, median observed distance, variance in the observeddistances, and most recent Z-coordinate value. In this work,“distance” refers to the Euclidean distance between eachvertex and the Kinect sensor. In addition to these features, theminimum, maximum, median, and variance in the distance iscalculated across the time windows of 5, 10, and 20 secondsthat precede the corresponding BROMP observation. Severaladditional features were distilled that calculated the totalchange in the position (relative to the prior vertex’s locationin 3D coordinate space) and distance (relative to the priorvertex’s distance from the Kinect sensor) across thepreceding 3- and 20-second time windows. Using the mediandistance of the head vertex across the entire dataset, the finalfeatures were calculated to represent whether the student wasleaning forward, backwards, and upright using the currentposition of the head vertex. These postural features wereaveraged across time windows of 5, 10, and 20 seconds, inaddition to the entire gameplay session that had transpiredprior to the current BROMP observation.In addition to the spatial posture features, temporallybased features were generated using the calculated distancebetween the (x, y, z) coordinates of two consecutive sensorreadings from the head vertex. These delta values were usedto generate velocity-based features averaged across timewindows of the preceding 3, 5, 10, and 20 seconds prior toeach BROMP observation. The mean, median, max, andvariance of the calculated velocity values were used asfeatures. Forty-eight new features were produced from thisprocess. As a result of the high number of features generatedfrom this process, the center shoulder and top skull verticeswere not utilized for generating temporally based features.D. Interaction-Based Feature EngineeringThe interaction-based features are distilled from thegenerated log files that record each student’s in-game actionsand movements, and the condition of particular NPCsthroughout the game [8]. Features that represent the conditionof the NPCs that receive medical attention include thechanges in systolic blood pressure, exposed wound type,

heart rate, and lung volume. Additional features were distilledthat represented students’ in-game actions such as performinga check of an NPC’s vital signs and requesting an emergencymedical evacuation. The interaction-based features werecalculated across 20-second time intervals prior to eachBROMP observation. Features were represented by using asummative count or averaging across the preceding timeinterval or were represented using statistical calculations suchas standard deviation or median values for information suchas a virtual patient’s blood pressure. In total, thirty-nineinteraction-based features were generated during this process.E. Feature SelectionDue to the high number of features from the differentmodalities, feature selection was performed on each modalityusing forward feature selection. Forward feature selectioniterates through a set of features in a greedy fashion beginningwith a single feature and increasing the number of featuresaccording to their predictive performance on the targetvariable. This process iterates until a pre-determinedthreshold has been reached or until all features have beenevaluated. However, due to the greedy search heuristic, thefeature selection is weighted more heavily towards featuresthat are evaluated earlier (e.g., the first feature evaluated isalways selected). To mitigate any bias based on the arbitraryordering of the feature candidates, we run 100 independentiterations of the forward feature selection. Each iteration usesa randomized feature ordering, and the features that are mostfrequently selected across all iterations are selected fortraining the affect recognition models. This approachprovides a compromise between the speed of a greedy searchheuristic and the computational cost of an exhaustive featureselection process [38]. Forward feature selection wasperformed on each modality separately, with the ten mostpredictive features per modality being combined at a featurelevel for training the affect recognition models.F. Affect Model EvaluationTo evaluate the performance of the multi-task models(fully connected and cross-stitch networks), we train a seriesof single-task neural and non-neural baseline models inaddition to several non-neural multi-task baseline models.The baseline models were k-nearest neighbor, elastic net,random forest, and feed-forward neural network. These wereselected as baselines due to their capabilities of both singletask and multi-task learning. The single-task baseline modelsdemonstrate the performance of models without any affectivedynamics context, while the multi-task non-neural baselinemodels verify that the deep learning-based approaches (fullyconnected and cross-stitch networks) achieve higherperformance with the affective dynamics context than nonneural multi-task models.Each model was evaluated with nested ten-fold crossvalidation, with each fold split at a student-level to preventdata leakage across the training, validation, and test sets.Within each outer cross-validation fold, the data werestandardized to ensure a mean of zero and standard deviationof one prior to performing feature selection. Hyperparametertuning was performed using three-fold cross-validationwithin the training data of the nested outer cross-validation.The hyperparameters evaluated were the number of nearestneighbors (k-nearest neighbors), ratio of L1 and L2regularization (elastic net), number of estimators (randomforest), and the number and size of the hidden layers (neuralnetwork). Each deep learning model’s hidden layer used ahyperbolic tangent activation function due to thestandardization of the data, as well as a dropout probabilityof 0.5 in the last hidden layer to mitigate potential overfitting.The loss function for the feedforward single-task networkwas binary cross entropy. Additionally, minority cloning wasemployed as an oversampling technique to resolve the classimbalance present within each affective state’s dataset. Thisprocess clones each instance of the minority class until theclass distribution is brought to a more uniform level.An active line of investigation in MTL is determining theoptimal distribution of the loss term across the different tasks.A common naïve approach to MTL loss is to assign a uniformweight to the loss term for each individual task t whencalculating the summative loss term:𝐿&'&() , 𝑊& 𝐿&&(2)However, as the auxiliary tasks of predicting futureaffective states is distinguishable from the task of predictingthe current affective state, we explore the use of a lossfunction that uses uncertainty weighting for each individualtask [26]. The weight Wt for each task is determined bymaximizing the log likelihood of an assumed multivariateGaussian distribution. By optimizing for the modelparameters θ and observation noise σ, the following lossfunction is derived:1* 𝐿& (θ) log (σ& )& 2σ&𝐿&'&() ,(3)In this way, optimizing for σt for each task t allows therelative weight of each task-specific loss function (i.e., thefirst term in Equation 3) to be learned from the data duringthe training process, while the second term in Equation 3 actsas a regularization term to prevent σ from increasingexponentially, which prohibits the model from learning. Thisallows the model to assign different weighted losses betweenthe primary task (predicting the current affective state) andthe secondary auxiliary tasks (predicting the subsequentaffective state).In addition to the single-task baseline model, four multitask deep learning models were evaluated, uniformlyweighted and uncertainty-weighted fully connected networksand cross-stitch networks. Each deep learning model wastrained for 100 epochs, with early stopping implementedusing the validation set and a patience of 10 epochs. Eachnetwork contained either two or three hidden layers with eachlayer containing either 8, 16, 32, or 64 nodes. For each crossstitch model, each pair of hidden layers contained a crossstitch unit. Data standardization, feature selection, andminority cloning occurred within each outer cross-validationiteration using the training folds

providing additional affect sequence information to improve the performance of affect recognition models. Our results indicate that the use of MTL to model affective sequence patterns from each student leads to improved prediction of multiple affective states, and the use of cross-stitch neural networks further strengthens predictive accuracy. II.