Semi-supervised New Event Type Induction And Event Detection

Semi-supervised New Event Type Induction and Event DetectionLifu Huang and Heng JiComputer Science DepartmentUniversity of Illinois at ctMost previous event extraction studies assumea set of target event types and correspondingevent annotations are given, which could bevery expensive. In this paper, we work on anew task of semi-supervised event type induction, aiming to automatically discover a set ofunseen types from a given corpus by leveraging annotations available for a few seen types.We design a Semi-Supervised Vector Quantized Variational Autoencoder framework toautomatically learn a discrete latent type representation for each seen and unseen type andoptimize them using seen type event annotations. A variational autoencoder is further introduced to enforce the reconstruction of eachevent mention conditioned on its latent typedistribution. Experiments show that our approach can not only achieve state-of-the-artperformance on supervised event detection butalso discover high-quality new event types. 11IntroductionEvent extraction is a task of automatically identifying and typing event trigger words (Event Detection), and extracting participants for each trigger(Argument Extraction) from natural language text.Traditional event extraction studies (Ji and Grishman, 2008; McClosky et al., 2011; Li et al., 2013;Chen et al., 2015; Yang and Mitchell, 2016; Liuet al., 2018; Nguyen and Nguyen, 2019; Lin et al.,2020; Li et al., 2020) usually assume there exists aset of predefined event types and argument roles, sothat supervised machine learning models, e.g., deepneural networks, can be employed to extract eventsfor each type based on human annotations. However, in practice, it is usually very expensive andtime-consuming to manually craft an event schema,which defines the types and complex templates of1The programs are publicly available for research purposeat https://github.com/wilburOne/SSVQVAEEvent MentionsEvent TypesAnnotated MentionsSeen TypesNew MentionsUnseen TypesFigure 1: Semi-supervised new event type induction:discovering a set of new event types and their eventmentions given the annotations for a few seen types.the expected events. Moreover, the coverage ofmanually crafted schemas is often very low, making them fail to generalize to new scenarios.Recent studies have shown that it’s possible toautomatically induce an event schema from rawtext. Some researchers explore probabilistic generative methods (Chambers, 2013; Nguyen et al.,2015; Yuan et al., 2018; Liu et al., 2019) or ad-hocclustering-based algorithms (Huang et al., 2016)to discover a set of event types and argumentroles. Several studies (Huang et al., 2018; Lai andNguyen, 2019) also explore zero-shot and few-shotlearning approaches to leverage available resourcesand extend event extraction to new types. Generally, event schema induction can be divided intotwo steps: event type induction, aiming to discovera set of new event types for the given scenario, andargument role induction which discovers a set of argument roles for each type. In this work, we focuson tackling the first problem only.We propose a task of semi-supervised event typeinduction, which is shown in Figure 1 and aims toleverage available event annotations for a few types,which are called as seen types, and automaticallydiscover a set of new unseen types, as well as theircorresponding event mentions. As a solution, wedesign a new Semi-supervised Vector QuantizedVariational Autoeocoder framework (short as SSVQ-VAE) which first assigns a discrete latent type

representation for each seen and unseen type, andoptimizes them during the process of projectingeach candidate trigger into a particular seen or unseen type. The candidate triggers are discoveredwith a heuristic approach.Experiments under the setting of both supervised event detection and new event type inductiondemonstrate that our approach can not only detectevent mentions for seen types with high precision,but also discover high-quality new unseen types.2ApproachSeen Types. .Encoderfc(vt)TriggerRepresentationyt Efc(vt)Encoderfe(vt)LinearDecoderfr(zt, yt)LinearBERT EncodingEAymanEwasEarrestedEandAyman was arrested andEwasEsentencedEtoElifeEinEprisonwas sentenced tolifeinprison.Figure 2: Architecture tolifeinprisonAs Figure 2 shows, given an input sentence, wefirst automatically discover all candidate triggersand encode each trigger with a contextual vectorLinear(DevlinClassifier et al., 2019) enusing a pre-trained BERTcoder. Then, we predict the type of each candidatetrigger by looking up a dictionary of discrete latentrepresentations of all seen and unseen types. Meanwhile, to avoid the type prediction to be over-fittedto seen types, we apply a variational autoencoder(VAE) as a regularizer to first project each trigger into a latent variational embedding and thenreconstruct the trigger conditioned on its type distribution.2.12.2Event Trigger IdentificationTrigger Representation LearningGiven a sentence s [w1 , ., wn ], where we assume wi is identified as a candidate trigger, we usea pre-trained BERT encoder to encode the wholesentence and get a contextual representation for wi .If wi can be split into multiple subwords or words,we use the average of all subword vectors as thefinal trigger representation.2.3Seen TypeAnnotationsUnseen Typesto OntoNotes senses as candidate triggers. In addition, the concepts that can be matched with verbsor nominal lexical units in FrameNet (Baker et al.,1998) are also considered as candidate triggers.Event Type Prediction with VectorQuantizationTo predict a type for a candidate trigger, an intuitive approach is to learn a classifier using the eventannotations of seen types. However, as we also aimto discover a set of unseen types, without any annotations, the classifier for the unseen types cannotbe optimized.To solve this problem, we employ a Vector Quantization (Gersho and Gray, 2012) strategy. We firstdefine a discrete latent event type embedding spaceE Rk d , where k is the number of candidateevent types, and d is the dimensionality of eachtype embedding ei . Each ei can be viewed as thecentroid of the triggers belonging to the corresponding event type. For each seen type, we initializee with the contextual vector of a trigger which israndomly selected from the corresponding annotations. For each unseen type, we initialize e with thecontextual vector of another trigger which is randomly picked from all unannotated event mentions.Assuming there are m seen types, we arbitrarilyassign E [1:m] as their type representations.Given a candidate trigger t and its contextual vector v t , we first apply a linear encoder fc (v t ) Rdto extract type-specific features. Then, we computea type distribution y based on fc (v t ) by looking upall the discrete latent event type embeddings withinner-product operationSimilar to (Huang et al., 2016), we identify allcandidate triggers based on word sense induction.Specifically, for each word, we disambiguate itssenses and link each sense to OntoNotes (Hovyet al., 2006) using a word sense disambiguationsystem — IMS (Zhong and Ng, 2010) 2 . We consider all noun and verb concepts that can be mappedThe feature encoder fc (.) is optimized using allevent annotations for seen types (the cross-entropyterm in Equation 2) and event mentions for unseentypes (the second term in Equation 2 3 ). The intuition of the second term in Equation 2 is that, for2We use the OntoNotes based IMS word sense disambiguator (https://github.com/c-amr/camr)3We only apply this term when we know the new eventmentions do not belong to any seen typesy t Softmax(E [1:k] · fc (v t ))(1)

each new event mention, we don’t know the correct type but we know that the type must be froma set of unseen types, so we maximize the marginbetween the probability of the most likely unseentype and the highest probability of the incorrectseen type.Lc X(t,ỹt ) Ds ỹ t log(y t ) X[1:m]max(y t) Gaussian distribution. For each unlabeled candidate trigger t, the likelihood p(t) approximates toanother variational lower boundlog p(t) Xq(y t)( L(t, y)) q(y t) log q(y t) L(t)ywhere q(y t) is obtained from Equation 1.As for model implementation, given a candidate trigger t and its contextual embedding v t ,we first pass it through an encoder fe (v t ) to extract features. As we assume the latent variatonal embedding z t follows Gaussian distributionz t N (µt , σ t ), we apply two linear functionsto obtain the mean vector µt fµ (fe (v t )) anda variance vector σ t fσ (fe (v t )). For decoding, we employ another linear function to reconstruct v t from the concatenation of z t and y t :0v t fr ([z t : y t ]). We optimize the followingobjective for the semi-supervised VAE[m:k]max(y t)t Du(2)where ỹt is the ground truth label. Ds and Dudenote the set of annotated event mentions for seentypes and new event mentions for unseen types.[1:m][m:k]ytand y tare the type prediction scores forseen and unseen types respectively.To optimize the type embeddings E, we followthe VQ objective (van den Oord et al., 2017) anduse l2 error to move the type vector ei towardsthe type-specific feature fc (v t ) (the first term inEquation 3) while ei of t is determined by y t . Tomake sure fc (.) commits to an embedding, we adda commitment loss (the second term in Equation 3)Lvq sg(fc (v t )) ei 2 fc (v t ) sg(ei ) 2(3)Lv Xt DuL(t) XL(t, y)The overall loss function for optimizing thewhole SS-VQ-VAE framework isL αLc βLvq γLvwhere sg stands for the stop gradient operator tomake its operand to be a non-updated constant. Theoutput of sg is the same as the input in the forwardpass, and it is zero when computing gradients inthe training process.2.4Variational Autoencoder as RegularizerTo avoid the type prediction to be over-fitted tothe seen types, we employ a semi-supervised variational autoencoder as a regularizer. The intuitionis that each event mention can be generated conditioned on a latent variational embedding z andits corresponding type distribution y, which is predicted by the approach described in Section 2.3.We first describe the semi-supervised variationalinference process. It consists of an inference network q(z t) which is a posterior of the learning ofa latent variable z given the trigger t, and a generative network p(t z, y) to reconstruct the candidatetrigger t from the latent variable z and type information y. For each candidate trigger t with humanannotated label y, the likelihood p(t, y) can be approximated to a variational lower boundlog p(t, y) log p(t y, z) KL(q(z t) p(z)) L(t, y)where log p(t z, y) is the expectation of reconstruction of t conditioned on z and y, p(z) is the prior(4)(t,y) Ds(5)where α, β and γ are hyper-parameters to balancethese three objectives.33.1Experiments and ResultsDatasetWe perform experiments on Automatic ContentExtraction (ACE) 2005 dataset and evaluate ourapproach under two settings: (1) supervised eventextraction, where the target types include 33 ACEpredefined types and other, thus k is set as 34. Giving all candidate triggers, the goal is to correctlyidentify all ACE event mentions and classify theminto corresponding types. We follow the same datasplit with prior work (Li et al., 2013; Nguyen et al.,2016; Yang and Mitchell, 2016) in which 529/30/40newswire documents are used for training/dev/testset. (2) new event type induction, where we followa previous study (Huang et al., 2018) and use top10 most popular event types from ACE05 data asseen and the remaining 23 types as unseen. Givenall ACE annotated event mentions, the goal of thistask is to test whether the approach can automatically discover the remaining 23 unseen ACE typesand categorize each candidate trigger into a particular seen or unseen type. In this experiment, k isset as 500.