Multi-task Learning - University Of British Columbia
Multi-task LearningRamtin Mehdizadeh SerajJan 2014SFU Machine Learning Reading Group
The standard methodology in machine learning-learning one task at a time-Large problems are broken into small,reasonably independent subproblems that arelearned separately and then recombined
Motivation- A net with a 1000x1000 pixel input retina isunlikely to learn to recognize complex objectsin real-world scenes- But what if we simultaneously train a net torecognize object outlines, shapes,edges,regions, subregions, textures, reflections,highlights, shadows, text, orientation,size,distance, etc.,
Concepts and General View- According to Wikipedia :Multi-task Learning is anapproach to learns a problem together with other relatedproblems at the same time, using a shared representation.Task1Data Set 1TrainingModel 1Task1 Data Set 1Task 2Data Set 2TrainingModel 2Task 2.Task nData Set nData Set 2.TrainingModel nTaskData Set nModel 1TrainingModel 2Model n
Relatedness-Learning tasks with the aim of mutual benefit-Assumption : All tasks are related- Example 1 : Different classification tasksSpam filtering - Everybody Has a slightly differentdistribution over spam or not-spam emailsbut there is a common aspect across users.Idea : Learning together can be a good regularizer
RelatednessExample 2 :ImageCategorization
RelatednessOther examples:- Web Page Categorization [chen et al ICML09]Page categories can be related- Movie Ranking [Yu et. al NIPS 06]similar tastes between users
Learning simultaneously- Inductions of multiple task are performedsimultaneously to capture intrinsic relatedness- The main question : How to learn ?
Learning Methods- Joint feature learning : the simplest idea- Mean-regularized MTL : Penalizes thedeviation of each task from the mean- Shared parameter gaussian process- Low rank regularized- Alternating structural optimization- [will discuss later]
Shared Representation-Shared Hidden node in a Neural Network:The simplest one can be a neural networkshared hidden units among tasks .- Shared Parameter:Like Gaussian process- Regularization-based :Mean , Joint feature table, .
Shared RepresentationSharing Hidden Nodes in Neural Network- A set of hidden units are shared amongmultiple tasks.(goal :improving generalization)Task1Task2Task3Output LayerHidden Layers (Shared)input1input2input3input Layer
Shared Representation-Joint Feature Learningcreating a common set of featuresTask1Feature 1Feature 2Feature 3Feature 4Feature 5Task2 Task3Task4
MTL with Joint Feature learning-Using Group Sparsityl1/l2-norm regularization
An Application In NLP- A unified architecture for Natural LanguageProcessing deep neural net with multi tasklearning (by Collobert and Watson)-Tasks :POS, NER, Chunking, SemanticRoles,.-Relatedness : Are these tasks related ?-Shared Representation: NN layers
An Application In NLP - Intro- Tasks :1. POS (Part of Speech Tagging): labelingeach word with a unique tag that shows itstactic roles, ex. adverb, noun,.2. Chunking: labeling segments of asentence with syntactic constituents
An Application In NLP - Intro3. Named Entity Recognition: Labeling atomicelements in the sentence into categories suchthat “Location”, “Person”4. Semantic Role Labeling: Giving a semanticrole to a syntactic constituent of a sentence.Example: [John]Arg0 [ate]Rel [the apple]Arg1
An Application In NLP Regular approachesRich HandDesignedFeaturesShallow ClassificationAlgorithm like SVMModel for acertain taskSelecting features by empirical process (trial and error)Task-based algorithm selection
An Application In NLP 1 new approach- Deep Neural Network- Feature extraction inseveral layers using backpropagation
An Application In NLP 2 new approach- First Layer : featuresfor each words- Second Layer : featuresfor the input sentence(sequenced based)- Following layers :Classical NN layers
An Application In NLP 3Look up tables layer- for word i in the Dictionaryconsidering a d-dimensionalspaceLTw(i) Wi-W : parameters to be learnt- For solving variable sentencelength: Considering fixed sizewindow size around each word.
An Application In NLP 4NN and Max Layer- Time Delay Neural Network :perform linear operation overthe input words.- Max Layer : Captures the mostrelevant features over thesentence.
An Application In NLP 5Output and Algorithm- Using softmax for jointlearning- Algorithm (training in thestochastic manner) :1. select the next task2. select a random training example forthis task3. Use gradient for updating NN4. go to step 1
Results
What if tasks are not totally related- If the tasks have a group structures Clustered Multi-task learninge.g. tasks in the yellow group arepredictions of heart related diseasesand in the blue group are brain relateddiseases.more information : Bakker and Heskes JMLR 2003
What if tasks are not totally related- If the tasks have a tree structures Multi-task Learningwith Tree Structuresmore information :Tree-Guided Group Lasso(Kim and Xing 2010 ICML)
What if tasks are not totally related- If the tasks have a graph structures Multi-task Learningwith Graph Structuresmore information :Graph-guided Fused Lasso (Chen et. al. UAI11)
Connection to other ML topics
Software PackagesMALSAR: Multi-tAsk Learning via StructurAlRegularization-Implemented by Biodesign Institute of Arizona StateUniversity
Main References- Caruana, R. (1997). Multitask Learning. Machine Learning, 28(1), 41–75. doi:10.1023/A:1007379606734- Collobert, R., & Weston, J. (2008). A unified architecture for naturallanguage processing: Deep neural networks with multitask learning.Presented at the Proceedings of the 25th international conference .- Lounici, K., Pontil, M., Tsybakov, A. B., & van de Geer, S. (2009, March 8).Taking Advantage of Sparsity in Multi-Task Learning. arXiv.org.- Zhang, Y., & Yeung, D.-Y. (2012, March 15). A Convex Formulation forLearning Task Relationships in Multi-Task Learning. arXiv.org.- Zhou, J., Chen, J., & Ye, J. (2012) Multi-Task Learning , Theory,Algorithms, and Applications, SDM
Thanks for you attentionAny Question ?
1. select the next task 2. select a random training example for this task 3. Use gradient for updating NN 4. go to step 1. Results. What if tasks are not totally related - If the tasks have a group structures Clustered Multi-
Registration Data Fusion Intelligent Controller Task 1.1 Task 1.3 Task 1.4 Task 1.5 Task 1.6 Task 1.2 Task 1.7 Data Fusion Function System Network DFRG Registration Task 14.1 Task 14.2 Task 14.3 Task 14.4 Task 14.5 Task 14.6 Task 14.7 . – vehicles, watercraft, aircraft, people, bats
Plan for Today Multi-Task Learning -Problem statement-Models, objectives, optimization -Challenges -Case study of real-world multi-task learning Transfer Learning -Pre-training & fine-tuning3 Goals for by the end of lecture: -Know the key design decisions when building multi-task learning systems -Understand the difference between multi-task learning and transfer learning
WORKED EXAMPLES Task 1: Sum of the digits Task 2: Decimal number line Task 3: Rounding money Task 4: Rounding puzzles Task 5: Negatives on a number line Task 6: Number sequences Task 7: More, less, equal Task 8: Four number sentences Task 9: Subtraction number sentences Task 10: Missing digits addition Task 11: Missing digits subtraction
Task 3C: Long writing task: Composition Description 25 A description of your favourite place Task 4A: Short writing task: Proofreading and editing 26 Task 4B: Short writing task: Planning 28 Task 4C: Long writing task: Composition Recount 30 The most memorable day of your life Summer term: Task 5A: Short writing
knowledge, our work is one of the first to apply the multi-task learning model for siRNA efficacy analysis for learn-ing regression models. To test our multi-task regression learning framework, extensive experiments were conducted to show that multi-task learning is naturally suitable for cross-plat-form siRNA efficacy prediction.
Task Updates: Right now, each team has a flow running every hour to check for updates and update the tasks list excel Manual Task Creation: Runs when Task is created manually in planner, removes task content and sends email to the creator to use forms for task creation Task Completion: Runs when task is completed to update
Nov 29, 2016 · Starting A New Committee, Task Force or Work Group. Once the recommendations of the task force have been received, the task force is foregone. RTC task forces include: Advising Policy Task Force Program Revisions Task Force . NOTE: In the future, work groups and task forces should u
1 In the Task tab, click the Gantt Chart button to select the Gantt Chart view. This view contains the Task Mode column. 2 Select the task mode from the drop-down list for the task. 3 Hover the pointer over the Task Mode icon to review the task mode. 4 Click the Task Mode drop-down list to change the task mode back to Manually Scheduled.
