Experimental Setup, Multi-class Vs. Multi-label Classification, And .

Experimental Setup,Multi-class vs. Multi-label classification,andEvaluationCMSC 678UMBC

Central Question: How Well Are We Doing?ClassificationRegressionClusteringthe task: what kindof problem are yousolving? Precision,Recall, F1 Accuracy Log-loss ROC-AUC (Root) Mean Square Error Mean Absolute Error Mutual Information V-score

Central Question: How Well Are We Doing?ClassificationRegressionClusteringthe task: what kindof problem are yousolving? Precision,Recall, F1 Accuracy Log-loss ROC-AUC (Root) Mean Square Error Mean Absolute Error Mutual Information V-score This doesnot have tobe the samething as thelossfunctionyouoptimize

OutlineExperimental Design: Rule 1Multi-class vs. Multi-label classificationEvaluationRegression MetricsClassification Metrics

Experimenting with Machine LearningModelsAll your dataTraining DataDevDataTestData

Rule #1

Experimenting with Machine LearningModelsWhat is “correct?”What is working “well?”Training Dataset hyperparametersLearn model parameters fromtraining setDevDataTestData

Experimenting with Machine LearningModelsWhat is “correct?”What is working “well?”Evaluate the learned model on devwith that hyperparameter settingTraining Dataset hyperparametersLearn model parameters fromtraining setDevDataTestData

Experimenting with Machine LearningModelsWhat is “correct?”What is working “well?”Evaluate the learned model on devwith that hyperparameter settingTraining Dataset hyperparametersLearn model parameters fromtraining setDevDataTestDataperform final evaluation on test,using the hyperparameters thatoptimized dev performance andretraining the model

Experimenting with Machine LearningModelsWhat is “correct?”What is working “well?”Evaluate the learned model on devwith that hyperparameter settingTraining Dataset hyperparametersLearn model parameters fromtraining setDevDataTestDataperform final evaluation on test,using the hyperparameters thatoptimized dev performance andretraining the modelRule 1: DO NOT ITERATE ON THE TEST DATA

On-board ExerciseProduce dev and test tables for a linearregression model with learned weights andset/fixed (non-learned) bias

OutlineExperimental Design: Rule 1Multi-class vs. Multi-label classificationEvaluationRegression MetricsClassification Metrics

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦Multi-label Classification

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦If 𝑦 {0,1} (or 𝑦 {True, False}), then abinary classification taskMulti-label Classification

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦If 𝑦 {0,1} (or 𝑦 {True, False}), then abinary classification taskIf 𝑦 {0,1, , 𝐾 1} (forfinite K), then a multi-classclassification taskQ: What are some examplesof multi-class classification?Multi-label Classification

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦If 𝑦 {0,1} (or 𝑦 {True, False}), then abinary classification taskQ: What are some examplesof multi-class classification?If 𝑦 {0,1, , 𝐾 1} (forfinite K), then a multi-classclassification taskA: Many possibilities. SeeA2, Q{1,2,4-7}Multi-label Classification

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦SingleoutputMultioutputIf 𝑦 {0,1} (or 𝑦 {True, False}), then abinary classification taskIf 𝑦 {0,1, , 𝐾 1} (forfinite K), then a multi-classclassification taskIf multiple 𝑦4 arepredicted, then a multilabel classification taskMulti-label Classification

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦SingleoutputMultioutputIf 𝑦 {0,1} (or 𝑦 {True, False}), then abinary classification taskIf 𝑦 {0,1, , 𝐾 1} (forfinite K), then a multi-classclassification taskIf multiple 𝑦4 arepredicted, then a multilabel classification taskGiven input 𝑥, predict multiple discrete labels 𝑦 (𝑦7 , , 𝑦8 )Multi-label Classification

Multi-class ClassificationGiven input 𝑥, predict discrete label 𝑦SingleoutputIf 𝑦 {0,1} (or 𝑦 {True, False}), then abinary classification taskIf 𝑦 {0,1, , 𝐾 1} (forfinite K), then a multi-classclassification taskMultioutputIf multiple 𝑦4 arepredicted, then a multilabel classification taskEach 𝑦4 could be binary ormulti-classGiven input 𝑥, predict multiple discrete labels 𝑦 (𝑦7 , , 𝑦8 )Multi-label Classification

Multi-Label Classification Will not be a primary focus of this courseMany of the single output classification methodsapply to multi-label classificationPredicting “in the wild” can be trickierEvaluation can be trickier

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)Loss function may (or may not)need to be extended & themodel structure may need tochange (big or small)

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)Loss function may (or may not)need to be extended & themodel structure may need tochange (big or small)Common change:instead of a single weight vector𝑤, keep a weight vector 𝑤 (;) foreach class cCompute class specific scores, e.g., (;)(;)𝑦 𝑤𝑥 𝑏(;)

Multi-class Option 1: LinearRegression/Perceptron𝑥𝐰𝑦𝑦 𝐰 𝑥 𝑏output:if y 0: class 1else: class 2

Multi-class Option 1: Linear Regression/Perceptron:A Per-Class View𝑥𝑥𝑦𝐰𝟏𝐰𝑦7 𝐰𝟏 𝑥 𝑏7𝑦7𝑦𝑦D𝑦D 𝐰𝟐 𝑥 𝑏D𝑦 𝐰 𝑥 𝑏output:if y 0: class 1else: class 2binary version isspecial case𝐰𝟐output:i argmax {y1, y2}class i

Multi-class Option 1: Linear Regression/Perceptron:A Per-Class View (alternative)𝑥𝑥𝑦𝐰𝟏𝑦7 𝒘𝟏 ; 𝒘𝟐 𝑻 [𝑥; 𝟎] 𝑏7𝐰𝑦7𝑦concatenation𝑦D𝑦 𝐰 𝑥𝑦D 𝒘𝟏 ; 𝒘𝟐 𝑻 [𝟎; 𝑥] 𝑏D 𝑏output:if y 0: class 1else: class 2𝐰𝟐output:i argmax {y1, y2}class iQ: (For discussion) Why doesthis work?

We’ve only developed binaryclassifiers so far Option 1: Develop a multi- With C classes:class versionTrain C different binary classifiers𝛾; (𝑥)Option 2: Build a one-vs𝛾; (𝑥) predicts 1 if x is likely class c,all (OvA) classifier0 otherwiseOption 3: Build an all-vsall (AvA) classifier(there can be others)

We’ve only developed binaryclassifiers so far Option 1: Develop a multi- With C classes:class versionTrain C different binary classifiers𝛾; (𝑥)Option 2: Build a one-vs𝛾; (𝑥) predicts 1 if x is likely class c,all (OvA) classifier0 otherwiseOption 3: Build an all-vsall (AvA) classifier(there can be others)To test/predict a new instance z:Get scores 𝑠 ; 𝛾; (𝑧)Output the max of these scores,𝑦N argmax; 𝑠 ;

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)With C classes:RDTraindifferent binaryclassifiers 𝛾;S,;T (𝑥)

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)With C classes:RDTraindifferent binaryclassifiers 𝛾;S,;T (𝑥)𝛾;S,;T (𝑥) predicts 1 if x is likelyclass 𝑐7 , 0 otherwise (likely class𝑐D )

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)With C classes:RDTraindifferent binaryclassifiers 𝛾;S,;T (𝑥)𝛾;S,;T (𝑥) predicts 1 if x is likelyclass 𝑐7 , 0 otherwise (likely class𝑐D )To test/predict a new instance z:Get scores or predictions 𝑠 ;S,;T 𝛾;S,;T 𝑧

We’ve only developed binaryclassifiers so far With C classes:Option 1: Develop a multiclass versionOption 2: Build a one-vs-all(OvA) classifierOption 3: Build an all-vs-all(AvA) classifier(there can be others)RDTraindifferent binaryclassifiers 𝛾;S,;T (𝑥)𝛾;S ,;T (𝑥) predicts 1 if x is likely class𝑐7 , 0 otherwise (likely class 𝑐D )To test/predict a new instance z:Get scores or predictions 𝑠 ;S ,;T 𝛾;S ,;T 𝑧Multiple options for final prediction:(1) count # times a class c waspredicted(2) margin-based approach

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vsall (OvA) classifierOption 3: Build an all-vsall (AvA) classifier(there can be others)Q: (to discuss)Why might you want touse option 1 or optionsOvA/AvA?What are the benefits ofOvA vs. AvA?

We’ve only developed binaryclassifiers so far Option 1: Develop a multiclass versionOption 2: Build a one-vs-all(OvA) classifierOption 3: Build an all-vs-all(AvA) classifier(there can be others)Q: (to discuss)Why might you want to useoption 1 or optionsOvA/AvA?What are the benefits ofOvA vs. AvA?What if you start with abalanced dataset, e.g.,100 instances per class?

OutlineExperimental Design: Rule 1Multi-class vs. Multi-label classificationEvaluationRegression MetricsClassification Metrics

Regression Metrics(Root) Mean Square Error\𝑅𝑀𝑆𝐸 1[ 𝑦 𝑦] 𝑁 D

Regression Metrics(Root) Mean Square ErrorMean Absolute Error\\𝑅𝑀𝑆𝐸 1[ 𝑦 𝑦] 𝑁 D1𝑀𝐴𝐸 [ 𝑦 𝑦] 𝑁

Regression Metrics(Root) Mean Square ErrorMean Absolute Error\\𝑅𝑀𝑆𝐸 1[ 𝑦 𝑦] 𝑁 Q: How can thesereward/punish predictionsdifferently?D1𝑀𝐴𝐸 [ 𝑦 𝑦] 𝑁

Regression Metrics(Root) Mean Square ErrorMean Absolute Error\\𝑅𝑀𝑆𝐸 1[ 𝑦 𝑦] 𝑁 Q: How can thesereward/punish predictionsdifferently?D1𝑀𝐴𝐸 [ 𝑦 𝑦] 𝑁 A: RMSE punishes outlierpredictions more harshly

OutlineExperimental Design: Rule 1Multi-class vs. Multi-label classificationEvaluationRegression MetricsClassification Metrics

Training Loss vs. Evaluation ScoreIn training, compute loss to update parametersSometimes loss is a computational compromise- surrogate lossThe loss you use might not be as informative asyou’d likeBinary classification: 90 of 100 trainingexamples are 1, 10 of 100 are -1

Some Classification MetricsAccuracyPrecisionRecallAUC (Area Under Curve)F1Confusion Matrix

Classification Evaluation:the 2-by-2 contingency tableActuallyCorrectSelected/GuessedNot selected/not guessedClasses/ChoicesActuallyIncorrect

Classification Evaluation:the 2-by-2 contingency tableSelected/GuessedNot selected/not guessedClasses/ChoicesActuallyCorrectTrue PositiveCorrect (TP) GuessedActuallyIncorrect

Classification Evaluation:the 2-by-2 contingency tableSelected/GuessedNot selected/not guessedClasses/ChoicesActuallyCorrectTrue PositiveCorrect (TP) GuessedActuallyIncorrectFalse PositiveCorrect (FP) Guessed

Classification Evaluation:the 2-by-2 contingency tableActuallyActuallyCorrectIncorrectSelected/True Positive False PositiveGuessedCorrect (TP) GuessedCorrect (FP) GuessedNot selected/ False Negativenot guessed Correct (FN) GuessedClasses/Choices

Classification Evaluation:the 2-by-2 contingency tableActuallyActuallyCorrectIncorrectSelected/True Positive False PositiveGuessedCorrect (TP) GuessedCorrect (FP) GuessedNot selected/ False Negative True Negativenot guessed Correct (FN) Guessed Correct (TN) GuessedClasses/Choices

Classification Evaluation:Accuracy, Precision, and RecallAccuracy: % of items correctTP TNTP FP FN TNActually CorrectActually IncorrectSelected/GuessedTrue Positive (TP)False Positive (FP)Not select/not guessedFalse Negative (FN)True Negative (TN)