Multivariate Data Analysis - I2PC

Multivariate Data AnalysisSession 0: Course outlineCarlos Óscar Sánchez Sorzano, Ph.D.Madrid

Motivation for this course2

Motivation for this course3

Course outline4

Course outline: Session 11. Introduction1.1. Types of variables1.2. Types of analysis and technique selection1.3. Descriptors (mean, covariance matrix)1.4. Variability and distance1.5. Linear dependence2. Data Examination2.1. Graphical examination2.2. Missing Data2.3. Outliers2.4. Assumptions of multivariate analysis5

Course outline: Session 23. Principal component analysis (PCA)3.1. Introduction3.2. Component computation3.3. Example3.4. Properties3.5. Extensions3.6. Relationship to SVD4. Factor Analysis (FA)4.1. Introduction4.2. Factor computation4.3. Example4.4. Extensions4.5. Rules of thumb4.6. Comparison with PCA6

Course outline: Session 35. Multidimensional Scaling (MDS)5.1. Introduction5.2. Metric scaling5.3. Example5.4. Nonmetric scaling5.5. Extensions6. Correspondence analysis6.1. Introduction6.2. Projection search6.3. Example6.4. Extensions7. Tensor analysis7.1 Introduction7.2 Parafac/Candecomp7.3 Example7.4 Extensions7

Course outline: Session 48. Multivariate Analysis of Variance (MANOVA)8.1. Introduction8.2. Computations (1-way)8.3. Computations (2-way)8.4. Post-hoc tests8.5. Example9. Canonical Correlation Analysis (CCA)9.1. Introduction9.2. Construction of the canonical variables9.3. Example9.4. Extensions10. Latent Class Analysis (LCA)10.1. Introduction8

Course ePractice9

Suggested readings: OverviewsIt is suggested to read (before coming): H. Abdi. Multivariate Analysis. In: Lewis-Beck M., Bryman, A., FutingT. (Eds.) (2003). Encyclopedia of Social Sciences Research Methods.Thousand Oaks (CA): Sage. S. Sanogo and X.B. Yang. Overview of Selected Multivariate StatisticalMethods and Their Use in Phytopathological Research. Phytopathology,94: 1004-1006 (2004)10

ResourcesData dd.ics.uci.eduLinks to organizations, events, software, tro.u-strasbg.fr/ fmurtagh/mda-swhttp://lib.stat.cmu.eduLecture /courses/ed231a1/lect.html11

Bibliography D. Peña. Análisis de datos multivariantes, Mc GrawHill, 2002 B. Manly. Multivariate statistical methods: a primer. Chapman & Hall/CRC,2004. J. Hair, W. Black, B. Babin, R. Anderson. Multivariate Data Analysis (6th ed),Prentice Hall, 2005. B. Everitt, G. Dunn. Applied multivariate data analysis. Hodder Arnold, 2001 N. H. Timm. Applied multivariate analysis. Springer, 2004 L. S. Meyers, G. C. Gamst, A. Guarino. Applied multivariate research: designand interpretation. Sage, 2005 J. L. Schafer. Analysis of incomplete multivariate data. Chapman & Hall/CRC,1997 M. Bilodeau, D. Brenner. Theory of multivariate statistics. Springer, 200612

Multivariate Data AnalysisSession 1: Introduction and data examinationCarlos Óscar Sánchez Sorzano, Ph.D.Madrid

Course outline: Session 11. Introduction1.1. Types of variables1.2. Types of analysis and technique selection1.3. Descriptors (mean, covariance matrix)1.4. Variability and distance1.5. Linear dependence2. Data Examination2.1. Graphical examination2.2. Missing Data2.3. Outliers2.4. Assumptions of multivariate analysis2

1. Introduction3

1.1 Introduction: Types of variablesDataDiscrete SexÎ{Male,Female}No. SonsÎ{0,1,2,3, }Continuous TemperatureÎ[0, /CategoricalScale0 Male1 FemaleOrdinalScale0 Love1 Like2 Neither like nor dislike3 Dislike4 HateIntervalScaleYears: 20062007 RatioScaleTemperature:0ºK1ºK 4

1.1 Introduction: Types of variablesCoding of categorical variablesHair Colour{Brown, Blond, Black, Red}No orderPeter: BlackMolly: BlondCharles: BrownCompany size{Small, Medium, Big}Company A: BigCompany B: SmallCompany C: Medium xBrown , xBlond , xBlack , xRed 0,1 4Peter:Molly:Charles:Implicit order 0, 0,1, 0 0,1, 0, 0 1, 0, 0, 0 xsize 0,1, 2 Company A: 2Company B: 0Company C: 15

1.2 Introduction: Types of analysisAnalysisY f (X )f ( X ,Y ) 0DependenceInterdependenceA variable or set of variables is identified as the dependent variable to bepredicted or explained by other variables known as independent variables.Example:(No. Sons, House Type) f(Income, Social Status, Studies) Multiple Discriminant Analysis Logit/Logistic Regression Multivariate Analysis of Variance (MANOVA) and Covariance Conjoint Analysis Canonical Correlation Multiple Regression Structural Equations Modeling (SEM)6

1.2 Introduction: Types of analysisAnalysisY f (X )Dependencef ( X ,Y ) 0InterdependenceExample: Who is similar to whom?(No. Sons, House Type, Income, Social Status, Studies, )Involves the simultaneous analysis of all variables in the set, withoutdistinction between dependent variables and independent variables. Principal Components and Common Factor Analysis Cluster Analysis Multidimensional Scaling (perceptual mapping) Correspondence Analysis Canonical Correlation7

1.2 Introduction: Technique selection Multiple regression: a single metric variable is predicted byseveral metric variables.Example:No. Sons f(Income, No. Years working) Structural Equation Modelling: several metric variables arepredicted by several metric (known and latent) variablesExample:(No. Sons, House m2) f(Income, No. Years working, (No. Years Married))8

1.2 Introduction: Technique selection Multiple Analysis of Variance (MANOVA): Several metricvariables are predicted by several categorical variables.Example:(Ability in Math, Ability in Physics) f(Math textbook, Physics textbook, College) Discriminant analysis, Logistic regression: a single categorical(usually two-valued) variable is predicted by several metricindependent variablesExample:Purchaser (or non purchaser) f(Income,No. Years working)9

1.2 Introduction: Technique selection Canonical correlation: Several metric variables are predicted byseveral metric variablesExample:(Grade Chemistry, Grade Physics) f(Grade Math, Grade Latin) Conjoint Analysis: An ordinal variable (utility function) ispredicted by several categorical/ordinal/metric variablesExample:TV utility f(Screen format, Screen size, Brand, Price) Classification Analysis: Predict categorical variable from severalmetric variables.Example:HouseType f(Income,Studies)10

1.2 Introduction: Technique selection Factor analysis/Principal Component Analysis: explain thevariability of a set of observed metric variables as a function ofunobserved variables (factors)Example:(Grade Math, Grade Latin, Grade Physics) f(Intelligence, Maturity) Correspondence analysis: similar to factor analysis but withcategorical data.Example:(Eye colour, Hair colour, Skin colour) f(gene A, gene B) Cluster analysis: try to group individuals according to similarcharacteristicsExample:(Grade Math, Grade Latin, Grade Physics, Grade Philosophy, Grade History)11

1.2 Introduction: Technique selection Multidimensional scaling: Find representative factors so that therelative dissimilarities in the original space are as conserved aspossibleExample:(x,y) f(City gross income, health indexes,population, political stability, )12

(Basic vector and matrix algebra)xt13

(Basic vector and matrix algebra)VectorNormxx x, xNInternal productDot productx, y y , x x y xt y xi yi x y cos i 1yx Projx y Orthogonalityx, yxux x, yx2x x(xt x) 1 xt yx y x, y 014

(Basic vector and matrix algebra)fieldLinear spanspan x1 , x 2 ,., x r x 1x1 2 x 2 . r x r 1 , 2 ,., r K Linearly dependent, i.e., 0 x 1x1 2 x 2 . r x r 0uzspan u y , u z uyuxspan u x span u y , u z span u x C span u y , u z Complementary spaces15

(Basic vector and matrix algebra)uzy Assuming that u x ,u yuyxuxE span u x , u y is a basis of the spanned spacex ProjE y Proju x y Proju y y X ( X t X ) 1 X t yBasis vectors of E as columns y x E y x C E y x y x22216

1.3 Descriptors: Data representation x1t x11 t x 2 x21 X . . t x n xn1x12x22.xn 2. x1 p . x2 p . . . xnp FeaturesIndividualsXx t217

1.3 Descriptors: Univariate analysis x1t x11 t x 2 x21 X . . t x n xn1Sample meanSample standarddeviationSample variationcoefficientx12x22.xn 2. x1 p . x2 p . . . xnp 1 nx2 xi 2n i 11 n2s2 x x i2 2 n i 1VC2 x22s22m2If outliersRobust statisticsSample medianPr X 2 m2 1 Pr X 2 m2 2MAD2 Sample MedianAbsolute DeviationMedian x2 m2 18

1.3 Descriptors: Mean and covariance x 1t x11 t x 21xX 2 . . t x n x n1x12.x 22.xn 2.x1 p x2 p . x np x x1 x2 . x p 1 tx X 1 Sample meannt x 1 x t x 2 x tX . t xx nX X 1 x t x xx12 x 2 . x1 p x p1 11 x 21 x1 x 22 x 2 . x 2 p x p . . x n1 x1 x n 2 x 2 . x np x p Matrix of centered dataVector of 1s1 nSample covariance s jk xij x j xik xk n i 1Symmetric,positivesemidefinite s12 sS 21 . s p1 s12s 22.sp2.s1 ps2 p.s 2pMeasures how variablesj and k are related n1 t t 1X Xxxxx ii n ni 1 E x x x x t 19

1.3 Descriptors: CovarianceX x1x2X 1 N (0,1)X 2 N (0,1)X 3 N (0,1)x 3 3 variables 13 E ( X 1 1 )( X 3 3 ) 200 samples 0.9641 S 0.0678 0.0509 E X 1 X 30.06780.85520.0398 0.0509 0.0398 0.9316 Sample covarianceX 1 N (0,1)X 2 X1X 3 X1X 1 N (0, 9)X 2 X1X 3 X1 0.9641 S 0.9641 0.9641 10.4146 S 10.4146 10.4146 1 0 0 0 0 1 010Covariance0.96410.9641 0.9641 0.9641 0.9641 0.9641 10.414610.4146 10.4146 10.4146 10.4146 10.4146 1 1 1 9 9 9 11 199 9 1 1 1 9 9 9 20

1.3 Descriptors: CovarianceX 1 N (1, 2)X 2 N (2, 3)X 3 X1 X 2 1.6338 S 0.0970 1.5368 0.09702.8298 2.7329 1 2 X1 X X 2 N 2 , 0 X 1 2 3 X 1 N ( 1 , 12 )X 2 N ( 2 , 22 )X 3 a1 X 1 a 2 X 21.5368 2.7329 4.2696 03 3X N μ , 0 22a 2 221 2 03 32 3 5 2 3 5 2 1 X1 1 X X2 N 2 , 0 X a a a 22 2 1 1 3 1 1fX (x) 2 0 2 N2 a1 12 a 2 22 a12 12 a 22 22 1 exp ( x μ ) t 1 ( x μ ) 2 21

1.3 Descriptors: CovarianceX N μ , fX (x) 1 2 N2 1 exp ( x μ ) t 1 ( x μ ) 2 4 0 μ 0 1 00.220 1 X10.1004-220-2-4 -4-2024X2-4-4-2024X 1 and X 2 are independentFor multivariate Gaussians, covariance 0 implies independency22

1.3 Descriptors: CovarianceX N μ , 1 2 N2 1 exp ( x μ ) t 1 ( x μ ) 2 4 1 μ 0 1 0f X (x) 0.220 3 0.1004-220-2-4 -4-2024-4-4-2024X 1 and X 2 are independent23

1.3 Descriptors: CovarianceX N μ , f X (x) 1 2 N2 1 exp ( x μ ) t 1 ( x μ ) 2 40.22μ 0 1 R 0X10.10 t R3 004-22 cos 60ºR sin 60º 2.5 0.8660.866 1.5 sin 60º cos 60º 0-2-4 -4-2024X2-4-4-2024X 1 and X 2 are NOT independentBUT there exist two independent variables24

1.3 Descriptors: CovariancePitfalls of the covariance matrix43X 1 N (0,1) XX2 1 X 12p 0.51 p 0.510-1-2 1 00 1 X 1 N (0,1)X 2 X 12 1 00 2 -3-4-4-3-2-101234C ov ( X 1 , X 2 ) 0 U ncorrelated IndependentC ov ( X 1 , X 2 ) 0 G aussian Independent25

1.3 Descriptors: CovarianceRedundant variablesX 3 N (0,1) 1 0 0 X 1 N (0,1)X 2 X1X 3 X1 1 1 1 1 1 1 R 00 t R3 X 1 N (0,1)X 2 N (0,1)X 1 N (1, 2)X 2 N (2, 3)X 3 X1 X 2 2 0 2 0 0 1 010103 3eig ( ) (1,1,1) 1 1 1 2 3 5 eig ( ) (1, 0, 0)eig ( ) (3,1)eig ( ) (7.64, 2.35, 0)26

1.4 Variability and distancex t2tx10How far are they?How far are they from the mean?d (xi , x j )n1-norm (Manhattan)d (xi , x j ) xis x jss 1nMost usedp-norm (Euclidean p 2)MinkowskiInfinity normp d (xi , x j ) xis x js s 1 d (xi , x j ) max xis x js1ps27

1.4 Variability and distanceHeight (m) Weight (kg)JuanJohnJeand (xi , x j )Juan1.801.701.65JuanHeight (cm) Weight (kg)807281JohnJuanJohnJeand (xi , x j )Jean----- 8.0004 1.0112JuanMahalanobis distance807281JuanJohnJean----- 11.3137 15.0333 M x x d (x , x ) x x I x x x x x x d (x , x ) x x x x d (x , x ) k x x tMatrix-based distance d 2 ( xi , x j ) xi x jEuclidean distance180170165 1it2ijitijt 1j2ijijijijij 1jNCorrentropy distanceijk 1 ikjk28

1.4 Variability and distance Mahalanobis distance d 2 ( xi , x j ) xi x j2 height r height weight x x t 1ijr height weight 100 70 2 weight 70 100 height 10cm weight 10kg4r 0.700-2-2024JuanJohnJean----- 0.7529 4.843142-2JuanIndependently of units!!2-4-4d (xi , x j )-4-4-202429