Visualizing And Exploring Data

Visualizing and Exploring Data

Visual Methods for findingstructures in data Power of human eye/brain to detect structures– Product of eons of evolution Display data in ways that capitalize on humanpattern processing abilities Can find unexpected relationships– Limitation: very large data sets

Exploratory Data Analysis Explore the data without any clear ideas of whatwe are looking for EDA techniques are– Interactive– Visual Many graphical methods for low-dimensional data For higher dimensions -- Principal ComponentsAnalysis

Topics in Visualization1. Summarizing DataMean, Variance, Standard Deviation, Skewness2.3.4.5.Tools for Single Variables (histogram)Tools for Pairs of Variables (scatterplot)Tools for Multiple VariablesPrincipal Components Analysis– Reduced number of dimensions

1. Summarizing the data1 nMean, µ x(i )n i 1 Centrality– Minimizes the sum of squared errors to all samples– If there are n data values, mean is the value such that the sum of ncopies of the mean equals the sum of data values Measures of Location– Mean is a measure of location– Median (value that has equal no of points above andbelow)– Quartile (value greater than a quarter of the data points)

Measures of Dispersion, orVariabilityn12Variance, σ 2 [x(i) µ] n 1 i 1Average squared errorin mean representing data1 n2[ x(i ) µ ]Standard Deviation, σ n 1 i 12Skewness 3ˆ(x(i) µ) ( ( x(i ) µˆ ) 2 ) 3 / 2Measures how much the datais one-sided (single long tail)

2. Tools for Displaying SingleVariables Basic display for univariate data is thehistogram– No of values of the variable that lie inconsecutive intervals

Manydid not use itat allHistogram of supermarket credit card usageThese used itevery weekexcept holidaysweeks

Histogram of Diastolic blood pressure of individuals(UCI ML archive)Zero BPmeansdata missing

Smoothing estimates Kernel Function K Estimated density at point x isn1x x(i )ˆf ( x) )K( n i 1h GaussianKernel with std dev hK (t , h) Ce1 t ( )22 hwhere t x x(i )

Kernel Estimateswith different values of h:Small values lead to spikyestimatesData is right skewedwith hint of multimodalityK (t , h) Ce1 t ( )22 hHigher smoothing

3. Tools for DisplayingRelationship between two variables Box PlotsScatter PlotsContour PlotsTime as one of the two variables

Box Plot1.5 times inter-quartile rangeUpperQuartileMedianLowerQuartile

MultipleVariablesHealthyDiabetic

ScatterplotCredit card repayment dataHighly correlated dataSignificant number departfrom pattern: worth investigating

Scatterplot Disadvantages1. With large no of data points reveals little structure2. Can conceal overprinting which can be significant for multimodaldata

Contourplot1. Overcomes some scatterplot problemsUnimodalitycan be seen:Not apparentin scatterplot2. Requires a 2-D density estimate to be constructedwith a 2-D kernel

Display when one of the variables is timeAnnualFees introducedJan 1963Dec 1970Peaks in early andlate summer and around new year

Tools for Displaying More thanTwo Variables Scatter plots for all pairs of variables Trellis Plot Parallel Coordinates Plot

More than two variables Sheets of Paper and Computer screens are fine fortwo variables Need projections from higher-dimensional data to2-D plane Methods– Examine all pairs of variables Scatterplot matrix Trellis plot Icons

CPU performanceScatter Plot MatrixIndependent209 CPU data:Cycle TimeMinimum MemoryMaximum MemoryCache Size (Kb)Minimum ChannelsMaximum ChannelsRelative PerformanceEstimated rel perf (wrt IBM)Correlated

Disadvantage of Scatter PlotMatrices Scatter Plot Matrices are multiple bivariatesolutions2-dprojection Not a multivariate solution Such projections sacrificeinformation3 variables8 cubes: alternately empty and fullEach 1-D and 2-D projection isuniformly distributed!

Trellis Plot Rather than displaying scatter plot for eachpair of variables Fix a particular pair of variables andproduce a series of scatter plots, histograms,time series plots, contour plots etc

Trellis PlotMaleFemale(with scatterplots)OlderEpilepticSeizures inlater 2 weekperiodYoungerBest fit lineEpilepticSeizures in 2 weekperiod

Icon PlotStar Plot: Each direction correspondsto a variable. Length correspondsto a value53 samples of minerals12 chemical properties

ParallelCoordinatesPlotEach path representsan individualEach countRepresents 2-weekperiod

produce a series of scatter plots, histograms, time series plots, contour plots etc. Male Female Younger Older Epileptic Seizures in 2 week period Epileptic Seizures in later 2 week period Best fit line Trellis Plot (with scatter