Day1 Basic Statistics Using R

Checking the objects and memory¾ To see what objects are in memory: ls( )¾ Length of a vector or factor length(v)¾ Dimentions of a data frame or matrix: dim(d)¾ Column and row names of a data frame or matrix col.names(d) row.names(d)

Exercise IV

Import tabular data¾ Download the file from the Internet: t¾ Put the file on desktop.¾ See how the data looks like (use Excel and Wordpad): Are there columns headers? What is the separator between the columns (space, tab, etc)? Are there row names in the data?¾ Now you should know what arguments to specify in theread.table() command, so use it for reading in the data.

Import the rest of the data¾ I have prepared several datasets for this course.¾ These can be downloaded from the web: urse data.txt")¾ The datasets are written as R commands, so the command abovedownloads and runs this command file.¾ Check what object were created in R memory?¾ Run the command showMetaData(). This should show some information about the datasets. Note that the command is written for this course only (by me), and can’t beused in R in general.

Object type conversions

Converting from a data type to another¾ Certain data types can easily be converted to other data types. Vector - factorData frame - matrixData frame - vector / factorMatrix - vector / factor¾ Typical need for converting a vector to a factor is whenperforming some statistical tests.¾ Data frame might need to be converted into a matrix (or viceversa) when running some statistical tests or when plotting thedata.¾ Several vectors can be cleaved from a data frame or a matrix.¾ Several vectors can be combined to a data frame or a matrix.

Converting from a vector to a factor¾ To convert a vector to factor, do v2 -as.factor(v, labels c(”Jan”, ”Feb”)) Unordered factor v2 -factor(v, ordered T, labels c(”Jan”, ”Feb”)) Ordered factor¾ Difference between ordered and unorder factors lies in the detailthat if the factor is unordered, the values are automaticallyordered in plots and statistical test according to lexical scoping(alphabetically).¾ If the factor is ordered, then the levels have an explicit meaning inthe specified order, for example, January becomes beforeFebruary.

Extracting a vector from a data frame I/III¾ As individual variables are storedin the columns of a data frame, itis typically of interest to be ableto extract these column from adata frame.¾ Columns can be addressed usingtheir names or their position(calculated from left to right)¾ Rows can be accessed similarlyto columns.¾ Remember how to check thenames? 7Vidal48Max511

Extracting a vector from a data frame II/III¾ This data frame is stored in anobject called dat.¾ The first column is named Jan, sowe can get the values in it bynotation: dat JanName of the data frame Nameof the columnThere are no brackets, so there is”no” command: we are accessing adata frame.JanFebJarno131Dario212Panu337Vidal48Max511

Extracting a vector from a data frame III/III¾¾This data frame is stored in an objectcalled dat.To get the first column, one can alsopoint to it with the notation: x511# 1, 31And the value on the first row of thefirst column: ¾dat[1,]Feb# 1, 2, 3, 4, 5This is called a subscript.Subscript consists of square brackets.Inside the bracket there are at leastone number.The number before a comma pointsrows, the number after the comma tocolumnsThe first row would be extracted by: ¾dat[,1]Jan#1Again, no brackets - no commands,so we are accessing an object

Extracting several columns of rows¾ One can want to extract severalcolumns or rows from a table.¾ This can be accomplished usinga vector instead of a singlenumber.¾ For example, to get the rows 1and 3 from the previous table: FebJarno131Panu337dat[c(1,3),]¾ Or create the vector first, andextract after that: Janv -c(1,3)dat[v,]¾ These should give you:

Deleting a column or a row¾ One can delete a row or a column(or several of them using a vecterin the place of number) from adata frame by using a negativesubscript: ax511JanFebDario212Vidal48Max511

Selecting a subset by some variable¾ How to get those rowsfor whochthe value for February is below20?¾ Function which gives on index ofthe rows: which(dat Feb 20)[1] 2 4 5¾ To get the rows, use then indexas a subscript: i -which(dat Feb 11

Writing data to disk

Using sink¾ Sink prints everything you would normally see on the screen to afile.¾ Usage: sink(”output.txt”) print(”Just testing!”) sink()# Opens a file# Commands# Closes the file

Using write.table¾ Writing a data frame or a matrix to disk is rather straight-forward. Command write.table()¾ Usage: write.table(dat, ”dat.txt”, sep ”\t”, quote F,row.names T, col.names T) dat”dat.txt”sep ”\t”quote Frow.names Tcol.names Tname of the table in Rname of the file on diskuse tabs to separate columnsdon’t quote anything, not even textwrite out row names (or F if there are no row names)write out column names

Quitting R

Quitting R¾ Command q()¾ Asks whether to save workspace image or not. Answering yes would save all objects on disk in a file .RData. Simultaneously all the commands given in this session are saved in a file.RHistory.¾ These workspace files can be later-on loaded back into memoryfrom the File-menu (Load workspace and Load history).

Exercise V

Extracting columns and rows I/II¾ What is the size of the Students dataset (number or rows andcolumns)?¾ What are column names for the Students dataset?¾ Extract the column containing data for population. How manystudents are from Tampere?¾ Extract the tenth row of the dataset. What is the shoesize of thisperson?¾ Extract the rows 25-29. What is the gender of these persons?¾ Extract from the data only those females who from Helsinki. Howmany observations (rows) are you left?¾ How many males are from Kuopio and Tampere?

Extracting columns and rows II/II¾ Examine Hygrometer dataset. Notice that the measuments weretaken on two different dates (day1 and day2 – each hygrometerwas read before and after a few rainy days).¾ Modify the dataset so that the order of the measurements isretained, but the measurements for the day1 and day2 are in twoseparate column in the same data frame.¾ We will later on use this data frame for running certain statisticaltests (e.g., paired t-test) that require the data in this format.

Recoding variables

Making new variables I/¾ There are several ways to recode variables in R.¾ One way to recode values is to use command ifelse(). ifelse(Students shoesize 40, ”small”, ”large”)1. Comparison: is shoesize smaller than 402. If comparison is true, return ”small”3. If comparison is false, return ”large” You can combine several comparisons with logical operators ifelse((Students shoesize 37 &Students gender "female"), "small", "large")

Making new variables II/¾ If the coding needs to be done in several steps (e.g. we want toassign shoesizes to four classes), a better approach could be thefollowing. s -Students shoesizes[Students shoesize 37] -"minuscule"s[Students shoesize 37 & Students shoesize 39] -"small"s[Students shoesize 39 & Students shoesize 43] -"medium"s[Students shoesize 43] -"large”¾ At each step we select the only the observations that fulfill thecomparsion. At the first step, all students who have a shoesize less than or equal to 37 arecoded as minuscule. At the second step, all students having shoesize larger than 37 but smaller thanor equal to 39 are coded as small. And so forth.

Exercise VI

Making new variables¾ Make a new vector of the shoesize measurements (extract thatcolumn from the data).¾ Code the shoesize as it was done on the previous slides (in therange minuscule large).¾ Turn this character vector into a factor. Make the factor ordered sothat the order of the factor levels is according to the size(minuscule, small, medium, large).¾ Add this new factor to the Students dataset (make a new dataframe).

Day 2

Topics¾ Data exploration¾ Graphics in R¾ Wrap-up of the first half of the course

Exploration

Exploration – first step of analysis¾ Usually the first step of a data analysis is graphical dataexploration¾ The most important aim is to get an overview of thedataset Where is data centered?How is the data spread (symmetric, skewed )?Any outliers?Are the variables normally distributed?How are the relationships between variables: Between dependent and independents Between independents¾ Graphical exploration complements descriptivestatistics

Variable types¾ Continuous (vectors in R) HeightAgeDegrees in centigrade¾ Categorical (factors in R) Make of a carGenderColor of the eyes

Exploration – methods I/II¾ Single continuous variable Plots: boxplot, histogram (density plot, stem-and-leaf), normalprobability plot, stripchartDescriptives: mean, median, standard deviation, fivenumsummary¾ Single categorical variable Plots: contingency table, stripchart, barplotDescriptives: mode, contingency table¾ Two continuous variables Plots: scatterplotDescriptives: individually, same as for a single variable¾ Two categorical variables Plots: contingency table, mosaic plotDescriptives: individually, same as for a single variable

Exploration – methods II/II¾ One continous, one categorical variable Plot: boxplot, histogram, but for each category separatelyDescriptives: mean, median, sd , for each categoryseparately¾ Several continous and / or categorical variables Plots: pairwise scatterplot, mosaic plotDescriptives: as for continuous or categorical variables

Descriptive statistics

Mean¾ Mean sum of all values / the number of values

Standard deviation and variance¾ SD each observation’s squared difference from themean divided by the number of observation minus one. Has the same unit as the original variable¾ Var SD*SD SD 2

Normal distribution I/III¾ Some measurementsare normallydistributed in the realworld HeigthWeight¾ Means of observationstaken from otherwisedistributed data arealso normallydistributed¾ Hence, manydesciptives, andstatistical tests havebeen deviced on theassumption ofnormality

Normal distribution II/III¾ Normal distribution are described by two statistics: MeanStandard deviation¾ These two are enough to tell: Where is the peak (center) of the distribution locatedHow the data are spread around this peak

Normal distribution III/III

Quartiles¾ 1st quartile(25%), Median (50%), and 3rd quartile (75%)¾ 1 2 3 4 5 6 7 8 975% QuMedian75% QuInterquartile range (IQR)¾ Fivenum summary: Minimum (1), 1st Quartile (3), Medium (5), 3rd Quartile (7),maximum (9)

What if distribution is skewed or there areoutliers/deviant observation?¾ Use nonparametric alternatives to descriptives Median instead of meanInter-quartile range instead of standard deviation

Summary of a continuous variable I/II¾ summary( ) x -rnorm(100)summary(x)Min. 1st Qu. Median Mean 3rd Qu. Max.0.005561 0.079430 0.202900 0.310300 0.401000 tile(x, probs c(0.25, 0.75)) 1st and 3rd quartiles

Summary of a continuous variable II/II¾¾¾¾IQR(x)mad(x)sd(x)var(x) # inter-quartile range# robust alternative to IQR# standard deviation# variancesd(x) 2¾ table( )# Makes a table (categ. var.)

Outliers and missing values

What are these outliers then?¾ Outliers Technical errors The measurement is too high, because the machineryfailedCoding errors Male 0, Female 1 Data has some values coded with 2¾ Deviant observations Measurements that are somehow largely different from others,but can’t be treated as outliersIf the observation is not definitely an outlier, better treat it as adeviant observation, and keep it in the data

Outliersgender0 1 211 8 1¾ What are those with gender coded as 2?¾ Probably a typing error What if they are missign values (gender is unknown)?¾ If a typing error, should be checked from the originaldata¾ If a missing value, should be coded as missing value We will come to this shortly

Deviant observations

Missing values¾ Missing values are observation that really are missing avalue Some samples were not measured during the experimentSome students did not answer to certain questions on thefeedback from¾ If the sample was measured, but the results was verylow or not detectable, it should be coded with a smallvalue (half the detection limit, or zero, or something)¾ So, no measurement and measurement, but a smallresult, should be coded separately

Missing values in R I/II¾ In R missing values are coded with NA NA not available¾ Although it is worth treating missing measurements asmissing values, they tend to interfera with the analysis Many graphical, descriptive, and testing procedure fail, if thereare missing values in the data¾ An example x -c(NA, rnorm(10)) mean(x) [1] NA

Missing values in R II/II¾ The most simple way to treat missing values is to deleteall cases (rows) that contain at least one missing value.¾ For vector this means just removing the missingvalues: x2 -na.omit(x) mean(x2) [1] -0.1692371¾ There are other ways to treat missing values, such asimputation, where the missing values are recoded with,e.g., the mean of the continuous variable, or with themost common observation, if the variable iscategorical. x2[is.na(x2)] -mean(na.omit(x))

Graphical methods

Continuous variables

-2024Boxplot

Link between quartiles and boxplot

Histogram I/IIdensity.default(x 0.320000.4Histogram of rnorm(10000)-4-20rnorm(10000)24-4-202N 10000 Bandwidth 0.14324

Histogram II/IIHistogram of Histogram of x995699589960x99629964995699589960x9962

Link between histogram and boxplot

Stem-and-leaf plot¾ The decimal point is at the ¾¾¾¾¾¾-2 90-1 88876664322221000-0 9988866655555444443333222222111100 0011111111122223344456677788888991 001123344555692 3

Scatterplot

QQ-plot¾ QQ-plot is a plot that can be used for graphically testingwhether a variable is normally distributed. Normal distribution is an assumption made by many statisticalprocedures.

Pairwise scatterplot

Categorical variables

Stripchart

Barchart

Mosaicplot

Contingency tableJanuary February March dnesday5444

Exercise VII

Checking distributions¾ Are these data normally distributed?

Checking distributions¾ Are these data normally distributed?

Checking distributions¾ Are these data normally distributed?

UCB admissions¾ Claim: UCB discriminates against females. I.e., More females than males are rejected, and don’t getadmitted to the university.Does UCB discriminate?

¾ Claim: UCB discriminates against females. Does it?Department ARejectedAdmittedDepartment exMaleSexMaleAdmittedDepartment BAdmitAdmitAdmitDepartment DDepartment EDepartment mit

Graphics in R

Basic idea¾ All graphs in R are displayedon a graphical device.¾ If no device is open when theplotting command is called, anew one is opened, and theimage is displayed in it.¾ Graphics device is simply anew window that displayes thegraphic.¾ Graphic device can also be afile where the plot is written. Open itMake the plotClose it

Traditional graphics commands is R¾ High level graphical commands create the plot plot( )hist( )stem( )boxplot( )qqnorm( )mosaicplot( )# Scatter plot, and general plotting# Histogram# Stem-and-leaf# Boxplot# Normal probability plot# Mosaic plot¾ Low level graphical commands add to the plot points( )lines( )text( )abline( )legend( )# Add points# Add lines# Add text# Add lines# Add legend¾ Most command accept also additional graphicalparameters par( )# Set parameters for plotting

Graphical parameters in R¾ par( ) cexcolltylwdmarmfrowomapchxlimylim# font size# color of plotting symbols# line type# line width# inner margins# splits plotting area (mult. figs. per page)# outer margins# plotting symbol# min and max of X axis range# min and max of Y axis range

A few worked examples

Drawing a scatterplot in R I/V¾ Let’s generate some data x -rnorm(100)y -rpois(100, 10)g -c(rep(”horse”, 50), rep(”hound”,50))¾ Simple scatter plot plot(x, y)

Adding a title and axis labels II/V¾ plot(x, y, main ”Horses and hounds”,xlab ”Performance”, ylab ”Races”)

Drawing a scatterplot in R III/V¾ Coloring spots according to the group (horse or hound)they belong to cols -ifelse(g ”horse”, ”Black”, ”Red”)plot(x, y, main ”Horses and hounds”, xlab ”Performance”,ylab ”Races”, col cols)

Drawing a scatterplot in R IV/V¾ Changing the plotting symbol plot(x, y, main ”Horses and hounds”, xlab ”Performance”,ylab ”Races”, col cols, pch 20)plot(x, y, main ”Horses and hounds”, xlab ”Performance”,ylab ”Races”, col cols, pch ” ”)

Drawing a scatterplot in R V/V¾ Saving the image Menu: File - Save As - JPEG / BMP / PDF / postscript¾ Directing the plotting to a file pdf(”hnh.pdf”)plot(x, y, main ”Horses and hounds”, xlab ”Performance”,ylab ”Races”, col cols, pch 20)dev.off()¾ Setting

it will be applied seperately for every observation in that vector: log2(v) [1] 0.000000 0.000000 0.000000 1.000000 1.000000 1.000000 [7] 1.000000 1.584963 1.584963 1.584963 1.584963 1.584963 ¾When applied to a vector, the lenght of the result is as long as the starting vector. ¾When a function