Applied Functional Data Analysis What Is Functional Data?

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisApplied Functional Data AnalysisWhat is Functional Data?What are the most obvious features of these data?Venue: Tuesday/Thursday11:40 - 12:55WN 360Lecturer: Giles HookerOffice Hours: Wednesday 2 - 4Comstock 1186Ph: 5-1638e-mail: gjh27http://www.bscb.cornell.edu/ hooker/FDA2008/See also BlackboardBTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?What are the most obvious features of these data?quantityWhat are the most obvious features of these data?quantityfrequency (resolution)

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?What are the most obvious features of these data?Most important: smoothnessquantityfrequency (resolution)similar trendsBTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?Most important: smoothnessMost important: smoothnessThese data describe (nearly) aprocess that changes smoothing,and continuously over time.These data describe (nearly) aprocess that changes smoothing,and continuously over time.Functional Data Analysis Analysis of data that arefunctions.

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?20 replicationsMost important: smoothnessThese data describe (nearly) aprocess that changes smoothing,and continuously over time.Functional Data Analysis Analysis of data that arefunctions.Domain is usually time, but canbe anything: space, energy .Functional data analysis involves repeated measures of the sameprocess.BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?20 replications, 1401 observations within replications, 2 dimensions20 replications, 1401 observations within replicationsFunctional data is oftencomplicated:not easily described bymathematical formulaevariation betweenreplications even harder todescribe

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?20 replications, 1401 observations within replications, 2 dimensions20 replications, 1401 observations within replications, 2 dimensionsFunctional data is oftencomplex:Functional data is oftencomplex:often a large number ofrelated quantitiesBTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?What is Functional Data?20 replications, 1401 observations within replications, 2 dimensionsFunctional data is oftencomplex:20 replications, 1401 observations within replications, 2 dimensionsFunctional data is oftencomplex:often a large number ofrelated quantitiesoften a large number ofrelated quantitiesviewing each replication asa single observation canmake the data easier tothink about (once we havethe right machinery)viewing each replication asa single observation canmake the data easier tothink about (once we havethe right machinery)What are these data, anyway?

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisWhat is Functional Data?Classical Functional Data20 replications, 1401 observations within replications, 2 dimensionsMeasures of position of nib of a pen writing "fda". 20 replications,measurements taken at 200 hertz.Functional data is oftencomplex:often a large number ofrelated quantitiesviewing each replication asa single observation canmake the data easier tothink about (once we havethe right machinery)What are these data, anyway?What if I plot one component against another?BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisCharacteristicsAbout Functional Data Analysis1Data are measurements of smooth processes over timeFirst named in Dalzell & Ramsay, 1991Relatively little penetration into applied fields ( easypublication)Several competing methodologies (we focus on one)Limited public software/resourcesdata analysis rather than inferenceWe usually do not want to make parametric assumptionsabout those processes.Often have multiple measurements of the same processWe are interested in describing the variation of processes.Frequently, collected data have high resolution and low noise.Can be applied to any estimate of a smooth process.FDA is New2Functional Data is ComplexRequires more thought/judgement than a t-testdata needs pre-processingparametric inference is rarely available/appropriate

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisPre-requisites and RecommendationsAudience: application areas with functional dataFocus:What can Functional Data Analysis do?How do I make it happen?Software: packages in R, MatlabGoals: Enabling you toUnderstand and interpret the result of FDAapplied to real dataUse existing FDA libraries to analyze functionaldataEvaluate its usefulness/correctnessExtend the methods in existing software if youneed toPre-requisites: BTRY 601 and 602 or equivalent (at least multiplelinear regression)Useful: Life will be easier if you do not need to learn some ofthe following:R/Matlab or other programming experienceCalculusMatrix algebraMultivariate statisticsComputational statisticsAny necessary material will be covered in class, butwill be out of context.Not Covered: reproducing-kernel Hilbert spaces, asymptotics,theorems.BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisResourcesAssessmentTextbook: Ramsay and Silverman, 2005, Functional DataAnalysis, Springer.Books:Online:Ramsay and Silverman, 2002, Applied FunctionalData Analysis, Springer.Chapters from Ramsay, Graves and Hooker,(2009, hopefully) Functional Data Analysis in R.http://www.functionaldata.org for FDAhttp://www.r-project.org a general site for Rhttp://www.bscb.cornell.edu/ hooker/FDA2008All class notes, exercises etc will be posted here.Class materials will also be posted to Blackboard;a general discussion board has also been set up.3 Assignments (20% each)Using the FDA libraries to analyzedataInterpreting results of this analysisSome simulation studiesAnalysis of real-world dataClass Project (40%)End of semester presentationShort written report.More details later.Policies:you are welcome to discuss homework, but youshould do and write it individuallyproject may be done as a group, but should besubmitted with a statement of who did whichparts

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisBack to "What is Functional Data"Data may be measured more noisilyOr What isn’t Functional Data?Do my data need to look thisgood?We need to find the smooth process under the data.BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisData may be measured more sparselyWe may not have repeated measurementsSingle time seriesData are low noise butlow-resolutionBut, repeated "shapes"over each yearMeasured at unequalintervalsWe can use this toinvestigate variation,development, dynamicsWe know that the curvesmust always increase

BTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisNecessities for Functional DataCommon Sourcesmust believably derive from a smooth processprocess should not be easily parameterizable (should not beable to write down a formula)medical monitoring: EEG, ECG, fMRI, blood pressure .medical tests: HIV antibodies, flu screens.biology: animal behavior (whale songs, fly egg-laying.)enough data to resolve the essential features of the process(peaks, zero-crossings, speed. will depend on application)environmental monitoring: weather, pollution, solar radiation,traffic .some repetition in the processoptotrack experiments: psychology/physiologydo not need equally-spaced or perfect measurementseconomics/marketing: macro-trends, futures marketsweb data: e-bay auction prices, google trendsBTRY 6150: Applied Functional Data AnalysisBTRY 6150: Applied Functional Data AnalysisEssential QuestionsApproximate Class AgendaOr what can FDA do for me?1Introduction, R, Projects (weeks 1 and 2)2From data to functional data (weeks 3 - 6/7)Basis expansions and smoothingThe fda libraryPositive and monotone smoothingNo classes Sept 16 and 18How do we go from discrete to functional data?How do we describe random variation in functional data?How do we decide if groups of functional data are different?How do we relate functional data to other data? To otherfunctional data?3Means, variances, covariancesFunctional PCAWhat is special about functional data?Aligning functions (registration)Use of rates of change (dynamics)Exploring Functional Data (weeks 7-9)4Functional Linear Models (weeks 9 - 11)5Registration (week 12)6Dynamic Models (weeks 13-14)7Project Presentations (week 15)

Textbook: Ramsay and Silverman, 2005, Functional Data Analysis, Springer. Books: Ramsay and Silverman, 2002, Applied Functional Data Analysis, Springer. Chapters from Ramsay, Graves and Hooker, . All class notes, exercises etc will be posted here. Class materials will also be posted to Blackboard; a general discussion board has also been set up.