Principles Of Statistical Inference - UFPR

xivPrefacevarious approaches in a dispassionate way but have added an appendix with amore personal assessment of the merits of different ideas.Some previous knowledge of statistics is assumed and preferably someunderstanding of the role of statistical methods in applications; the latterunderstanding is important because many of the considerations involved areessentially conceptual rather than mathematical and relevant experience isnecessary to appreciate what is involved.The mathematical level has been kept as elementary as is feasible and ismostly that, for example, of a university undergraduate education in mathematics or, for example, physics or engineering or one of the more quantitativebiological sciences. Further, as I think is appropriate for an introductory discussion of an essentially applied field, the mathematical style used here eschewsspecification of regularity conditions and theorem–proof style developments.Readers primarily interested in the qualitative concepts rather than their development should not spend too long on the more mathematical parts of thebook.The discussion is implicitly strongly motivated by the demands of applications, and indeed it can be claimed that virtually everything in the book hasfruitful application somewhere across the many fields of study to which statistical ideas are applied. Nevertheless I have not included specific illustrations.This is partly to keep the book reasonably short, but, more importantly, to focusthe discussion on general concepts without the distracting detail of specificapplications, details which, however, are likely to be crucial for any kind ofrealism.The subject has an enormous literature and to avoid overburdening the readerI have given, by notes at the end of each chapter, only a limited number of keyreferences based on an admittedly selective judgement. Some of the referencesare intended to give an introduction to recent work whereas others point towardsthe history of a theme; sometimes early papers remain a useful introduction toa topic, especially to those that have become suffocated with detail. A briefhistorical perspective is given as an appendix.The book is a much expanded version of lectures given to doctoral students ofthe Institute of Mathematics, Chalmers/Gothenburg University, and I am verygrateful to Peter Jagers and Nanny Wermuth for their invitation and encouragement. It is a pleasure to thank Ruth Keogh, Nancy Reid and Rolf Sundberg fortheir very thoughtful detailed and constructive comments and advice on a preliminary version. It is a pleasure to thank also Anthony Edwards and DeborahMayo for advice on more specific points. I am solely responsible for errors offact and judgement that remain.

PrefacexvThe book is in broadly three parts. The first three chapters are largely introductory, setting out the formulation of problems, outlining in a simple casethe nature of frequentist and Bayesian analyses, and describing some specialmodels of theoretical and practical importance. The discussion continues withthe key ideas of likelihood, sufficiency and exponential families.Chapter 4 develops some slightly more complicated applications. The longChapter 5 is more conceptual, dealing, in particular, with the various meaningsof probability as it is used in discussions of statistical inference. Most of the keyconcepts are in these chapters; the remaining chapters, especially Chapters 7and 8, are more specialized.Especially in the frequentist approach, many problems of realistic complexityrequire approximate methods based on asymptotic theory for their resolutionand Chapter 6 sets out the main ideas. Chapters 7 and 8 discuss various complications and developments that are needed from time to time in applications.Chapter 9 deals with something almost completely different, the possibility of inference based not on a probability model for the data but rather onrandomization used in the design of the experiment or sampling procedure.I have written and talked about these issues for more years than it is comfortable to recall and am grateful to all with whom I have discussed the topics,especially, perhaps, to those with whom I disagree. I am grateful particularlyto David Hinkley with whom I wrote an account of the subject 30 years ago.The emphasis in the present book is less on detail and more on concepts but theeclectic position of the earlier book has been kept.I appreciate greatly the care devoted to this book by Diana Gillooly, Commissioning Editor, and Emma Pearce, Production Editor, Cambridge UniversityPress.

1PreliminariesSummary. Key ideas about probability models and the objectives of statistical analysis are introduced. The differences between frequentist and Bayesiananalyses are illustrated in a very special case. Some slightly more complicatedmodels are introduced as reference points for the following discussion.1.1 Starting pointWe typically start with a subject-matter question. Data are or become availableto address this question. After preliminary screening, checks of data quality andsimple tabulations and graphs, more formal analysis starts with a provisionalmodel. The data are typically split in two parts (y : z), where y is regarded as theobserved value of a vector random variable Y and z is treated as fixed. Sometimesthe components of y are direct measurements of relevant properties on studyindividuals and sometimes they are themselves the outcome of some preliminaryanalysis, such as means, measures of variability, regression coefficients and soon. The set of variables z typically specifies aspects of the system under studythat are best treated as purely explanatory and whose observed values are notusefully represented by random variables. That is, we are interested solely in thedistribution of outcome or response variables conditionally on the variables z; aparticular example is where z represents treatments in a randomized experiment.We use throughout the notation that observable random variables are represented by capital letters and observations by the corresponding lower caseletters.A model, or strictly a family of models, specifies the density of Y to befY ( y : z; θ ),1(1.1)

2Preliminarieswhere θ θ is unknown. The distribution may depend also on design features of the study that generated the data. We typically simplify the notation tofY (y; θ ), although the explanatory variables z are frequently essential in specificapplications.To choose the model appropriately is crucial to fruitful application.We follow the very convenient, although deplorable, practice of using the termdensity both for continuous random variables and for the probability functionof discrete random variables. The deplorability comes from the functions beingdimensionally different, probabilities per unit of measurement in continuousproblems and pure numbers in discrete problems. In line with this conventionin what follows integrals are to be interpreted as sums where necessary. Thuswe write E(Y ) E(Y ; θ ) y fY (y; θ)dy(1.2)for the expectation of Y , showing the dependence on θ only when relevant. Theintegral is interpreted as a sum over the points of support in a purely discrete case.Next, for each aspect of the research question we partition θ as (ψ, λ), where ψis called the parameter of interest and λ is included to complete the specificationand commonly called a nuisance parameter. Usually, but not necessarily, ψ andλ are variation independent in that θ is the Cartesian product ψ λ . Thatis, any value of ψ may occur in connection with any value of λ. The choice ofψ is a subject-matter question. In many applications it is best to arrange that ψis a scalar parameter, i.e., to break the research question of interest into simplecomponents corresponding to strongly focused and incisive research questions,but this is not necessary for the theoretical discussion.It is often helpful to distinguish between the primary features of a modeland the secondary features. If the former are changed the research questions ofinterest have either been changed or at least formulated in an importantly different way, whereas if the secondary features are changed the research questionsare essentially unaltered. This does not mean that the secondary features areunimportant but rather that their influence is typically on the method of estimation to be used and on the assessment of precision, whereas misformulation ofthe primary features leads to the wrong question being addressed.We concentrate on problems where θ is a subset of Rd , i.e., d-dimensionalreal space. These are so-called fully parametric problems. Other possibilitiesare to have semiparametric problems or fully nonparametric problems. Thesetypically involve fewer assumptions of structure and distributional form butusually contain strong assumptions about independencies. To an appreciable

1.3 Some simple models3extent the formal theory of semiparametric models aims to parallel that ofparametric models.The probability model and the choice of ψ serve to translate a subject-matterquestion into a mathematical and statistical one and clearly the faithfulness ofthe translation is crucial. To check on the appropriateness of a new type of modelto represent a data-generating process it is sometimes helpful to consider howthe model could be used to generate synthetic data. This is especially the casefor stochastic process models. Understanding of new or unfamiliar models canbe obtained both by mathematical analysis and by simulation, exploiting thepower of modern computational techniques to assess the kind of data generatedby a specific kind of model.1.2 Role of formal theory of inferenceThe formal theory of inference initially takes the family of models as given andthe objective as being to answer questions about the model in the light of thedata. Choice of the family of models is, as already remarked, obviously crucialbut outside the scope of the present discussion. More than one choice may beneeded to answer different questions.A second and complementary phase of the theory concerns what is sometimescalled model criticism, addressing whether the data suggest minor or majormodification of the model or in extreme cases whether the whole focus ofthe analysis should be changed. While model criticism is often done ratherinformally in practice, it is important for any formal theory of inference that itembraces the issues involved in such checking.1.3 Some simple modelsGeneral notation is often not best suited to special cases and so we use moreconventional notation where appropriate.Example 1.1. The normal mean. Whenever it is required to illustrate somepoint in simplest form it is almost inevitable to return to the most hackneyedof examples, which is therefore given first. Suppose that Y1 , . . . , Yn are independently normally distributed with unknown mean µ and known variance σ02 .Here µ plays the role of the unknown parameter θ in the general formulation.In one of many possible generalizations, the variance σ 2 also is unknown. Theparameter vector is then (µ, σ 2 ). The component of interest ψ would often be µ

4Preliminariesbut could be, for example, σ 2 or µ/σ , depending on the focus of subject-matterinterest.Example 1.2. Linear regression. Here the data are n pairs ( y1 , z1 ), . . . , (yn , zn )and the model is that Y1 , . . . , Yn are independently normally distributed withvariance σ 2 and withE(Yk ) α βzk .(1.3)Here typically, but not necessarily, the parameter of interest is ψ β and thenuisance parameter is λ (α, σ 2 ). Other possible parameters of interest includethe intercept at z 0, namely α, and α/β, the intercept of the regression lineon the z-axis.Example 1.3. Linear regression in semiparametric form. In Example 1.2replace the assumption of normality by an assumption that the Yk are uncorrelated with constant variance. This is semiparametric in that the systematic partof the variation, the linear dependence on zk , is specified parametrically and therandom part is specified only via its covariance matrix, leaving the functionalform of its distribution open. A complementary form would leave the systematic part of the variation a largely arbitrary function and specify the distributionof error parametrically, possibly of the same normal form as in Example 1.2.This would lead to a discussion of smoothing techniques.Example 1.4. Linear model. We have an n 1 vector Y and an n q matrix zof fixed constants such thatE(Y ) zβ,cov(Y ) σ 2 I,(1.4)where β is a q 1 vector of unknown parameters, I is the n n identitymatrix and with, in the analogue of Example 1.2, the components independentlynormally distributed. Here z is, in initial discussion at least, assumed of fullrank q n. A relatively simple but important generalization has cov(Y ) σ 2 V , where V is a given positive definite matrix. There is a correspondingsemiparametric version generalizing Example 1.3.Both Examples 1.1 and 1.2 are special cases, in the former the matrix zconsisting of a column of 1s.Example 1.5. Normal-theory nonlinear regression. Of the many generalizations of Examples 1.2 and 1.4, one important possibility is that the dependenceon the parameters specifying the systematic part of the structure is nonlinear.For example, instead of the linear regression of Example 1.2 we might wish toconsiderE(Yk ) α β exp(γ zk ),(1.5)

1.3 Some simple models5where from the viewpoint of statistical theory the important nonlinearity is notin the dependence on the variable z but rather that on the parameter γ .More generally the equation E(Y ) zβ in (1.4) may be replaced byE(Y ) µ(β),(1.6)where the n 1 vector µ(β) is in general a nonlinear function of the unknownparameter β and also of the explanatory variables.Example 1.6. Exponential distribution. Here the data are ( y1 , . . . , yn ) and themodel takes Y1 , . . . , Yn to be independently exponentially distributed with density ρe ρy , for y 0, where ρ 0 is an unknown rate parameter. Note thatpossible parameters of interest are ρ, log ρ and 1/ρ and the issue will arise ofpossible invariance or equivariance of the inference under reparameterization,i.e., shifts from, say, ρ to 1/ρ. The observations might be intervals betweensuccessive points in a Poisson process of rate ρ. The interpretation of 1/ρ isthen as a mean interval between successive points in the Poisson process. Theuse of log ρ would be natural were ρ to be decomposed into a product of effectsof different explanatory variables and in particular if the ratio of two rates wereof interest.Example 1.7. Comparison of binomial probabilities. Suppose that the data are(r0 , n0 ) and (r1 , n1 ), where rk denotes the number of successes in nk binary trialsunder condition k. The simplest model is that the trials are mutually independentwith probabilities of success π0 and π1 . Then the random variables R0 and R1have independent binomial distributions. We want to compare the probabilitiesand for this may take various forms for the parameter of interest, for exampleψ log{π1 /(1 π1 )} log{π0 /(1 π0 )},orψ π1 π0 ,(1.7)and so on. For many purposes it is immaterial how we define the complementaryparameter λ. Interest in the nonlinear function log{π/(1 π )} of a probabilityπ stems partly from the interpretation as a log odds, partly because it maps theparameter space (0, 1) onto the real line and partly from the simplicity of someresulting mathematical models of more complicated dependences, for exampleon a number of explanatory variables.Example 1.8. Location and related problems. A different generalization ofExample 1.1 is to suppose that Y1 , . . . , Yn are independently distributed all withthe density g(y µ), where g(y) is a given probability density. We call µ

6Preliminariesa location parameter; often it may by convention be taken to be the mean ormedian of the density.A further generalization is to densities of the form τ 1 g{( y µ)/τ }, where τis a positive parameter called a scale parameter and the family of distributionsis called a location and scale family.Central to the general discussion of such models is the notion of a family oftransformations of the underlying random variable and the parameters. In thelocation and scale family if Yk is transformed to aYk b, where a 0 and b arearbitrary, then the new random variable has a distribution of the original formwith transformed parameter valuesaµ b, aτ .(1.8)The implication for most purposes is that any method of analysis should obeythe same transformation properties. That is, if the limits of uncertainty for sayµ, based on the original data, are centred on ỹ, then the limits of uncertainty forthe corresponding parameter after transformation are centred on aỹ b.Typically this represents, in particular, the notion that conclusions should notdepend on the units of measurement. Of course, some care is needed with thisidea. If the observations are temperatures, for some purposes arbitrary changesof scale and location, i.e., of the nominal zero of temperature, are allowable,whereas for others recognition of the absolute zero of temperature is essential.In the latter case only transformations from kelvins to some multiple of kelvinswould be acceptable.It is sometimes important to distinguish invariance that springs from somesubject-matter convention, such as the choice of units of measurement frominvariance arising out of some mathematical formalism.The idea underlying the above example can be expressed in much more general form involving two groups of transformations, one on the sample spaceand one on the parameter space. Data recorded as directions of vectors on acircle or sphere provide one example. Another example is that some of thetechniques of normal-theory multivariate analysis are invariant under arbitrarynonsingular linear transformations of the observed vector, whereas other methods, notably principal component analysis, are invariant only under orthogonaltransformations.The object of the study of a theory of statistical inference is to provide aset of ideas that deal systematically with the above relatively simple situationsand, more importantly still, enable us to deal with new models that arise in newapplications.

1.5 Two broad approaches to statistical inference71.4 Formulation of objectivesWe can, as already noted, formulate possible objectives in two parts as follows.Part I takes the family of models as given and aims to: give intervals or in general sets of values within which ψ is in some senselikely to lie; assess the consistency of the data with a particular parameter value ψ0 ; predict as yet unobserved random variables from the same random systemthat generated the data; use the data to choose one of a given set of decisions D, requiring thespecification of the consequences of various decisions.Part II uses the data to examine the family of models via a process of modelcriticism. We return to this issue in Section 3.2.We shall concentrate in this book largely but not entirely on the first twoof the objectives in Part I, interval e

1.2 Role of formal theory of inference 3 1.3 Some simple models 3 1.4 Formulation of objectives 7 1.5 Two broad approaches to statistical inference 7 1.6 Some further discussion 10 1.7 Parameters 13 Notes 1 14 2 Some concepts and simple applications 17 Summary 17 2.1