Notes For A Graduate-level Course In Asymptotics For .

1.1Limits and ContinuityFundamental to the study of large-sample theory is the idea of the limit of a sequence. Muchof these notes will be devoted to sequences of random variables; however, we begin here byfocusing on sequences of real numbers. Technically, a sequence of real numbers is a functionfrom the natural numbers {1, 2, 3, . . .} into the real numbers R; yet we always write a1 , a2 , . . .instead of the more traditional function notation a(1), a(2), . . .We begin by defining a limit of a sequence of real numbers. This is a concept that will beintuitively clear to readers familiar with calculus. For example, the fact that the sequencea1 1.3, a2 1.33, a3 1.333, . . . has a limit equal to 4/3 is unsurprising. Yet there aresome subtleties that arise with limits, and for this reason and also to set the stage for arigorous treatment of the topic, we provide two separate definitions. It is important toremember that even these two definitions do not cover all possible sequences; that is, notevery sequence has a well-defined limit.Definition 1.1 A sequence of real numbers a1 , a2 , . . . has limit equal to the real number a if for every 0, there exists N such that an a for all n N .In this case, we write an a as n or limn an a and we could say that“an converges to a”.Definition 1.2 A sequence of real numbers a1 , a2 , . . . has limit if for every realnumber M , there exists N such thatan M for all n N .In this case, we write an as n or limn an and we could saythat “an diverges to ”. Similarly, an as n if for all M , there existsN such that an M for all n N .Implicit in the language of Definition 1.1 is that N may depend on . Similarly, N maydepend on M (in fact, it must depend on M ) in Definition 1.2.The symbols and are not considered real numbers; otherwise, Definition 1.1 wouldbe invalid for a and Definition 1.2 would never be valid since M could be taken tobe . Throughout these notes, we will assume that symbols such as an and a denote realnumbers unless stated otherwise; if situations such as a are allowed, we will state thisfact explicitly.A crucial fact regarding sequences and limits is that not every sequence has a limit, evenwhen “has a limit” includes the possibilities . (However, see Exercise 1.4, which asserts4

that every nondecreasing sequence has a limit.) A simple example of a sequence without alimit is given in Example 1.3. A common mistake made by students is to “take the limit ofboth sides” of an equation an bn or an inequality an bn . This is a meaningless operationunless it has been established that such limits exist. On the other hand, an operation that isvalid is to take the limit superior or limit inferior of both sides, concepts that will be definedin Section 1.1.1. One final word of warning, though: When taking the limit superior of astrict inequality, or must be replaced by or ; see the discussion following Lemma1.10.Example 1.3 Definean log n;bn 1 ( 1)n /n;cn 1 ( 1)n /n2 ;dn ( 1)n .Then an , bn 1, and cn 1; but the sequence d1 , d2 , . . . does not have alimit. (We do not always write “as n ” when this is clear from the context.)Let us prove one of these limit statements, say, bn 1. By Definition 1.1, givenan arbitrary 0, we must prove that there exists some N such that bn 1 whenever n N . Since bn 1 1/n, we may simply take N 1/ : With thischoice, whenever n N , we have bn 1 1/n 1/N , which completes theproof.We always assume that log n denotes the natural logarithm, or logarithm base e,of n. This is fairly standard in statistics, though in some other disciplines it ismore common to use log n to denote the logarithm base 10, writing ln n insteadof the natural logarithm. Since the natural logarithm and the logarithm base 10differ only by a constant ratio—namely, loge n 2.3026 log10 n—the difference isoften not particularly important. (However, see Exercise 1.27.)Finally, note that although limn bn limn cn in Example 1.3, there is evidentlysomething different about the manner in which these two sequences approach thislimit. This difference will prove important when we study rates of convergencebeginning in Section 1.3.Example 1.4 A very important example of a limit of a sequence is c n exp(c)lim 1 n nfor any real number c. This result is proved in Example 1.20 using l’Hôpital’srule (Theorem 1.19).Two or more sequences may be added, multiplied, or divided, and the results follow intuitively pleasing rules: The sum (or product) of limits equals the limit of the sums (orproducts); and as long as division by zero does not occur, the ratio of limits equals the limit5

of the ratios. These rules are stated formally as Theorem 1.5, whose complete proof is thesubject of Exercise 1.1. To prove only the “limit of sums equals sum of limits” part of thetheorem, if we are given an a and bn b then we need to show that for a given 0,there exists N such that for all n N , an bn (a b) . But the triangle inequalitygives an bn (a b) an a bn b ,(1.1)and furthermore we know that there must be N1 and N2 such that an a /2 for n N1and bn b /2 for n N2 (since /2 is, after all, a positive constant and we know an aand bn b). Therefore, we may take N max{N1 , N2 } and conclude by inequality (1.1)that for all n N , an bn (a b) ,2 2which proves that an bn a b.Theorem 1.5 Suppose an a and bn b as n . Then an bn a b andan bn ab; furthermore, if b 6 0 then an /bn a/b.A similar result states that continuous transformations preserve limits; see Theorem 1.16.Theorem 1.5 may be extended by replacing a and/or b by , and the results remain trueas long as they do not involve the indeterminate forms , 0, or / .1.1.1Limit Superior and Limit InferiorThe limit superior and limit inferior of a sequence, unlike the limit itself, are defined for anysequence of real numbers. Before considering these important quantities, we must first definesupremum and infimum, which are generalizations of the ideas of maximum and minumum.That is, for a set of real numbers that has a minimum, or smallest element, the infimumis equal to this minimum; and similarly for the maximum and supremum. For instance,any finite set contains both a minimum and a maximum. (“Finite” is not the same as“bounded”; the former means having finitely many elements and the latter means containedin an interval neither of whose endpoints are .) However, not all sets of real numberscontain a minimum (or maximum) value. As a simple example, take the open interval (0, 1).Since neither 0 nor 1 is contained in this interval, there is no single element of this intervalthat is smaller (or larger) than all other elements. Yet clearly 0 and 1 are in some senseimportant in bounding this interval below and above. It turns out that 0 and 1 are theinfimum and supremum, respectively, of (0, 1).An upper bound of a set S of real numbers is (as the name suggests) any value m such thats m for all s S. A least upper bound is an upper bound with the property that no smaller6

upper bound exists; that is, m is a least upper bound if m is an upper bound such that forany 0, there exists s S such that s m . A similar definition applies to greatestlower bound. A useful fact about the real numbers—a consequence of the completeness ofthe real numbers which we do not prove here—is that every set that has an upper (or lower)bound has a least upper (or greatest lower) bound.Definition 1.6 For any set of real numbers, say S, the supremum sup S is defined tobe the least upper bound of S (or if no upper bound exists). The infimuminf S is defined to be the greatest lower bound of S (or if no lower boundexists).Example 1.7 Let S {a1 , a2 , a3 , . . .}, where an 1/n. Then inf S, which may alsobe denoted inf n an , equals 0 even though 0 6 S. But supn an 1, which iscontained in S. In this example, max S 1 but min S is undefined.If we denote by supk n ak the supremum of {an , an 1 , . . .}, then we see that this supremum istaken over a smaller and smaller set as n increases. Therefore, supk n ak is a nonincreasingsequence in n, which implies that it has a limit as n (see Exercise 1.4). Similarly,inf k n ak is a nondecreasing sequence, which implies that it has a limit.Definition 1.8 The limit superior of a sequence a1 , a2 , . . ., denoted lim supn an orsometimes limn an , is the limit of the nonincreasing sequencesup ak ,sup ak ,k 1k 2.The limit inferior, denoted lim inf n an or sometimes limn an , is the limit of thenondecreasing sequenceinf ak , inf ak , . . . .k 1k 2Intuitively, the limit superior and limit inferior may be understood as follows: If we definea limit point of a sequence to be any number which is the limit of some subsequence, thenlim inf and lim sup are the smallest and largest limit points, respectively (more precisely,they are the infimum and supremum, respectively, of the set of limit points).Example 1.9 In Example 1.3, the sequence dn ( 1)n does not have a limit. However, since supk n dk 1 and inf k n dk 1 for all n, it follows thatlim inf dn 1.lim sup dn 1 andnnIn this example, the set of limit points of the sequence d1 , d2 , . . . is simply { 1, 1}.Here are some useful facts regarding limits superior and inferior:7

Lemma 1.10 Let a1 , a2 , . . . and b1 , b2 , . . . be arbitrary sequences of real numbers. lim supn an and lim inf n an always exist, unlike limn an . lim inf n an lim supn an limn an exists if and only if lim inf n an lim supn an , in which caselim an lim inf an lim sup an .nnn Both lim sup and lim inf preserve nonstrict inequalities; that is, if an bnfor all n, then lim supn an lim supn bn and lim inf n an lim inf n bn . lim supn ( an ) lim inf n an .The next-to-last claim in Lemma 1.10 is no longer true if “nonstrict inequalities” is replacedby “strict inequalities”. For instance, 1/(n 1) 1/n is true for all positive n, but the limitsuperior of each side equals zero. Thus, it is not true thatlim supn11 lim sup .n 1n nWe must replace by (or by ) when taking the limit superior or limit inferior of bothsides of an inequality.1.1.2ContinuityAlthough Definitions 1.1 and 1.2 concern limits, they apply only to sequences of real numbers.Recall that a sequence is a real-valued function of the natural numbers. We shall also requirethe concept of a limit of a real-valued function of a real variable. To this end, we make thefollowing definition.Definition 1.11 For a real-valued function f (x) defined for all points in a neighborhood of x0 except possibly x0 itself, we call the real number a the limit of f (x)as x goes to x0 , writtenlim f (x) a,x x0if for each 0 there is a δ 0 such that f (x) a whenever 0 x x0 δ.First, note that Definition 1.11 is sensible only if both x0 and a are finite (but see Definition1.13 for the case in which one or both of them is ). Furthermore, it is very importantto remember that 0 x x0 δ may not be replaced by x x0 δ: The latter would8

imply something specific about the value of f (x0 ) itself, whereas the correct definition doesnot even require that this value be defined. In fact, by merely replacing 0 x x0 δby x x0 δ (and insisting that f (x0 ) be defined), we could take Definition 1.11 to bethe definition of continuity of f (x) at the point x0 (see Definition 1.14 for an equivalentformuation).Implicit in Definition 1.11 is the fact that a is the limiting value of f (x) no matter whetherx approaches x0 from above or below; thus, f (x) has a two-sided limit at x0 . We may alsoconsider one-sided limits:Definition 1.12 The value a is called the right-handed limit of f (x) as x goes to x0 ,writtenlim f (x) a or f (x0 ) a,x x0 if for each 0 there is a δ 0 such that f (x) a whenever 0 x x0 δ.The left-handed limit, limx x0 f (x) or f (x0 ), is defined analagously: f (x0 ) a if for each 0 there is a δ 0 such that f (x) a whenever δ x x0 0.The preceding definitions imply thatlim f (x) a if and only if f (x0 ) f (x0 ) a;x x0(1.2)in other words, the (two-sided) limit exists if and only if both one-sided limits exist and theycoincide. Before using the concept of a limit to define continuity, we conclude the discussionof limits by addressing the possibilities that f (x) has a limit as x or that f (x) tendsto :Definition 1.13 Definition 1.11 may be expanded to allow x0 or a to be infinite:(a) We write limx f (x) a if for every 0, there exists N such that f (x) a for all x N .(b) We write limx x0 f (x) if for every M , there exists δ 0 such thatf (x) M whenever 0 x x0 δ.(c) We write limx f (x) if for every M , there exists N such that f (x) Mfor all x N .Definitions involving are analogous, as are definitions of f (x0 ) andf (x0 ) .9

As mentioned above, the value of f (x0 ) in Definitions 1.11 and 1.12 is completely irrelevant;in fact, f (x0 ) might not even be defined. In the special case that f (x0 ) is defined and equalto a, then we say that f (x) is continuous (or right- or left-continuous) at x0 , as summarizedby Definition 1.14 below. Intuitively, f (x) is continuous at x0 if it is possible to draw thegraph of f (x) through the point [x0 , f (x0 )] without lifting the pencil from the page.Definition 1.14 If f (x) is a real-valued function and x0 is a real number, then we say f (x) is continuous at x0 if limx x0 f (x) f (x0 ); we say f (x) is right-continuous at x0 if limx x0 f (x) f (x0 ); we say f (x) is left-continuous at x0 if limx x0 f (x) f (x0 ).Finally, even though continuity is inherently a local property of a function (since Definition 1.14 applies only to the particular point x0 ), we often speak globally of “a continuousfunction,” by which we mean a function that is continuous at every point in its domain.1.0Statement (1.2) implies that every (globally) continuous function is right-continuous. However, the converse is not true, and in statistics the canonical example of a function that isright-continuous but not continuous is the cumulative distribution function for a discreterandom variable.F(t)0.60.8 0.00.20.4 0.50.00.51.01.5tFigure 1.1: The cumulative distribution function for a Bernoulli (1/2) random variable isdiscontinuous at the points t 0 and t 1, but it is everywhere right-continuous.10

Example 1.15 Let X be a Bernoulli (1/2) random variable, so that the events X 0and X 1 each occur with probability 1/2. Then the distribution functionF (t) P (X t) is right-continuous but it is not continuous because it has“jumps” at t 0 and t 1 (see Figure 1.1). Using one-sided limit notation ofDefinition 1.12, we may write0 F (0 ) 6 F (0 ) 1/2 and 1/2 F (1 ) 6 F (1 ) 1.Although F (t) is not (globally) continuous, it is continuous at every point in theset R \ {0, 1} that does not include the points 0 and 1.We conclude with a simple yet important result relating continuity to the notion of thelimit of a sequence. Intuitively, this result states that continuous functions preserve limitsof sequences.Theorem 1.16 If a is a real number such that an a as n and the real-valuedfunction f (x) is continuous at the point a, then f (an ) f (a).Proof: We need to show that for any 0, there exists N such that f (an ) f (a) for all n N . To this end, let 0 be a fixed arbitrary constant. From the definition ofcontinuity, we know that there exists some δ 0 such that f (x) f (a) for all x suchthat x a δ. Since we are told an a and since δ 0, there must by definition besome N such that an a δ for all n N . We conclude that for all n greater than thisparticular N , f (an ) f (a) . Since was arbitrary, the proof is finished.Exercises for Section 1.1Exercise 1.1 Assume that an a and bn b, where a and b are real numbers.(a) Prove that an bn abHint: Show that an bn ab (an a)(bn b) a(bn b) b(an a) usingthe triangle inequality.(b) Prove that if b 6 0, an /bn a/b.Exercise 1.2 For a fixed real number c, define an (c) (1 c/n)n . Then Equation(1.9) states that an (c) exp(c). A different sequence with the same limit isobtained from the power series expansion of exp(c):bn (c) n 1 iXci 011i!

For each of the values c { 10, 1, 0.2, 1, 5}, find the smallest value of n suchthat an (c) exp(c) / exp(c) .01. Now replace an (c) by bn (c) and repeat.Comment on any general differences you observe between the two sequences.Exercise 1.3 (a) Suppose that ak c as k for a sequence of real numbersa1 , a2 , . . . Prove that this implies convergence in the sense of Cesáro, whichmeans thatn1Xak cas n .(1.3)n k 1In this case, c may be real or it may be .Hint: If c is real, consider the definition of ak c: There exists N such that ak c for all k N . Consider what happens when the sum in expression(1.3) is broken into two sums, one for k N and one for k N . The casec follows a similar line of reasoning.(b) Is the converse true? In other words, does (1.3) imply ak c?Exercise 1.4 Prove that if a1 , a2 , . . . is a nondecreasing (or nonincreasing) sequence,then limn an exists and is equal to supn an (or inf n an ). We allow the possibilitysupn an (or inf n an ) here.Hint: For the case in which supn an is finite, use the fact that the least upperbound M of a set S is defined by the fact that s M for all s S, but for any 0 there exists s S such that s M .Exercise 1.5 Let an sin n for n 1, 2, . . .(a) What is supn an ? Does maxn an exist?(b) What is the set of limit points of {a1 , a2 , . . .}? What are lim supn an andlim inf n an ? (Recall that a limit point is any point that is the limit of a subsequence ak1 , ak2 , . . ., where k1 k2 · · ·.)(c) As usual in mathematics, we assume above that angles are measured inradians. How do the answers to (a) and (b) change if we use degrees instead (i.e.,an sin n )?Exercise 1.6 Prove Lemma 1.10.Exercise 1.7 For x 6 {0, 1, 2}, definef (x) x3 x .x(x 1)(x 2)12

(a) Graph f (x). Experiment with various ranges on the axes until you attain avisually pleasing and informative plot that gives a sense of the overall behaviorof the function.(b) For each of x0 { 1, 0, 1, 2}, answer these questions: Is f (x) continuous atx0 , and if not, could f (x0 ) be defined so as to make the answer yes? What arethe right- and left-hand limits of f (x) at x0 ? Does it have a limit at x0 ? Finally,what are limx f (x) and limx f (x)?Exercise 1.8 Define F (t) as in Example 1.15 (and as pictured in Figure 1.1). Thisfunction is not continuous, so Theorem 1.16 does not apply. That is, an a doesnot imply that

ect a traditional view in graduate-level statistics education that students should learn measure-theoretic probability before large-sample the-ory. The philosophy of these notes is that these priorities are backwards, and that in fact statisticians have more to gain from an und