AnIntroductionto StatisticalSignalProcessing
iAn Introduction toStatistical Signal Processingf 1 (F )fFPr(f F ) P ({ω : ω F }) P (f 1 (F ))-January 4, 2011
iiAn Introduction toStatistical Signal ProcessingRobert M. GrayandLee D. DavissonInformation Systems LaboratoryDepartment of Electrical EngineeringStanford UniversityandDepartment of Electrical Engineering and Computer ScienceUniversity of Marylandc 2004 by Cambridge University Press. Copies of the pdf file may bedownloaded for individual use, but multiple copies cannot be made or printedwithout permission.
iiito our Families
ContentsPrefaceAcknowledgementsGlossary1 Introduction2 Probability2.1Introduction2.2Spinning pointers and flipping coins2.3Probability spaces2.4Discrete probability spaces2.5Continuous probability spaces2.6Independence2.7Elementary conditional probability2.8Problems3 Random variables, vectors, and processes3.1Introduction3.2Random variables3.3Distributions of random variables3.4Random vectors and random processes3.5Distributions of random vectors3.6Independent random variables3.7Conditional distributions3.8Statistical detection and classification3.9Additive noise3.10 Binary detection in Gaussian noise3.11 Statistical estimation3.12 Characteristic functions3.13 Gaussian random vectors3.14 Simple random processesvpage 32135142144145151152
vi456Contents3.15 Directly given random processes3.16 Discrete time Markov processes3.17 Nonelementary conditional probability3.18 ProblemsExpectation and averages4.1Averages4.2Expectation4.3Functions of random variables4.4Functions of several random variables4.5Properties of expectation4.6Examples4.7Conditional expectation4.8 Jointly Gaussian vectors4.9Expectation as estimation4.10 Implications for linear estimation4.11 Correlation and linear estimation4.12 Correlation and covariance functions4.13 The central limit theorem4.14 Sample averages4.15 Convergence of random variables4.16 Weak law of large numbers4.17 Strong law of large numbers4.18 Stationarity4.19 Asymptotically uncorrelated processes4.20 ProblemsSecond-order theory5.1Linear filtering of random processes5.2Linear systems I/O relations5.3Power spectral densities5.4Linearly filtered uncorrelated processes5.5Linear modulation5.6White noise5.7 Time averages5.8 Mean square calculus5.9 Linear estimation and filtering5.10 ProblemsA menagerie of processes6.1Discrete time linear models6.2Sums of iid random 92296299303331349363364369
Contents6.3Independent stationary increment processes6.4 Second-order moments of isi processes6.5Specification of continuous time isi processes6.6Moving-average and autoregressive processes6.7The discrete time Gauss–Markov process6.8Gaussian random processes6.9The Poisson counting process6.10 Compound processes6.11 Composite random processes6.12 Exponential modulation6.13 Thermal noise6.14 Ergodicity6.15 Random fields6.16 ProblemsAppendix A PreliminariesA.1 Set theoryA.2 Examples of proofsA.3 Mappings and functionsA.4 Linear algebraA.5 Linear system fundamentalsA.6 ProblemsAppendix B Sums and integralsB.1SummationB.2 Double sumsB.3IntegrationB.4 The Lebesgue integralAppendix C Common univariate distributionsAppendix D Supplementary 443446448453457
PrefaceThe origins of this book lie in our earlier book Random Processes: A Mathematical Approach for Engineers (Prentice Hall, 1986). This book began asa second edition to the earlier book and the basic goal remains unchanged– to introduce the fundamental ideas and mechanics of random processes toengineers in a way that accurately reflects the underlying mathematics, butdoes not require an extensive mathematical background and does not belabor detailed general proofs when simple cases suffice to get the basic ideasacross. In the years since the original book was published, however, it hasevolved into something bearing little resemblance to its ancestor. Numerous improvements in the presentation of the material have been suggestedby colleagues, students, teaching assistants, and reviewers, and by our ownteaching experience. The emphasis of the book shifted increasingly towardsexamples and a viewpoint that better reflected the title of the courses wetaught using the book for many years at Stanford University and at theUniversity of Maryland: An Introduction to Statistical Signal Processing.Much of the basic content of this course and of the fundamentals of randomprocesses can be viewed as the analysis of statistical signal processing systems: typically one is given a probabilistic description for one random object,which can be considered as an input signal. An operation is applied to theinput signal (signal processing) to produce a new random object, the outputsignal. Fundamental issues include the nature of the basic probabilistic description, and the derivation of the probabilistic description of the outputsignal given that of the input signal and the particular operation performed.A perusal of the literature in statistical signal processing, communications,control, image and video processing, speech and audio processing, medical signal processing, geophysical signal processing, and classical statisticalareas of time series analysis, classification and regression, and pattern recognition shows a wide variety of probabilistic models for input processes andix
xPrefacefor operations on those processes, where the operations might be deterministic or random, natural or artificial, linear or nonlinear, digital or analog, orbeneficial or harmful. An introductory course focuses on the fundamentalsunderlying the analysis of such systems: the theories of probability, randomprocesses, systems, and signal processing.When the original book went out of print, the time seemed ripe to convertthe manuscript from the prehistoric troff format to the widely used LATEXformat and to undertake a serious revision of the book in the process. As therevision became more extensive, the title changed to match the course nameand content. We reprint the original preface to provide some of the originalmotivation for the book, and then close this preface with a description ofthe goals sought during the many subsequent revisions.Preface to Random Processes: An Introduction for EngineersNothing in nature is random . . . A thing appears random onlythrough the incompleteness of our knowledge.Spinoza, Ethics II do not believe that God rolls dice.attributed to EinsteinLaplace argued to the effect that given complete knowledge of the physics of anexperiment, the outcome must always be predictable. This metaphysical argumentmust be tempered with several facts. The relevant parameters may not be measurable with sufficient precision due to mechanical or theoretical limits. For example,the uncertainty principle prevents the simultaneous accurate knowledge of both position and momentum. The deterministic functions may be too complex to computein finite time. The computer itself may make errors due to power failures, lightning,or the general perfidy of inanimate objects. The experiment could take place in aremote location with the parameters unknown to the observer; for example, in acommunication link, the transmitted message is unknown a priori, for if it were not,there would be no need for communication. The results of the experiment could bereported by an unreliable witness – either incompetent or dishonest. For these andother reasons, it is useful to have a theory for the analysis and synthesis of processes that behave in a random or unpredictable manner. The goal is to constructmathematical models that lead to reasonably accurate prediction of the long-termaverage behavior of random processes. The theory should produce good estimatesof the average behavior of real processes and thereby correct theoretical derivationswith measurable results.In this book we attempt a development of the basic theory and applications ofrandom processes that uses the language and viewpoint of rigorous mathematicaltreatments of the subject but which requires only a typical bachelor’s degree level ofelectrical engineering education including elementary discrete and continuous timelinear systems theory, elementary probability, and transform theory and applica-
Prefacexitions. Detailed proofs are presented only when within the scope of this background.These simple proofs, however, often provide the groundwork for “handwaving” justifications of more general and complicated results that are semi-rigorous in thatthey can be made rigorous by the appropriate delta-epsilontics of real analysis ormeasure theory. A primary goal of this approach is thus to use intuitive argumentsthat accurately reflect the underlying mathematics and which will hold up underscrutiny if the student continues to more advanced courses. Another goal is to enable the student who might not continue to more advanced courses to be able toread and generally follow the modern literature on applications of random processesto information and communication theory, estimation and detection, control, signalprocessing, and stochastic systems theory.RevisionsThrough the years the original book has continually expanded to roughlydouble its original size to include more topics, examples, and problems. Thematerial has been significantly reorganized in its grouping and presentation.Prerequisites and preliminaries have been moved to the appendices. Majoradditional material has been added on jointly Gaussian vectors, minimummean squared error estimation, linear and affine least squared error estimation, detection and classification, filtering, and, most recently, mean squarecalculus and its applications to the analysis of continuous time processes.The index has been steadily expanded to ease navigation through the book.Numerous errors reported by reader email have been fixed and suggestionsfor clarifications and improvements incorporated.This book is a work in progress. Revised versions will be made availablethrough the World Wide Web page http://ee.stanford.edu/ gray/sp.html.The material is copyrighted by Cambridge University Press, but is freelyavailable as a pdf file to any individuals who wish to use it provided onlythat the contents of the entire text remain intact and together. Comments,corrections, and suggestions should be sent to rmgray@stanford.edu. Everyeffort will be made to fix typos and take suggestions into account on at leastan annual basis.
AcknowledgementsWe repeat our acknowledgements of the original book: to Stanford University and the University of Maryland for the environments in which thebook was written, to the John Simon Guggenheim Memorial Foundationfor its support of the first author during the writing in 1981–2 of the original book, to the Stanford University Information Systems Laboratory Industrial Affiliates Program which supported the computer facilities used tocompose this book, and to the generations of students who suffered throughthe ever changing versions and provided a stream of comments and corrections. Thanks are also due to Richard Blahut and anonymous refereesfor their careful reading and commenting on the original book. Thanks aredue to the many readers who have provided corrections and helpful suggestions through the Internet since the revisions began being posted. Particularthanks are due to Yariv Ephraim for his continuing thorough and helpfuleditorial commentary. Thanks also to Sridhar Ramanujam, Raymond E.Rogers, Isabel Milho, Zohreh Azimifar, Dan Sebald, Muzaffer Kal, GregCoxson, Mihir Pise, Mike Weber, Munkyo Seo, James Jacob Yu, and severalanonymous reviewers for Cambridge University Press. Thanks also to PhilipMeyler, Lindsay Nightingale, and Joseph Bottrill of Cambridge UniversityPress for their help in the production of the final version of the book. Thanksto the careful readers who informed me of typos and mistakes in the bookfollowing its publication, all of which have been reported and fixed in the errata (http://ee.stanford.edu/ gray/sperrata.pdf) and incorporated into theelectronic version: Ian Lee, Michael Gutmann, André Isidio de Melo, andespecially Ron Aloysius, who has contributed greatly to the fixing of typos.Lastly, the first author would like to acknowledge his debt to his professors who taught him probability theory and random processes, especially AlDrake and Wilbur B. Davenport Jr. at MIT and Tom Pitcher at USC.xii
Glossary{}[]()( ], [ ) Ω#(F )a collection of points satisfying some property, e.g. {r :r a} is the collection of all real numbers less than orequal to a value aan interval of real points including the end points, e.g.for a b [a, b] {r : a r b}. Called a closed intervalan interval of real points excluding the end points, e.g.for a b (a, b) {r : a r b}. Called an open interval. Note this is empty if a bdenote intervals of real points including one endpointand excluding the other, e.g. for a b (a, b] {r : a r b}, [a, b) {r : a r b}the empty set, the set that contains no points.for allthe sample space or universal set, the set that containsall of the pointsthe number of elements in a set F equal by definitionexpthe exponential function, exp(x) ex , used for claritywhen x is complicatedsigma-field or event spaceBorel field of Ω, that is, the sigma-field of subsets ofthe real line generated by the intervals or the Cartesianproduct of a collection of such sigma-fieldsif and only iflimit in the meanfunction of u that goes to zero as u 0 faster than uFB(Ω)iffl.i.m.o(u) xiii
xivGlossaryPPXpXfXΦQprobability measuredistribution of a random variable or vector Xprobability mass function (pmf) of a random variable Xprobability density function (pdf) of a random variableXcumulative distribution function (cdf) of a random variable Xexpectation of a random variable Xcharacteristic function of a random variable Xaddition modulo 2indicator function of a set F : 1F (x) 1 if x F and 0otherwiseΦ-function (Eq. (2.78))complementary Phi function (Eq. (2.79))Zk {0, 1, 2, . . . , k 1}FXE(X)MX (ju) 1F (x)Z Z {0, 1, 2, . . .}, the collection of nonnegative integers {. . . , 2, 1, 0, 1, 2, . . .}, the collection of all integers
1IntroductionA random or stochastic process is a mathematical model for a phenomenonthat evolves in time in an unpredictable manner from the viewpoint of theobserver. The phenomenon may be a sequence of real-valued measurementsof voltage or temperature, a binary data stream from a computer, a modulated binary data stream from a modem, a sequence of coin tosses, thedaily Dow–Jones average, radiometer data or photographs from deep spaceprobes, a sequence of images from a cable television, or any of an infinitenumber of possible sequences, waveforms, or signals of any imaginable type.It may be unpredictable because of such effects as interference or noise in acommunication link or storage medium, or it may be an information-bearingsignal, deterministic from the viewpoint of an observer at the transmitterbut random to an observer at the receiver.The theory of random processes quantifies the above notions so thatone can construct mathematical models of real phenomena that are bothtractable and meaningful in the sense of yielding useful predictions of future behavior. Tractability is required in order for the engineer (or anyoneelse) to be able to perform analyses and syntheses of random processes, perhaps with the aid of computers. The “meaningful” requirement is that themodels must provide a reasonably good approximation of the actual phenomena. An oversimplified model may provide results and conclusions thatdo not apply to the real phenomenon being modeled. An overcomplicatedone may constrain potential applications, render theory too difficult to beuseful, and strain available computational resources. Perhaps the most distinguishing characteristic between an average engineer and an outstandingengineer is the ability to derive effective models providing a good balancebetween complexity and accuracy.Random processes usually occur in applications in the context of environments or systems which change the processes to produce other processes.1
2IntroductionThe intentional operation on a signal produced by one process, an “inputsignal,” to produce a new signal, an “output signal,” is generally referred toas signal processing, a topic easily illustrated by examples.r A time-varying voltage waveform is produced by a human speaking into a microphone or telephone. The signal can be modeled by a random process. Thissignal might be modulated for transmission, then it might be digitized and codedfor transmission on a digital link. Noise in the digital link can cause errors inreconstructed bits, the bits can then be used to reconstruct the original signalwithin some fidelity. All of these operations on signals can be considered as signalprocessing, although the name is most commonly used for manmade operationssuch as modulation, digitization, and coding, rather than the natural possiblyunavoidable changes such as the addition of thermal noise or other changes outof our control.r For digital speech communications at very low bit rates, speech is sometimesconverted into a model consisting of a simple linear filter (called an autoregressivefilter) and an input process. The idea is that the parameters describing the modelcan be communicated with fewer bits than can the original signal, but the receivercan synthesize the human voice at the other end using the model so that it soundsvery much like the original signal. A system of this type is called a vocoder .r Signals including image data transmitted from remote spacecraft are virtuallyburied in noise added to them on route and in the front end amplifiers of thereceivers used to retrieve the signals. By suitably preparing the signals prior totransmission, by suitable filtering of the received signal plus noise, and by suitabledecision or estimation rules, high quality images are transmitted through this verypoor channel.r Signals produced by biomedical measuring devices can display specific behaviorwhen a patient suddenly changes for the worse. Signal processing systems can lookfor these changes and warn medical personnel when suspicious behavior occurs.r Images produced by laser cameras inside elderly North Atlantic pipelines canbe automatically analyzed to locate possible anomalies indicating corrosion bylooking for locally distinct random behavior.How are these signals characterized? If the signals are random, how does onefind stable behavior or structures to describe the processes? How do operations on these signals change them? How can one use observations based onrandom signals to make intelligent decisions regarding future behavior? Allof these questions lead to aspects of the theory and application of randomprocesses.Courses and texts on random processes usually fall into either of twogeneral and distinct categories. One category is the common engineeringapproach, which involves fairly elementary probability theory, standard un-
Introduction3dergraduate Riemann calculus, and a large dose of “cookbook” formulas –often with insufficient attention paid to conditions under which the formulas are valid. The results are often justified by nonrigorous and occasionallymathematically inaccurate handwaving or intuitive plausibility argumentsthat may not reflect the actual underlying mathematical structure and maynot be supportable by a precise proof. While intuitive arguments can beextremely valuable in providing insight into deep theoretical results, theycan be a handicap if they do not capture the essence of a rigorous proof.A development of random processes that is insufficiently mathematicalleaves the student ill prepared to generalize the techniques
2.3 Probability spaces 22 2.4 Discrete probability spaces 44 2.5 Continuous probability spaces 54 2.6 Independence 68 2.7 Elementary conditional probability 70 2.8 Problems 73 3 Random variables, vectors, and processes 82 3.1 Introduction 82 3.2 Random variables 93 3.3 Distributions of random variables 102 3.4 Random vectors and random .
2011-present Assistant Professor,Department of Electrical and Computer Engineering,Tufts University. 2014-present Adjunct Assistant Professor,Department of Computer Science,TuftsUniversity. Research Technical foci erseProblems,Information Th
Anatomy is the study of the structure of living things. b. Physiology is the science of the functioning of living organisms and their component parts. SELF-ASSESSMENT EXERCISE 2 i. Factors that determine divisions in anatomy are: a. Degree of structural detail under consideration 5. HEM 604 BASIC ANATOMY AND PHYSIOLOGY OF HUMAN BODY b. Specific processes c. Medical application ii. The analysis .
Weight of pile above scour level Wp1 220.893 kN Weight of pile below scour level Wp2 301.548 kN Tota l ultimate resistance of pile Qsf Qb – Wp2 8717.452 kN Allowable load (8717.452 / F.S.) – Wp1 3266 kN. From above calculations, Required depth 26.03m below design seabed level E.G.L. ( ) 1.15 m CD . International Journal of Engineering Trends and Technology (IJETT) – Volume .
Graeme falls in love with Barbara Allan. He is so lovesick that he is bound to his deathbed. When Barbara comes to visit her ailing lover, she reminds him that he slighted her in front of others at a local tavern. He dies, and then she feels guilty, so she asks her mother to prepare her deathbed for the following day. The message might be that one doesn’t need to take love for granted, or it .
BS Honors in Botany 8 Semesters / 4 years Degree Program for the year 2011 and onwards Department of Botany. Scheme of Studies BS Hons in Botany GC University, Faisalabad 2 BS (HONS) BOTANY Semester 1 Semester 2 Course Code Course Title Credit Hours BOT-301 Diversity of Plants 4(3-1) ENG-321 English –I (Functional English) , (EAP) English for Academic Purpose 3(3-0) PST-321 Pakistan Studies .
Flyers A–Z Word List Grammatical key adj adjective adv adverb conj conjunction det determiner dis discourse marker excl exclamation int interrogative n noun poss possessive prep preposition pron pronoun v verb for exams from 2018. concert n conversation n cooker n cookie (UK biscuit) n corner n costume n could (for possibility) v creature n crown n cushion n cut v cycle v D dark adj date (as .
Careers Leader: Strategic The strategic leadership tasks will be done by a Careers Leader who is on a college’s senior leadership team, which includes managing the budget and other staff. CDI framework A structure for designing, delivering and assessing the school/college careers programme, from the UK-wide professional body for the career development sector. Compass An online tool provided .
Throughout the world, DoD installations are exposed to the risks of climate change, jeopardizing our nation’s security. By anticipating future climate change conditions, Army can reduce climate impacts to missions and operations and protect its real property investments by reducing exposure. The ACRH is intended to help Army
