ELEG 3143 Probability & Stochastic Process Ch. 5 Elements .

Department of Electrical EngineeringUniversity of ArkansasELEG 3143 Probability & Stochastic ProcessCh. 5 Elements of StatisticsDr. Jingxian Wuwuj@uark.edu

OUTLINE Introduction: what is statistics? Sample mean and sample variance Confidence intervals Hypothesis testing2

INTRODUCTION Statistics– A bridge between the probability theory and the real world– The science of gathering and analyzing data, and with the drawing ofconclusions or inferences from the data. Example– 1. collect data about the life span of all females in US– 2. analyze the data to find out:» What is the average (mean) female life span in US?» What is the variance of the female life span?» What is the distribution of the female life span? Example– 1. consider a communication system transmitting -1 and 1– 2. received signal is distorted by noise (e.g., Tx a 1, Rx a 0.2)– 3. Based on the noisy received information, find out what wastransmitted (1 or -1)3

INTRODUCTION Classifications of statistics– Sampling theory: How to select samples from some collection of data that is too large tobe examined completely.– Estimation theory: Make some estimation or prediction based on the data that areavailable (e.g. estimate the average life span)– Hypothesis testing Attempts to decide which of two or more hypotheses about the dataare true (e.g. find out whether a 1 or -1 are transmitted in acommunication system)4

OUTLINE Introduction: what is statistics? Sample mean and sample variance Confidence interval Hypothesis testing5

SAMPLE MEAN AND VARIANCE Definition: population– the collection of ALL objects or elements under study.– E.g. measure the female life span in US member: the life span of one female in US (can be modeled as an RV:X) Population: the life spans of ALL the females in US– Most of the time, it is extremely difficult and expensive, if not impossible,to get the data for the entire population.– We take a limited number of samples to represent the population Definition: random sample (or, sample)– A random sample is part of the population that has been selected atrandom. E.g. Consider a population with N members. We can randomly pickn N members. The n members form a sample of the population. All members of the population are equally likely being picked. The picks are independent of each others.6

SAMPLE MEAN AND VARIANCE Random sample– Consider a random sample with n members––––Each member is a random variableThe n random variables are mutually independentThe n random variables are identically distributedWith the random sample, we can infer (estimate) the properties of thepopulation with N n members E.g. mean, variance, distribution, etc.– Intuitively, the larger the value of n, the better the estimation. We will prove this intuition.7

8SAMPLE MEAN AND VARIANCE Sample mean– The sample mean ofmˆ X is1nn Xii 1– It is an estimation of the actual mean of the population m– m̂ is a random variable, because it is a function of n RVsX E(X )X The first moment of the sample mean–m̂ XE [ mˆ X ] Definition: unbiased estimator– The mean of the estimation is the same as its true value– The sample mean is an unbiased estimation of the meanE [ mˆ X ] m X

9SAMPLE MEAN AND VARIANCE The 2nd central moment (variance) of the sample mean– Var ( mˆ X ) E mˆ X m X 2 E [ mˆX XVar ( mˆ X ) 2m̂ X] E [ mˆ X ]22n– Recall: variance measures the deviation from the mean The smaller the variance, the less the randomness If variance is 0, then the RV is a constant– The larger the sample size n, the more accurate the estimation.

SAMPLE MEAN AND VARIANCE Example– A population of 10 resistors is to be tested. The true standard deviation is5 Ohm, and the true mean is 100 Ohm. How large must be the sample size, if we want to obtain a samplemean, whose standard deviation is 2% of the true population mean? If the sample size is 8, what is the standard deviation of the samplemean?10

SAMPLE MEAN AND VARIANCE Example– The sample of a random time function follows a pdf given as follows:f X (x) ( x 3)exp 1010 12 The function is sampled so as to obtain independent sample values. Howmany sample values are required to obtain a sample mean, whose standarddeviation is 1% from the true mean?11

12SAMPLE MEAN AND VARIANCE Sample variance– The sample variance of 2S 1is defined asn n 1( X i mˆ X )2i 11– The n 1 is used to make – S is a random variable. 2San unbiased estimator .2 First moment of the sample variance 2E[S ] 2S2X– The sample variance is an unbiased estimation of 2X

SAMPLE MEAN AND VARIANCE Example– Consider 5 random numbers: 0.3, 0.2, 0.8, 0.7, 0.9 1. Find the sample mean and sample variance 2. If the 5 random numbers are randomly picked from a population ofrandom numbers that are uniformly distributed in [0, 1], find thevariance of the sample mean.13

SAMPLE MEAN AND VARIANCE Example– Write a Matlab program to generate 100 independent random numbers.The random number are samples of uniformly distributed randomvariable in (0, 10). (1) Write functions to find the sample mean and samplevariance. (2) What is the variance of the sample mean?%-----------------------------% main.mclear all;random sample 10*rand (1, 100); % generate 100 random numberssample mean find sample mean(random sample);sample var find sample var(random sample);%-------------------------------% find sample mean.mfunctionoutput find sample mean(input)n sample length(input);output sum(input)/n sample;14

SAMPLE MEAN AND VARIANCE Example (Cont’d)%-------------------------------% find sample var.mfunctionoutput find sample var(input)% find out how many members are in the samplen sample length(input);% calculate the sample meansample mean find sample mean(input);% calculate the sample varianceoutput sum( (input-sample mean). 2 )/(n sample-1);15

OUTLINE Introduction: what is statistics? Sample mean and sample variance Confidence interval Hypothesis testing16

17CONFIDENCE INTERVAL Central Limit Theorem– Letbe a random sample of size n. They are independentand identically distributed with mean m and variance . When n ,1the sample mean, mˆ X , converges in distribution to a Gaussian2XnXdistribution with meannXii 1mXand variance 2X.– When n is large (n 30), m̂ follows approximately to a Gaussiandistribution, regardless of the distribution of XXi

18CONFIDENCE INTERVAL Confidence interval– Example: estimate the mean, m , of a population of data. The sample mean, mˆ 1 X , is an RV nXnXii 1 thus it could be quite different from the true mean.– Specify an interval that is highly likely to contain the true value of m X E.g. It is 99% likely that the true value of m is in the interval [ mˆ a , mˆXXPr mˆ X a m X mˆ X a 0 . 99 [ mˆ X a , mˆ X a ]is called the 99% confidence interval of the meanX a]

19 Confidence Interval– Confidence interval v.s. sample mean1ˆm Sample mean attempts to use a single number, X , tonrepresent the sample mean– The sample mean is an RV, thus it could be quite different fromthe true mean.– How much can we trust this single number estimation? Confidence interval, instead, attempts to specify an interval that ishighly likely to contain the true value of the estimation.nXii 1– The q 100 % confidence interval for the estimation of the parameter m isdefined as [ mˆ a , mˆ a ] , such thatXXPr mˆ X a m X mˆ X a q

20CONFIDENCE INTERVAL Confidence Interval (Cont’d)Pr mˆ X a m X mˆ X a Pr m X a mˆ X m X a mX amX af mˆ ( x ) dxX– Based on the central limit theorem, m̂ X is Gaussian distributed with mean m X and variance2Xn mX amX aaf mˆ ( x ) dx X n a nXPr mˆ X a m X mˆ XPr m X1X2 ez22 a n dz 1 2 Q X a n a 1 2Q X a n ˆ a m X m X a 1 2Q X

21CONFIDENCE INTERVAL Example– A very large population of resistor values has a true mean of 100 Ohm anda standard deviation of 5 Ohm. Find the 95% confidence interval(confidence limits) on the sample mean if the sample size is 100.Q(1.96) 0.025 a n Pr m X a mˆ X m X a 1 2 Q X

OUTLINE Introduction: what is statistics? Sample mean and sample variance Confidence interval Hypothesis testing22

HYPOTHESIS TESTING Hypothesis testing– Testing an assertion about a population based on a random sample.– Example: Hypothesis: a given coin is fair Test: flip the coin 100 times, count the number of heads– If the coin is fair, we expect approximately 50 heads.– E.g. if the number of heads is in [47, 53], the hypothesis is true. Thehypothesis is false otherwise.– The interval [47, 53] is chosen arbitrarily. How to systematicallychoose the interval?– Example: Hypothesis: the light bulb from a certain manufacture can last 1000 hrs. Test: take 50 light bulbs, measure their life, and find the sample mean– If the sample mean is greater than t hrs, the hypothesis is true.– How do we determine the value t? 900 hrs? 950 hrs? 999 hrs?23

HYPOTHESIS TESTING Hypothesis testing:– Null hypothesis The hypothesis to be tested– Alternative hypothesis The complement (opposite) of the null hypothesis– Example: Test the hypothesis that a given coin is fair. Test the hypothesis that a lightbulb can last 1,000 hours Test the hypothesis that a certain patient does not have cancer24

25HYPOTHESIS TESTING Errors– Type I error: Reject H 0 when False positive, false alarm– Type II error: Accept False negativeH0whenHHis true00is false

HYPOTHESIS TESTING Hypothesis testing: significance testing– Test a hypothesis H 0 about a parameter p of a random variable X Example: test whether a coin is fair– X: Bernoulli RV with parameter p: P(X 1) p, P(X 0) 1-p– H 0 : p 0.5 (the coin is fair).– Objective: accept or reject the hypothesis based on a random sample26

27HYPOTHESIS TESTING Example– A certain coin is claimed to be fair with a 95% confidence interval. To testthe hypothesis, we flip the coin 100 times, and find the sample mean, m̂ X .If mˆ X 0 . 43 , can we accept the claim? p 0 .5 , 2 p pSol: Hypothesis: this is a fair coin m Xfind out if mˆ X 0 . 43 is in the 95% confidence intervalPr m X a n ˆ a m X m X a 1 2Q X X2 0 . 25

HYPOTHESIS TESTING Example:– A resistor manufacture is testing the quality of a batch of resistors withnominal value 1K Ohm, with a 95% confidence level. A sample of 100resistors are tested, and the sample mean is 1040 Ohm, and the samplestandard deviation is 100 Ohm. Do the resistors pass the quality check?28

HYPOTHESIS TESTING Binary hypothesis testing: Example29

HYPOTHESIS Receiver operating characteristics (ROC) curve– Plot true positive probability (complement of type II error) as a function offalse positive probability (type I error)30

HYPOTHESIS TESTING Hypothesis testing –Testing an assertion about a population based on a random sample. –Example: Hypothesis: a given coin is fair Test: flip the coin 100 times, count the number of heads –If the coin is fair, we expect approximately 50 heads. –E.g. if the number of heads is in [47, 53], the hypothesis is true .