Chapter 02_Finite Sample Properties Of The OLS Estimator

Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Chapter FINITE SAMPLE PROPERTIES OF THE OLS ESTIMATOR Y = X + ε • with ε ~ N [0, σ I ] rank(X) = k non-stochastic ε random → Y random • βˆ = ( X ′X ) −1 X ′Y ; βˆ is a statistics on a sample, βˆ is random because Y is random Being random: - βˆ has a probability distribution, called the sampling distribution - Repeatedly draw all possible random sample of size n calculate " βˆ " each time Let explore some statistical properties of the OLS estimator βˆ & build up its sampling distribution I UNBIASED: βˆ = ( X ′X ) −1 X ′Y = ( X ′X ) −1 X ′( Xβ + ε ) ′X ) −1 X ′X β + ( X ′X ) −1 X ′ε X = (  I = β + ( X ′X ) −1 X ′ε E( βˆ ) = E[ β + ( X ′X ) −1 X ′ε ] Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator = β + E[( X ′X ) −1 X ′ε ] E (ε ) = = β + ( X ′X ) −1 X ′ E ( βˆ ) = β ⇒ βˆ is an estimator of , it is a function of the random sample (the element of Y) Note: we talk about the sample → that means we talk about Y only Because X is a constant - fix matrix "Repeatedly draw all possible random samples of size n → draw Y" The least squares estimator is unbiased for (E(ε) = 0, X is non-stochastic) → ˆ ˆ E ( βˆ ))' ] (β VarCov( βˆ ) = E[( βˆ −  E ))( β −   β VarCov( βˆ ) βˆ − β = ( X ′X ) −1 X ′ε β = E [( βˆ − β )( βˆ − β )' ] = E[( X ′X ) −1 X ′ε )(( X ′X ) −1 X ′ε )' ] = E [( X ′X ) −1 X ′εε ' X ( X ′X ) −1 ] = ( X ′X ) −1 X ′E (εε ' ) X ( X ′X ) −1 = ( X ′X ) −1 X ′σ ε2 X ( X ′X ) −1 = σ ε2 ( X ′X ) −1 X ′X ( X ′X ) −1  I = σ ε2 ( X ′X ) −1 So: VarCov( βˆ ) = σ ε2 ( X ′X ) −1 For the model: ~ ~ ~ Yi = βˆ2 X i + βˆ3 X i + ei  βˆ2    βˆ3  βˆ =  σ ε ( X ′X ) −1 ~  ∑ X i23 = σ ε  − X~ X~  ∑ i i Nam T Hoang University of New England - Australia ∑ X ~X  ∑ X  ∑ X~ ~ ~ i2 i3 i2 i2 ~ X i23 − (∑ X~ i2 ~ X i3 ) University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator  βˆ  = VarCov    βˆ   3 σ ε2 ∑ X i23 ~ → Var ( βˆ ) = ∑X ~2 ~2 X i3 − i2 (∑ X~ i2 ~ X i3 ) σ ε2 / ∑ X i22 ~ ∑( X ~ ~ X i3 ) i2 = n2 ~2 ~ ∑ X i ∑ X i23 1− nn   r23 sample correlation between X i ; X i → Var ( βˆ ) = ∑X σ ε2 ~2 i2 (1 − r232 ) determined by: i σ ε2 ↑ → Var ( βˆ ) ↑ ii r232 ↑ → Var ( βˆ ) ↑ iii Variation in Xi2 iv n sample size ↑ → Var ( βˆ ) ↓ ∑X ~2 i2 ↑ → Var ( βˆ ) ↓ VarCov ( βˆ ) = σ ε2 ( X ′X ) −1 → we don't know σ ε2 → need an estimator for σ ε2 Define: σˆ ε2 = e' e n−k n: observations k: number of estimators e' e = ∑ ei2 = sum of squares • Show σˆ ε2 is an unbiased estimator e = Mε → e'e = ε'M'Mε=ε'Mε • Note: trace of a square matrix Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator n A is the sum of its principal diagonal elements (= n ×n ∑a i =1 ii ) Rules: A, B nxn matrix tr(A+B) = tr(A) + tr(B) tr(A.B) = tr(B.A) tr(λA) = λtr(A) Trace is a linear operation → sum of certain elements E ( e' e ) = E (ε ' Mε ) = E[tr (ε ' Mε )] = E[tr (εε ' M )] = trE (ε ' Mε ) = tr[σ ε2 I M )] = σ ε2 tr ( M ) = σ ε2 [tr ( I n ) − tr ( X ( X ' X ) −1 X ' )] = σ ε2 [n − tr ( X ( X ' X ) −1 X ')] = σ ε2 ( n − k )    I k ×k And: E ( e' e) σ ε2 ( n − k ) = σ ε2 = n−k n−k So: E (σˆ ε2 ) = σ ε2 → σˆ ε2 is an unbiased estimator of σ ε2 II LINEARITY: Any estimator that is a linear function of the random sample data is called a linear estimator Yi: random sample data ˆ β X ′X ) −1 X ′Y =  A Y  = (  k × n n ×1 k ×1 A where A is non-random: Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator  βˆ1   a11   a  βˆ2   21   =       βˆk  a k   a12 a 22  X k2  a1n  Y1   a n  Y2            a kn  Yn  → βˆ1 = a11Y1 + a12Y2 + + a1nYk → βˆ , OLS estimator is linear and unbiased for Because βˆ is a linear function of Y and Y is a linear function of ε, → if ε is normal then βˆ is normal So the sampling distribution of the OLS estimator of is: βˆ ~ N[ , σ ε2 ( X ′X ) −1 ] III EFFICIENCY: Suppose we have unbiased estimators, θˆ1 ; θˆ2 for θ Then we say θˆ1 is more efficient than θˆ2 if Var (θˆ1 ) ≤ Var (θˆ2 ) If θˆ1 ; θˆ2 are vectors unbiased estimators of θ , then θˆ1 is more efficient than θˆ2 if   k ×1 k ×1 k ×1 ∆ = [V (θˆ1 ) − V (θˆ2 )] is positive semi-definite IV GAUSS - MARKOV THEOREM: "Under the assumptions of the classical regression model, the least squares estimators of , βˆ = ( X ′X ) −1 X ′Y are the best linear unbiased estimators" (BLUE) Linear: in Y Best: Best for any alternative linear on unbiased estimators Var ( βˆ j ) ≤ Var (b j ) ∀j Proof: Let b is any other linear estimator of : Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator b =  A Y k ×1 Unbiased: k × n n ×1 E(b) = E(b) = E(AY) =E(AX + Aε) E(b) = AX + = AX = → AX =I Let A = (X'X)-1X' + C where C is any non-stochastic (k×n) matrix I = AX = [( X ' X ) −1 X '+C ] X = ( X ' X ) −1 X ' X + CX = CX =   I b = AY = [( X ' X ) −1 X '+C ][ Xβ + ε ] = ( X ' X ) −1 X ' X β + ( X ' X ) −1 X ' ε + CXβ + Cε I = β + ( X ' X ) −1 X ' ε + Cε VarCov(b) = E[(b − β )(b − β )' ] = E{[( X ' X ) −1 X ' ε + Cε ][( X ' X ) −1 X ' ε + Cε ]' } = E[( X ' X ) −1 X ' (εε ' ) X ( X ' X ) −1 + ( X ' X ) −1 (εε ' )C '+Cεε ' X ( X ' X ) −1 + Cεε ' C ' ] = σ ε2 ( X ' X ) −1 X ' X ( X ' X ) −1 + σ ε2 ( X ' X ) −1 X ' C '+σ ε2 CX ( X ' X ) −1 + σ ε2 CC ' I = σ ε2 ( X ' X ) −1 + σ ε2 CC '  VarCov ( βˆ ) The jth diagonal element: n Var (b j ) = Var ( βˆ j ) + σ ε2 ∑ c 2ji ≥ Var ( βˆ j ) ∀j = 1, k i =1 → Var (b j ) ≥ Var ( βˆ j ) ∀j = 1, k → βˆ j is the best linear unbiased estimator (BLUE) → βˆ j is efficient estimator (smallest variance) Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator V REVIEW: STATISTICAL INFERENCE: Linear function of normal random variables are also normal: u N( µ ) , Σ ~ n ×1 n × n n ×1 Z P u is normally distributed  =  → m × n n ×1 m ×1 E ( Z ) = E ( Pu ) = PE (u ) = Pµ VarCov( Z ) = E [( Z − E ( Z ))( Z − E ( Z ))' ] = E[( Pu − Pµ )( Pu − Pµ )' ] µ = P E[( u − )( u −µ )' ]P' = PΣP'  Σ Then Z N ( Pµ , PΣP' ) ~ Chi-squared distribution: If Z r×1 or Z ' Z ~ N (0, I ) then Z'Z has the Chi-squared distribution with r degree of freedom χ [2r ] Z'Z ~ r: number of these independent standard normal variables in the sum of squares: Theorem: If Z r×1 ~ N (0, I ) and A is idempotent with rank equal to r, then: n ×n ~ χ [2r ] i Z ' AZ ii r = tr ( A) = rank ( A) Eigenvalue - eigenvector problem: For a square matrix A , we can find n pairs of (λ j , c j ) such that: n ×n A c j = (λ j c j ) n ×n n ×1 1×1 n ×1 j = 1,2, , n 1×1 n ×1 Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator n ( ∑ c 2j = 1) normalizing: c j ' c j = j =1 The eigenvectors are orthogonal to each other: ci ' c j = (∀i ≠ j ) so c = [c1, c2, , cn] is an orthogonal matrix: ( c ' = c −1 ) c' c = I Eigenvalue - eigenvector problem: A c j = (λ j c j ) n ×n n ×1 cj'cj = Let: → j = 1,2, , n 1×1 n ×1 ci ' c j = C = [c1 c1 j    c2 j cj =       cnj  (∀i ≠ j ) c2  cn ] ⇒ c' c = I n ×n n ×n c' = c-1: orthogonal matrix: AC = A[c1 AC = [c1 Ac2  Acn ] = [c1λ1 c2  cn ] = [ Ac1 c2 c λ2  c n λn ] λ1   0 λ  0  = CΛ  cn ]         λn  0   Λ where Λ is a diagonal matrix: C ' AC = C ' CΛ = Λ and also Rank ( A) = Rank ( Λ ) = number of no-zero of λj's Note: C' AC = Λ → C ' −1 C ' ACC −1 = (C ' ) −1 ΛC −1 = CΛC ' Remember: A = CΛC ' and C' AC = Λ ; C'C = I, C' = C-1 Theorem: Let A be an idempotent matrix with rank = r and let Z r×1 Z ' AZ Nam T Hoang University of New England - Australia ~ ~ N (0, I ) then: χ [2r ] and rank ( A) = tr ( A) University of Economics - HCMC - Vietnam Advanced Econometrics Proof: C' AC = Λ , Chapter 2: Finite Sample Properties Of The OLS Estimator Z ~ r×1 N (0, I ) For A idempotent, λj = or Because: AC j = C j λ j → AAC j = AC j λ j = C j λ2j So: C j λ2j = C j λ j → C j (λ2j − λ j ) = → C j λ j (λ j − 1) = → λ j = or λ j = 1 0  Write: C' AC = Λ =    0 0 0 0     0 0       There must be r nonzero elements of Λ , because rank ( A) = r = rank ( Λ ) = tr ( Λ ) since all diagonal elements are or (Rule: tr(A.B) = tr(B.A)) Also tr ( Λ ) = tr ( ACC ' ) = tr ( A) so rank ( A) = tr ( A) = r u = C ) ' , Z n ×1 Z n×1 n × n n ×1 ~ N (0, I ) ' )C = C ' C = I E (uu ' ) = E (C ' ZZ ' C ) = C '  E ( ZZ I Contruct quadratic form: n u' Λu = Z ' C (C ' AC )C ' Z = Z ' AZ = ∑ ui2 ~ χ [2r ] i =1 So if Z ~ N (0, I ) and A is idempotent with rank equal to r, then n ×n Z ' AZ Extension: So if Z ~ N (0, σ I ) , then ~ Z ' AZ σ χ [2r ] ~ χ [2r ] Other distribution: Let Z be N(0,I) and let W be χ [r2 ] and let Z and W be independently distributed, then: Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Z W ~ t[ r ] r has the t-distribution with r degree of freedom Let W be χ [r2 ] and let v be χ [s2 ] and W and v be independently distributed, then: W v r ~ Fsr s has the F-distribution with r (numerator) and s (denominator) degree of freedom VI TESTING HYPOTHESIS ON INDIVIDUAL COEFFICIENT: Y = X + ε • ε ~ N [0, σ I ] with Recall: βˆ ~ N[ , σ ε2 ( X ′X ) −1 ] So βˆ j ~ N[ j, σ ε2 [( X ′X ) −1 ]ij ] → βˆ j − β j σ ( X ' X ) −jj1 ~ N [0,1] but σ2, so this can't be used directly for constructing test or confidence intervals e' e = ε ' M ' Mε = ε ' Mε , M is idempotent with with rank(M) = its trace = n-k ε ~ N [0, σ I ] → ε / σ ~ N [0, I ] ( n ×1) ⇒ e' e σ = ε ' Mε σ2 ~ χ [2n − k ] βˆ j − β j So follow theorem: σ ( X ' X ) −jj1 ~ tn −k e' e σ2 ⇔ βˆ j − β j e' e ( X ' X ) −jj1 n k −  (n − k ) ~ tn −k σˆ Nam T Hoang University of New England - Australia 10 University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator ⇔ βˆ j − β j σˆ ( X ' X ) −jj1 ~ tn −k σˆ ( X ' X ) −jj1 = σˆ β2ˆ = standard error of βˆ j j Finally: βˆ j − β j σˆ β2ˆ ~ tn −k j This basic result enables us to test hypothesis about elements of and to construct confidence intervals for them (note that we need the assumption of normality of ε's) EX: yˆ i = 1.4 + 0.2 xi + 0.6 xi ( 0.7 ) 0.05 H0: =0 H1: >0 t= βˆ j − β j SE ( βˆi ) (1.4 ) = 0.2 − =4 0.05 tα (5%) = 1.74 d.o.f = n-k =17 tα (1%) = 2.567 t > tα → reject H0 EX: H0: = 1.5 H1: ≠ 1.5 ( or ≥ 1.5 or ≤ 1.5) t= βˆ j − β j SE ( βˆi ) = 1.4 − 1.5 = −0.1429 d.o.f = n-k =17 0.7 2.5% Nam T Hoang University of New England - Australia 2.5% 11 University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator t < tα / ⇒ cannot reject H0 at 5% VII CONFIDENCE INTERVALS: βˆi − β i SE ( βˆi ) Recall: ti = so Pr[ −tα / ≤ ti ≤ −tα / ] = − α Pr[ −tα / ≤ ~ tn −k βˆi − β i ≤ − tα / ] = − α SE ( βˆi ) Pr[ βˆi − tα / SE ( βˆi ) ≤ β i ≤ βˆi + tα / SE ( βˆi )] = − α • If we were to take a sample of size "n", construct this repeat many times then 100(1-α)% of such intervals would cover the true value of i • If we construct the interval once, there is no guarantee that the internal will cover the true i] • Type of errors: size & power of tests Type I: Reject H0 when it is true Type II: Accept H0 when it is false Assume: Prob(type I error) = α Prob(type II error) = If sample size is fixed: α↓ ⇒ ↑ call α: significant level or size of the test → Fix α and try to design the test so to minimize • Definition: The power of a test is 1- Power = - Pr(accept H0/H0 false) = Pr(reject H0/H0 false) Nam T Hoang University of New England - Australia 12 University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator • A test is "uniformly most powerful" if its power exceeds that of any other test (for the same choice of α) over all possible alternative hypothesis • A test is "consistent" if its power → as n →∞ for any false hypothesis • A test is unbiased of its power never falls below α VIII FAMILY OF F-TEST: For general linear restrictions, unrestricted model (U-model), original model H0: some restrictions on β These define the restricted model (R-model): k ×1 r Fdfu = ( ESS R − ESSU ) / r ESSU ) / dfu ESSR = error sum of squares from R-model: e′R e R ESSU = error sum of squares from U-model: eU′ eU r: number of restrictions in H0 dfu: degree of freedom in U-model = n-k ESSU σ = =  ESS R  σ   ESSU  σ eU′ eU σ = ε ′Mε σ2 ε′ ε M σ σ ~ ~ ~ χ [2n − k ] χ [2n − ( k − r )] → χ [2n − k ] ESS R σ − ESSU σ2 ~ χ [2r ] ( ESS R − ESSU ) / σ r ( ESS R − ESSU ) / r = ESSU ) /(n − k )σ ESSU ) /(n − k ) → ( ESS R − ESSU ) / r ESSU ) /(n − k ) Nam T Hoang University of New England - Australia ~ Fnr− k 13 University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator Case 1: Join significant of all slopes: β  β =  1 k ×1  β  k −1 H0: β = → r = k −1 ( k −1) ×1 U-model: Y = X β +ε → ESSU =e'e R-model: Yi = β1 + ε i → βˆ1 + Y → Yi = Y + ei k ×1 dfu = n-k n ESS R = ∑ (Yi − Y ) i =1 n → Fnk−−k1 = ( ∑ (Yi − Y ) − e' e) /(k − 1) i =1 e' e /(n − k ) = R /(k − 1) (1 − R ) /(n − k ) Case 2: k −r β  β =  1 k ×1 β  r H0: β = U-model: Y = Xβ + ε → ESSU = eU′ eU R-model: Y = X β +ε → ESSU = e′R e R r ×1 r ×1 ( k − r ) ×1 n ESS R = ∑ (Yi − Y ) i =1 → EX: Fnr− k = ( ESS R − ESSU ) / r ESSU ) /(n − k ) Translog of production function: log Y = β1 + β log K + β log L + β (log K ) / + β (log L) / + β (log K log L) + ε H : β = β = β = Cobb-Douglas restrictions n = 27 ESSU = 0.67993 r=3 ESSR = 0.85163 n - k = 21 Nam T Hoang University of New England - Australia 14 University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator → Fnr− k = 1.768 Critical value: F213 ,5% = 3.1 → Fnr− k < Critical value ⇒ So not reject H0 and conclude that are consistent with the Cobb-Douglas model Case 3: General restrictions  β1  β =  β   β  R β =C r × k k ×1 r ×1 Restrictions: β2 + β3 = r ×1 r ×1 r ×1 → [0 1]β = ( r = 1)    R If restrictions: β + β = ( r = 2)  β1 = 0 1  1  →  β =   1 0  0 Jarque - Beta statistics: H0: εi are normally distributed H1: εi are not normally distributed JB ~ χ 22 JB = SK2 +(Kur)2 Reject H0 for large JB Reject H0 if JB >7 (critical) or if p-value < 0.05 Nam T Hoang University of New England - Australia 15 University of Economics - HCMC - Vietnam ... Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator = β + E[( X ′X ) −1 X ′ε ] E (ε ) = = β + ( X ′X ) −1 X ′ E ( βˆ ) = β ⇒ βˆ is an estimator of , it is a function of the random sample. .. trace of a square matrix Nam T Hoang University of New England - Australia University of Economics - HCMC - Vietnam Advanced Econometrics Chapter 2: Finite Sample Properties Of The OLS Estimator. .. , OLS estimator is linear and unbiased for Because βˆ is a linear function of Y and Y is a linear function of ε, → if ε is normal then βˆ is normal So the sampling distribution of the OLS estimator