advanced engineering mathematics – mathematics

Homogeneous linear diff eqns with constant coefficients.. 1..[r]

(1)

Introduction to Time Series Analysis Lecture 7.

Peter Bartlett

Last lecture:

1 ARMA(p,q) models: stationarity, causality, invertibility The linear process representation of ARMA processes: ψ Autocovariance of an ARMA process

(2)

Peter Bartlett

1 Review: ARMA(p,q) models and their properties Review: Autocovariance of an ARMA process Homogeneous linear difference equations

Forecasting

1 Linear prediction

(3)

Review: Autoregressive moving average models

An ARMA(p,q) process {Xt} is a stationary process that satisfies

φ(B)Xt = θ(B)Wt,

where φ, θ are degree p, q polynomials and {Wt} ∼

W N(0, σ2)

(4)

Review: Properties of ARMA(p,q) models

Theorem: If φ and θ have no common factors, a (unique)

sta-tionary solution {Xt} to φ(B)Xt = θ(B)Wt

exists iff

φ(z) = − φ1z − · · · − φpzp = ⇒ |z| 6=

This ARMA(p,q) process is causal iff

φ(z) = − φ1z − · · · − φpzp = ⇒ |z| > It is invertible iff

(5)

Review: Properties of ARMA(p,q) models

φ(B)Xt = θ(B)Wt, ⇔ Xt = ψ(B)Wt

so θ(B) = ψ(B)φ(B)

⇔

⇔ = ψ0,

θ1 = ψ1 − φ1ψ0,

θ2 = ψ2 − φ1ψ1 − · · · − φ2ψ0,

(6)

Review: Autocovariance functions of ARMA processes

φ(B)Xt = θ(B)Wt, ⇔ Xt = ψ(B)Wt,

γ(h) − φ1γ(h − 1) − φ(2)γ(h − 2) − · · · = σw2

q

X

j=h

θjψj−h

We need to solve the homogeneous linear difference equation φ(B)γ(h) = (h > q), with initial conditions

γ(h)−φ1γ(h−1)− · · · −φpγ(h−p) = σw2 q

X

j=h

(7)

2 Review: Autocovariance of an ARMA process Homogeneous linear difference equations

Forecasting

1 Linear prediction

(8)

Homogeneous linear diff eqns with constant coefficients

a0xt + a1xt−1 + · · · + akxt−k =

⇔ a0 + a1B + · · · + akBkxt =

⇔ a(B)xt = auxiliary equation: a0 + a1z + · · · + akzk =

⇔ (z − z1)(z − z2)· · · (z − zk) =

where z1, z2, , zk ∈ C are the roots of this characteristic polynomial.

Thus,

(9)

a(B)xt = ⇔ (B − z1)(B − z2)· · ·(B − zk)xt =

So any {xt} satisfying (B −zi)xt = for some i also satisfies a(B)xt = Three cases:

1 The zi are real and distinct

(10)

1 The zi are real and distinct.

a(B)xt =

⇔ (B − z1)(B − z2) · · ·(B − zk)xt =

⇔ xt is a linear combination of solutions to

(B − z1)xt = 0, (B − z2)xt = 0, , (B − zk)xt =

⇔ xt = c1z

−t

1 + c2z

−t

2 + · · · + ckz

−t

k ,

(11)

1 The zi are real and distinct e.g., z1 = 1.2, z2 = −1.3

0 10 12 14 16 18 20 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 c

1 z1 −t

+ c

2 z2 −t

c

1=1, c2=0

c

1=0, c2=1

c

(12)

Reminder: Complex exponentials

a + ib = reiθ = r(cosθ + isinθ), where r = |a + ib| = pa2 + b2

θ = tan−1

b a

∈ (−π, π]

Thus, r1eiθ1r

2eiθ2 = (r1r2)ei(θ1+θ2),

(13)

2 The zi are complex and distinct.

As before, a(B)xt =

⇔ xt = c1z−t

1 + c2z

−t

2 + · · · + ckz

−t

k

If z1 6∈ R, since a1, , ak are real, we must have the complex conjugate

root, zj = ¯z1 And for xt to be real, we must have cj = ¯c1 For example:

xt = c z−t

1 + ¯c z¯1

−t

= r eiθ|z1|

−t

e−iωt

+ r e−iθ

|z1|

−t

eiωt = r|z1|−t

ei(θ−ωt)

+ e−i(θ−ωt) = 2r|z1|−t

(14)

2 The zi are complex and distinct e.g., z1 = 1.2 + i, z2 = 1.2 − i

0 10 12 14 16 18 20 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 c

1 z1 −t

+ c

2 z2 −t

(15)

2 The zi are complex and distinct e.g., z1 = + 0.1i, z2 = − 0.1i

0 10 20 30 40 50 60 70 −2

−1.5 −1 −0.5 0.5 1.5

c

1 z1 −t

+ c

2 z2 −t

(16)

3 Some zi are repeated.

a(B)xt =

⇔ (B − z1)(B − z2) · · ·(B − zk)xt =

If z1 = z2, (B − z1)(B − z2)xt =

⇔ (B − z1)2xt =

We can check that (c1 + c2t)z−t

1 is a solution

More generally, (B − z1)mxt = has the solution c1 + c2t + · · · + cm−1tm

−1 z−t

(17)

3 Some zi are repeated e.g., z1 = z2 = 1.5

0 10 12 14 16 18 20 −1

−0.5 0.5 1.5

(c

1 + c2 t) z1 −t

c

1=1, c2=0

c

1=0, c2=2

c

(18)

Solving linear diff eqns with constant coefficients

Solve: a0xt + a1xt−1 + · · · + akxt−k = 0, with initial conditions x1, , xk

Auxiliary equation in z ∈ C: a0 + a1z + · · · + akzk = 0 ⇔ (z − z1)m1(z − z2)m2 · · · (z − zl)ml = 0,

where z1, z2, , zl ∈ C are the roots of the characteristic polynomial, and

zi occurs with multiplicity mi

Solutions: c1(t)z

−t

1 + c2(t)z

−t

2 + · · · + cl(t)z

−t

l ,

where ci(t) is a polynomial in t of degree mi −

(19)

Autocovariance functions of ARMA processes: Example

(1 + 0.25B2)Xt = (1 + 0.2B)Wt, ⇔ Xt = ψ(B)Wt,

ψj =

1, 5,−

1 4,−

1 20,

1 16,

1 80,−

1 64,−

1

320,

γ(h) − φ1γ(h − 1) − φ2γ(h − 2) = σw2

q−h

X

j=0

θh+jψj

⇔γ(h) + 0.25γ(h − 2) =

      

σw2 (ψ0 + 0.2ψ1) if h = 0, 0.2σw2 ψ0 if h = 1,

(20)

We have the homogeneous linear difference equation γ(h) + 0.25γ(h − 2) = for h ≥ 2, with initial conditions

(21)

Homogeneous lin diff eqn:

γ(h) + 0.25γ(h − 2) = The characteristic polynomial is

1 + 0.25z2 =

4 + z

2

=

4(z − 2i)(z + 2i), which has roots at z1 = 2eiπ/2,z¯1 = 2e−iπ/2

The solution is of the form

γ(h) = cz−h

1 + ¯cz¯1

(22)

z1 = 2eiπ/2,z¯1 = 2e−iπ/2

, c = |c|eiθ We have γ(h) = cz−h

1 + ¯cz¯1

−h

= 2−h

|c|ei(θ−hπ/2)

+ |c|ei(−θ+hπ/2)

= c12−h cos

hπ

2 − θ

And we determine c1, θ from the initial conditions

(23)

We determine c1, θ from the initial conditions:

We plug γ(0) = c1 cos(θ)

γ(1) = c1

2 sin(θ) γ(2) = −c1

4 cos(θ) into γ(0) + 0.25γ(2) = σw2 (1 + 1/25)

(24)

−10 −8 −6 −4 −2 10 −0.4

−0.2 0.2 0.4 0.6 0.8 1.2

(25)

2 Review: Autocovariance of an ARMA process Homogeneous linear difference equations

Forecasting

1 Linear prediction

(26)

Review: least squares linear prediction

Consider a linear predictor of Xn+h given Xn = xn:

f(xn) = α0 + α1xn

For a stationary time series {Xt}, the best linear predictor is f∗

(xn) = (1 − ρ(h))µ + ρ(h)xn:

E(Xn+h − (α0 + α1Xn))2 ≥ E(Xn+h − f∗

(27)

Linear prediction

Given X1, X2, , Xn, the best linear predictor Xnn+m = α0 +

n

X

i=1

αiXi

of Xn+m satisfies the prediction equations E Xn+m − Xnn+m =

E Xn+m − Xnn+mXi = for i = 1, , n

(28)

Projection Theorem

If H is a Hilbert space,

M is a closed linear subspace of H, and y ∈ H,

then there is a point P y ∈ M

(the projection of y on M) satisfying

1 kP y − yk ≤ kw − yk for w ∈ M,

2 kP y −yk < kw−yk for w ∈ M, w 6= y hy − P y, wi = for w ∈ M

y

y−Py

Py

(29)

Hilbert spaces

Hilbert space = complete inner product space:

Inner product space: vector space, with inner product ha, bi:

• ha, bi = hb, ai,

• hα1a1 + α2a2, bi = α1ha1, bi + α2ha2, bi, • ha, = ⇔ a =

Norm: kak2 = ha,

complete = limits of Cauchy sequences are in the space

Examples:

1 Rn, with Euclidean inner product, hx, yi = P

i xiyi

2 {random variables X: EX2 < ∞},

(30)

Projection theorem

Example: Linear regression

Given y = (y1, y2, , yn)′ ∈ Rn, and Z = (z1, , zq) ∈ Rn×q , choose β = (β1, , βq)′

∈ Rq to minimize ky − Zβk2.

Here, H = Rn, with ha, bi = P

i aibi, and

(31)

M is a closed subspace of H, and y ∈ H,

1 kP y − yk ≤ kw − yk

2 hy − P y, wi = for w ∈ M

y

y−Py

Py

(32)

y

y−Py

Py

M

hy − P y, wi =

⇔ hy − Zβ, zˆ ii = 0, ∀i

⇔ Z′

Zβˆ = Z′ y

⇔ βˆ = (Z′

Z)−1 Z′

(33)

Example: Linear prediction

Given 1, X1, X2, , Xn ∈ r.v.s X : EX2 < ∞ , choose α0, α1, , αn ∈ R

so that Z = α0 + Pn

i=1 αiXi minimizes E(Xn+m − Z)2

Here, hX, Y i = E(XY ),

M = {Z = α0 + Pn

i=1 αiXi : αi ∈ R} = ¯sp{1, X1, , Xn}, and

(34)

M is a closed subspace of H, and y ∈ H,

1 kP y − yk ≤ kw − yk

2 hy − P y, wi = for w ∈ M

y

y−Py

Py

(35)

Projection theorem: Linear prediction

Let Xnn+m denote the best linear predictor:

kXnn+m − Xn+mk2 ≤ kZ − Xn+mk2 for all Z ∈ M The projection theorem implies the orthogonality

hXnn+m − Xn+m, Zi = for all Z ∈ M

⇔ hXnn+m − Xn+m, Zi = for all Z ∈ {1, X1, , Xn}

⇔ E X

n

n+m − Xn+m

= E Xnn+m − Xn+mXi =

That is, the prediction errors (Xnn+m − Xn+m) are uncorrelated with the

(36)

Error (Xnn+m − Xn+m) is uncorrelated with the prediction variable 1: E Xnn+m − Xn+m

=

⇔ E α0 +

X

i

αiXi − Xn+m

!

=

⇔ µ − X

i

αi

!

(37)

µ − X

i

αi

!

= α0

Substituting for α0 in

Xnn+m = α0 + X

i

αiXi,

we get Xnn+m = µ + X

i

αi (Xi − µ)

So we can subtract µ from all variables: Xnn+m − µ = X

i

αi (Xi − µ)

(38)

One-step-ahead linear prediction

Write Xnn+1 = φn1Xn + φn2Xn−1+ · · · + φnnX1

Prediction equations: E (Xnn+1 − Xn+1)Xi = 0, for i = 1, , n

⇔

n

X

j=1

φnjE (Xn+1−jXi) = E(Xn+1Xi)

⇔

n

X

j=1

φnjγ(i − j) = γ(i)

(39)

One-step-ahead linear prediction

Prediction equations: Γnφn = γn

Γn =

       

γ(0) γ(1) · · · γ(n − 1) γ(1) γ(0) γ(n − 2)

γ(n − 1) γ(n − 2) · · · γ(0)

        ,

φn = (φn1, φn2, , φnn)

′

, γn = (γ(1), γ(2), , γ(n))

(40)

Mean squared error of one-step-ahead linear prediction

Pnn+1 = E Xn+1 − Xnn+12

= E Xn+1 − Xnn+1 Xn+1 − Xnn+1

= E Xn+1 Xn+1 − Xnn+1 = γ(0) − E(φ′

nXXn+1)

= γ(0) − γ′

nΓ

−1

n γn,

where X = (Xn, Xn−1, , X1) ′

(41)

Mean squared error of one-step-ahead linear prediction

Variance is reduced:

Pnn+1 = E Xn+1 − Xnn+12 = γ(0) − γ′

nΓ

−1

n γn

= Var(Xn+1) − Cov(Xn+1, X)Cov(X, X)−1

Cov(X, Xn+1) = E (Xn+1 − 0)2 − Cov(Xn+1, X)Cov(X, X)−1

Cov(X, Xn+1), where X = (Xn, Xn−1, , X1)

(42)

Peter Bartlett

1 Review: ARMA(p,q) models and their properties Review: Autocovariance of an ARMA process Homogeneous linear difference equations

Forecasting

1 Linear prediction