Homogeneous linear diff eqns with constant coefficients.. 1..[r]
(1)Introduction to Time Series Analysis Lecture 7.
Peter Bartlett
Last lecture:
1 ARMA(p,q) models: stationarity, causality, invertibility The linear process representation of ARMA processes: ψ Autocovariance of an ARMA process
(2)Introduction to Time Series Analysis Lecture 7.
Peter Bartlett
1 Review: ARMA(p,q) models and their properties Review: Autocovariance of an ARMA process Homogeneous linear difference equations
Forecasting
1 Linear prediction
(3)Review: Autoregressive moving average models
An ARMA(p,q) process {Xt} is a stationary process that satisfies
φ(B)Xt = θ(B)Wt,
where φ, θ are degree p, q polynomials and {Wt} ∼
W N(0, σ2)
(4)Review: Properties of ARMA(p,q) models
Theorem: If φ and θ have no common factors, a (unique)
sta-tionary solution {Xt} to φ(B)Xt = θ(B)Wt
exists iff
φ(z) = − φ1z − · · · − φpzp = ⇒ |z| 6=
This ARMA(p,q) process is causal iff
φ(z) = − φ1z − · · · − φpzp = ⇒ |z| > It is invertible iff
(5)Review: Properties of ARMA(p,q) models
φ(B)Xt = θ(B)Wt, ⇔ Xt = ψ(B)Wt
so θ(B) = ψ(B)φ(B)
⇔
⇔ = ψ0,
θ1 = ψ1 − φ1ψ0,
θ2 = ψ2 − φ1ψ1 − · · · − φ2ψ0,
(6)
Review: Autocovariance functions of ARMA processes
φ(B)Xt = θ(B)Wt, ⇔ Xt = ψ(B)Wt,
γ(h) − φ1γ(h − 1) − φ(2)γ(h − 2) − · · · = σw2
q
X
j=h
θjψj−h
We need to solve the homogeneous linear difference equation φ(B)γ(h) = (h > q), with initial conditions
γ(h)−φ1γ(h−1)− · · · −φpγ(h−p) = σw2 q
X
j=h
(7)Introduction to Time Series Analysis Lecture 7. Review: ARMA(p,q) models and their properties
2 Review: Autocovariance of an ARMA process Homogeneous linear difference equations
Forecasting
1 Linear prediction
(8)Homogeneous linear diff eqns with constant coefficients
a0xt + a1xt−1 + · · · + akxt−k =
⇔ a0 + a1B + · · · + akBkxt =
⇔ a(B)xt = auxiliary equation: a0 + a1z + · · · + akzk =
⇔ (z − z1)(z − z2)· · · (z − zk) =
where z1, z2, , zk ∈ C are the roots of this characteristic polynomial.
Thus,
(9)Homogeneous linear diff eqns with constant coefficients
a(B)xt = ⇔ (B − z1)(B − z2)· · ·(B − zk)xt =
So any {xt} satisfying (B −zi)xt = for some i also satisfies a(B)xt = Three cases:
1 The zi are real and distinct
(10)Homogeneous linear diff eqns with constant coefficients
1 The zi are real and distinct.
a(B)xt =
⇔ (B − z1)(B − z2) · · ·(B − zk)xt =
⇔ xt is a linear combination of solutions to
(B − z1)xt = 0, (B − z2)xt = 0, , (B − zk)xt =
⇔ xt = c1z
−t
1 + c2z
−t
2 + · · · + ckz
−t
k ,
(11)Homogeneous linear diff eqns with constant coefficients
1 The zi are real and distinct e.g., z1 = 1.2, z2 = −1.3
0 10 12 14 16 18 20 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 c
1 z1 −t
+ c
2 z2 −t
c
1=1, c2=0
c
1=0, c2=1
c
(12)Reminder: Complex exponentials
a + ib = reiθ = r(cosθ + isinθ), where r = |a + ib| = pa2 + b2
θ = tan−1
b a
∈ (−π, π]
Thus, r1eiθ1r
2eiθ2 = (r1r2)ei(θ1+θ2),
(13)Homogeneous linear diff eqns with constant coefficients
2 The zi are complex and distinct.
As before, a(B)xt =
⇔ xt = c1z−t
1 + c2z
−t
2 + · · · + ckz
−t
k
If z1 6∈ R, since a1, , ak are real, we must have the complex conjugate
root, zj = ¯z1 And for xt to be real, we must have cj = ¯c1 For example:
xt = c z−t
1 + ¯c z¯1
−t
= r eiθ|z1|
−t
e−iωt
+ r e−iθ
|z1|
−t
eiωt = r|z1|−t
ei(θ−ωt)
+ e−i(θ−ωt) = 2r|z1|−t
(14)Homogeneous linear diff eqns with constant coefficients
2 The zi are complex and distinct e.g., z1 = 1.2 + i, z2 = 1.2 − i
0 10 12 14 16 18 20 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 c
1 z1 −t
+ c
2 z2 −t
(15)Homogeneous linear diff eqns with constant coefficients
2 The zi are complex and distinct e.g., z1 = + 0.1i, z2 = − 0.1i
0 10 20 30 40 50 60 70 −2
−1.5 −1 −0.5 0.5 1.5
c
1 z1 −t
+ c
2 z2 −t
(16)Homogeneous linear diff eqns with constant coefficients
3 Some zi are repeated.
a(B)xt =
⇔ (B − z1)(B − z2) · · ·(B − zk)xt =
If z1 = z2, (B − z1)(B − z2)xt =
⇔ (B − z1)2xt =
We can check that (c1 + c2t)z−t
1 is a solution
More generally, (B − z1)mxt = has the solution c1 + c2t + · · · + cm−1tm
−1 z−t
(17)Homogeneous linear diff eqns with constant coefficients
3 Some zi are repeated e.g., z1 = z2 = 1.5
0 10 12 14 16 18 20 −1
−0.5 0.5 1.5
(c
1 + c2 t) z1 −t
c
1=1, c2=0
c
1=0, c2=2
c
(18)Solving linear diff eqns with constant coefficients
Solve: a0xt + a1xt−1 + · · · + akxt−k = 0, with initial conditions x1, , xk
Auxiliary equation in z ∈ C: a0 + a1z + · · · + akzk = 0 ⇔ (z − z1)m1(z − z2)m2 · · · (z − zl)ml = 0,
where z1, z2, , zl ∈ C are the roots of the characteristic polynomial, and
zi occurs with multiplicity mi
Solutions: c1(t)z
−t
1 + c2(t)z
−t
2 + · · · + cl(t)z
−t
l ,
where ci(t) is a polynomial in t of degree mi −
(19)Autocovariance functions of ARMA processes: Example
(1 + 0.25B2)Xt = (1 + 0.2B)Wt, ⇔ Xt = ψ(B)Wt,
ψj =
1, 5,−
1 4,−
1 20,
1 16,
1 80,−
1 64,−
1
320,
γ(h) − φ1γ(h − 1) − φ2γ(h − 2) = σw2
q−h
X
j=0
θh+jψj
⇔γ(h) + 0.25γ(h − 2) =
σw2 (ψ0 + 0.2ψ1) if h = 0, 0.2σw2 ψ0 if h = 1,
(20)Autocovariance functions of ARMA processes: Example
We have the homogeneous linear difference equation γ(h) + 0.25γ(h − 2) = for h ≥ 2, with initial conditions
(21)Autocovariance functions of ARMA processes: Example
Homogeneous lin diff eqn:
γ(h) + 0.25γ(h − 2) = The characteristic polynomial is
1 + 0.25z2 =
4 + z
2
=
4(z − 2i)(z + 2i), which has roots at z1 = 2eiπ/2,z¯1 = 2e−iπ/2
The solution is of the form
γ(h) = cz−h
1 + ¯cz¯1
(22)Autocovariance functions of ARMA processes: Example
z1 = 2eiπ/2,z¯1 = 2e−iπ/2
, c = |c|eiθ We have γ(h) = cz−h
1 + ¯cz¯1
−h
= 2−h
|c|ei(θ−hπ/2)
+ |c|ei(−θ+hπ/2)
= c12−h cos
hπ
2 − θ
And we determine c1, θ from the initial conditions
(23)Autocovariance functions of ARMA processes: Example
We determine c1, θ from the initial conditions:
We plug γ(0) = c1 cos(θ)
γ(1) = c1
2 sin(θ) γ(2) = −c1
4 cos(θ) into γ(0) + 0.25γ(2) = σw2 (1 + 1/25)
(24)Autocovariance functions of ARMA processes: Example
−10 −8 −6 −4 −2 10 −0.4
−0.2 0.2 0.4 0.6 0.8 1.2
(25)Introduction to Time Series Analysis Lecture 7. Review: ARMA(p,q) models and their properties
2 Review: Autocovariance of an ARMA process Homogeneous linear difference equations
Forecasting
1 Linear prediction
(26)Review: least squares linear prediction
Consider a linear predictor of Xn+h given Xn = xn:
f(xn) = α0 + α1xn
For a stationary time series {Xt}, the best linear predictor is f∗
(xn) = (1 − ρ(h))µ + ρ(h)xn:
E(Xn+h − (α0 + α1Xn))2 ≥ E(Xn+h − f∗
(27)Linear prediction
Given X1, X2, , Xn, the best linear predictor Xnn+m = α0 +
n
X
i=1
αiXi
of Xn+m satisfies the prediction equations E Xn+m − Xnn+m =
E Xn+m − Xnn+mXi = for i = 1, , n
(28)Projection Theorem
If H is a Hilbert space,
M is a closed linear subspace of H, and y ∈ H,
then there is a point P y ∈ M
(the projection of y on M) satisfying
1 kP y − yk ≤ kw − yk for w ∈ M,
2 kP y −yk < kw−yk for w ∈ M, w 6= y hy − P y, wi = for w ∈ M
y
y−Py
Py
(29)Hilbert spaces
Hilbert space = complete inner product space:
Inner product space: vector space, with inner product ha, bi:
• ha, bi = hb, ai,
• hα1a1 + α2a2, bi = α1ha1, bi + α2ha2, bi, • ha, = ⇔ a =
Norm: kak2 = ha,
complete = limits of Cauchy sequences are in the space
Examples:
1 Rn, with Euclidean inner product, hx, yi = P
i xiyi
2 {random variables X: EX2 < ∞},
(30)Projection theorem
Example: Linear regression
Given y = (y1, y2, , yn)′ ∈ Rn, and Z = (z1, , zq) ∈ Rn×q , choose β = (β1, , βq)′
∈ Rq to minimize ky − Zβk2.
Here, H = Rn, with ha, bi = P
i aibi, and
(31)Projection theorem
If H is a Hilbert space,
M is a closed subspace of H, and y ∈ H,
then there is a point P y ∈ M
(the projection of y on M) satisfying
1 kP y − yk ≤ kw − yk
2 hy − P y, wi = for w ∈ M
y
y−Py
Py
(32)Projection theorem
y
y−Py
Py
M
hy − P y, wi =
⇔ hy − Zβ, zˆ ii = 0, ∀i
⇔ Z′
Zβˆ = Z′ y
⇔ βˆ = (Z′
Z)−1 Z′
(33)Projection theorem
Example: Linear prediction
Given 1, X1, X2, , Xn ∈ r.v.s X : EX2 < ∞ , choose α0, α1, , αn ∈ R
so that Z = α0 + Pn
i=1 αiXi minimizes E(Xn+m − Z)2
Here, hX, Y i = E(XY ),
M = {Z = α0 + Pn
i=1 αiXi : αi ∈ R} = ¯sp{1, X1, , Xn}, and
(34)Projection theorem
If H is a Hilbert space,
M is a closed subspace of H, and y ∈ H,
then there is a point P y ∈ M
(the projection of y on M) satisfying
1 kP y − yk ≤ kw − yk
2 hy − P y, wi = for w ∈ M
y
y−Py
Py
(35)Projection theorem: Linear prediction
Let Xnn+m denote the best linear predictor:
kXnn+m − Xn+mk2 ≤ kZ − Xn+mk2 for all Z ∈ M The projection theorem implies the orthogonality
hXnn+m − Xn+m, Zi = for all Z ∈ M
⇔ hXnn+m − Xn+m, Zi = for all Z ∈ {1, X1, , Xn}
⇔ E X
n
n+m − Xn+m
= E Xnn+m − Xn+mXi =
That is, the prediction errors (Xnn+m − Xn+m) are uncorrelated with the
(36)Linear prediction
Error (Xnn+m − Xn+m) is uncorrelated with the prediction variable 1: E Xnn+m − Xn+m
=
⇔ E α0 +
X
i
αiXi − Xn+m
!
=
⇔ µ − X
i
αi
!
(37)Linear prediction
µ − X
i
αi
!
= α0
Substituting for α0 in
Xnn+m = α0 + X
i
αiXi,
we get Xnn+m = µ + X
i
αi (Xi − µ)
So we can subtract µ from all variables: Xnn+m − µ = X
i
αi (Xi − µ)
(38)One-step-ahead linear prediction
Write Xnn+1 = φn1Xn + φn2Xn−1+ · · · + φnnX1
Prediction equations: E (Xnn+1 − Xn+1)Xi = 0, for i = 1, , n
⇔
n
X
j=1
φnjE (Xn+1−jXi) = E(Xn+1Xi)
⇔
n
X
j=1
φnjγ(i − j) = γ(i)
(39)One-step-ahead linear prediction
Prediction equations: Γnφn = γn
Γn =
γ(0) γ(1) · · · γ(n − 1) γ(1) γ(0) γ(n − 2)
γ(n − 1) γ(n − 2) · · · γ(0)
,
φn = (φn1, φn2, , φnn)
′
, γn = (γ(1), γ(2), , γ(n))
(40)Mean squared error of one-step-ahead linear prediction
Pnn+1 = E Xn+1 − Xnn+12
= E Xn+1 − Xnn+1 Xn+1 − Xnn+1
= E Xn+1 Xn+1 − Xnn+1 = γ(0) − E(φ′
nXXn+1)
= γ(0) − γ′
nΓ
−1
n γn,
where X = (Xn, Xn−1, , X1) ′
(41)Mean squared error of one-step-ahead linear prediction
Variance is reduced:
Pnn+1 = E Xn+1 − Xnn+12 = γ(0) − γ′
nΓ
−1
n γn
= Var(Xn+1) − Cov(Xn+1, X)Cov(X, X)−1
Cov(X, Xn+1) = E (Xn+1 − 0)2 − Cov(Xn+1, X)Cov(X, X)−1
Cov(X, Xn+1), where X = (Xn, Xn−1, , X1)
(42)Introduction to Time Series Analysis Lecture 7.
Peter Bartlett
1 Review: ARMA(p,q) models and their properties Review: Autocovariance of an ARMA process Homogeneous linear difference equations
Forecasting
1 Linear prediction