1 Backtesting and Stress Testing Elements of Financial Risk Management Chapter 13 Peter Christoffersen Elements of Financial Risk Management Second Edition â 2012 by Peter Christoffersen Overview The objective in this chapter is to consider the ex ante risk measure forecasts from the model and compare them with the ex post realized portfolio return • The risk measure forecast could take the form of a Value-atRisk (VaR), an Expected Shortfall (ES), the shape of the entire return distribution, or perhaps the shape of the left tail of the distribution only • We want to be able to backtest any of these risk measures of interest • The backtest procedures can be seen as a final diagnostic check on the aggregate risk model, thus complementing the other various specific diagnostics Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Overview • The material in the chapter will be covered as follows: • We take a brief look at the performance of some real-life VaRs from six large commercial banks • The clustering of VaR violations in these real-life VaRs provides sobering food for thought • We establish procedures for backtesting VaRs • We start by introducing a simple unconditional test for the average probability of a VaR violation • We then test the independence of the VaR violations • Finally, combine unconditional test and independence test in a test of correct conditional VaR coverage Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Overview • We consider using explanatory variables to backtest the VaR • This is done in a regression-based framework • We establish backtesting procedures for the Expected Shortfall measure • We broaden the focus to include the entire shape of the distribution of returns • The distributional forecasts can be backtested as well, and we suggest ways to so • Risk managers typically care most about having a good forecast of the left tail of the distribution Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Overview • We therefore modify the distribution test to focus on backtesting the left tail of the distribution only • We define stress testing and give a critical survey of the way it is often implemented • Based on this critique we suggest a coherent framework for stress testing • Figure 13.1 shows the performance of some real-life VaRs • Figure shows the exceedances of the VaR in six large U.S commercial banks during the January 1998 to March 2001 period Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Figure 13.1: Value-at-Risk Exceedences From Six Major Commercial Banks Elements of Financial Risk Management Second Edition â 2012 by Peter Christoffersen Overview Whenever the realized portfolio return is worse than the VaR, the difference between the two is shown • Whenever the return is better, zero is shown • The difference is divided by the standard deviation of the portfolio across the period • The return is daily, and the VaR is reported for a 1% coverage rate • To be exact, we plot the time series of Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Overview • Bank has no violations at all, and in general the banks have fewer violations than expected • Thus, the banks on average report a VaR that is higher than it should be • This could either be due to the banks deliberately wanting to be cautious or the VaR systems being biased • Another culprit is that the returns reported by the banks contain nontrading-related profits, which increase the average return without substantially increasing portfolio risk Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Overview • More important, notice the clustering of VaR violations • The violations for each of Banks 1, 2, 3, 5, and fall within a very short time span and often on adjacent days • This clustering of VaR violations is a serious sign of risk model misspecification • These banks are most likely relying on a technique such as Historical Simulation (HS), which is very slow at updating the VaR when market volatility increases • This issue was discussed in the context of the 1987 stock market crash in Chapter Elements of Financial Risk Management Second Edition â 2012 by Peter Christoffersen Overview 10 Notice also how the VaR violations tend to be clustered across banks • Many violations appear to be related to the Russia default and Long Term Capital Management bailout in the fall of 1998 • The clustering of violations across banks is important from a regulator perspective because it raises the possibility of a countrywide banking crisis • Motivated by the sobering evidence of misspecification in existing commercial bank VaRs, we now introduce a set of statistical techniques for backtesting risk management models Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Backtesting Only the Left Tail of the Distribution 60 • We have simply zoomed in on the leftmost 10% of the histogram from Figure 13.2 • The systematic deviation from a flat histogram is again obvious • To formal statistical testing, we can again construct an alternative hypothesis as in • for t+1 such that RPF,t+1 < -VaRpt+1 Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Backtesting Only the Left Tail of the Distribution 61 • We can then calculate a likelihood ratio test • where nb again is the number of elements in the parameter vector b1 Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Stress Testing 62 • Due to the practical constraints from managing large portfolios, risk managers often work with relatively short data samples • This can be a serious issue if the historical data available not adequately reflect the potential risks going forward • The available data may, for example, lack extreme events such as an equity market crash, which occurs very infrequently Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Stress Testing 63 • To make up for the inadequacies of the available data, it can be useful to artificially generate extreme scenarios of main factors driving portfolio returns and then assess the resulting output from the risk model • This is referred to as stress testing, since we are stressing the model by exposing it to data different from the data used when specifying and estimating the model Elements of Financial Risk Management Second Edition â 2012 by Peter Christoffersen Stress Testing 64 At first pass, the idea of stress testing may seem vague and ad hoc • Two key issues appear to be – how should we interpret the output of the risk model from the stress scenarios, and – how should we create the scenarios in the first place? • We deal with each of these issues in turn Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Combining Distributions for Coherent Stress Testing 65 • VaR and ES are proper probabilistic statements: – What is the loss such that I will loose more only 1% of the time (VaR)? – What is the expected loss when I violate my VaR (ES)? • Standard stress testing does not tell the portfolio manager anything about the probability of the scenario happening, and it is therefore not at all clear what the portfolio rebalancing decision should be Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Combining Distributions for Coherent Stress Testing 66 • Once scenario probabilities are assigned, then stress testing can be very useful • To be explicit, consider a simple example of one stress scenario, which we define as a probability distribution fstress() of the vector of factor returns • We simulate a vector of risk factor returns from the risk model, calling it f (), and we also simulate from the scenario distribution, fstress() Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Combining Distributions for Coherent Stress Testing 67 • If we assign a probability of a draw from the scenario distribution occurring, then we can combine the two distributions as in • Data from the combined distribution is generated by drawing a random variable Ui from a Uniform(0,1) distribution • If Ui is smaller than , then we draw a return from fstress(); otherwise we draw it from f() Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Combining Distributions for Coherent Stress Testing 68 • Once we have simulated data from the combined data set, we can calculate the VaR or ES risk measure on the combined data • If the risk measure is viewed to be inappropriately high then the portfolio can be rebalanced • Assigning the probability, , also allows the risk manager to backtest the VaR system using the combined probability distribution fcomb() • Any of these tests can be used to test the risk model using the data drawn from fcomb() Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Choosing Scenarios 69 • Having decided to stress testing, a key challenge to the risk manager is to create relevant scenarios • The risk manager ought to the following: • Simulate shocks which are more likely to occur than the historical data base suggests • Simulate shocks that have never occurred but could • Simulate shocks reflecting the possibility that current statistical patterns could break down • Simulate shocks which reflect structural breaks which could occur Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Choosing Scenarios 70 • While largely portfolio specific, the long and colorful history of financial crises may give inspiration for scenario generation • Scenarios could include crises set off by political events or natural disasters • Scenarios could be the culmination of pressures such as a continuing real appreciation building over time resulting in a loss of international competitiveness • The effects of market crises can also be very different • They can result in relatively brief market corrections or they can have longer lasting effects Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Figure 13.4: The Fifteen Largest One-day Percentage Declines on the Dow Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen 71 Stress Testing the Term Structure of Risk 72 • The Filtered Historical Simulation (or bootstrapping) method to construct the term structure of risk can be used to stress test the term structure of risk as well • Rather than feeding randomly drawn shocks through the model over time we can feed a path of historical shocks from a stress scenario through the model • The stress scenario can for example be the string of daily shocks observed from September 2008 through March 2009 • The outcome of this simulation will show how a stressed market scenario will affect the portfolio under consideration Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Figure 13.5: Bear Market Episodes in the Dow Jones Index Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen 73 Summary • Real life VaRs • Backtesting VaR Unconditional and conditional approaches • A Regression-based Approach • Backtesting Expected Shortfall • Backtesting Distributions and Distribution tails • A Coherent Approach to Stress Testing • Stress Testing the Term Structure of Risk Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen 74 ... sobering evidence of misspecification in existing commercial bank VaRs, we now introduce a set of statistical techniques for backtesting risk management models Elements of Financial Risk Management. .. sequence of random tosses of a coin, which comes up heads 1% or 5% of the time depending on the VaR coverage rate Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen... 2012 by Peter Christoffersen Figure 13.1: Value-at -Risk Exceedences From Six Major Commercial Banks Elements of Financial Risk Management Second Edition © 2012 by Peter Christoffersen Overview