Brennan et al. Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models

1Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 2Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 4Alan Brennan, MSc(a) 5Samer Kharroubi, PhD(b) 6Anthony O’Hagan, PhD(b) 7Jim Chilcott, MSc(a) 9(a) School of Health and Related Research, The University of Sheffield, Regent Court, Sheffield S1 104DA, England 11(b) Department of Probability and Statistics, The University of Sheffield, Hounsfield Road, Sheffield S3 127RH, England 13 14Reprint requests to: 15Alan Brennan, MSc 16School of Health and Related Research, 17The University of Sheffield, 18Regent Court, 19Sheffield S1 4DA, 20England 21 a.brennan@sheffield.ac.uk 3Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 22ABSTRACT 23 24Partial EVPI calculations can quantify the value of learning about particular subsets of uncertain 25parameters in decision models Published case studies have used different computational approaches 26This paper aims to clarify computation of partial EVPI and encourage its use Our mathematical 27description defining partial EVPI shows two nested expectations, which must be evaluated separately 28because of the need to compute a maximum between them We set out a generalised Monte Carlo 29sampling algorithm using two nested simulation loops, firstly an outer loop to sample parameters of 30interest and only then an inner loop to sample the remaining uncertain parameters, given the sampled 31parameters of interest Alternative computation methods and ‘shortcut’ algorithms are assessed and 32mathematical conditions for their use are considered Maxima of Monte Carlo estimates of expectations 33are always biased upwards, and we demonstrate the effect of small samples on bias in computing partial 34EVPI A series of case studies demonstrates the accuracy or otherwise of using ‘short-cut’ algorithm 35approximations in simple decision trees, models with increasingly correlated parameters, and many 36period Markov models with increasing non-linearity The results show that even if relatively small 37correlation or non-linearity is present, then the ‘shortcut’ algorithm can be substantially inaccurate The 38case studies also suggest that fewer samples on the outer level and larger numbers of samples on the 39inner level could be the most efficient approach to gaining accurate partial EVPI estimates Remaining 40areas for methodological development are set out 41 42 43 5Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 44Acknowledgements: 45 46The authors are members of CHEBS: The Centre for Bayesian Statistics in Health Economics, 47University of Sheffield Thanks go to Karl Claxton and Tony Ades who helped our thinking at a CHEBS 48“focus fortnight” event, to Gordon Hazen, Doug Coyle, Maria Hunink and others for feedback on the 49poster at SMDM Finally, thanks to the UK National Coordinating Centre for Health Technology 50Assessment which originally commissioned two of the authors to review the role of modelling methods 51in the prioritisation of clinical trials (Grant: 96/50/02) 52 7Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 53INTRODUCTION 54 55Quantifying expected value of perfect information (EVPI) is important for developers and users of 56decision models Many guidelines for cost-effectiveness analysis now recommend probabilistic 57sensitivity analysis (PSA)1,2 and EVPI is seen as a natural and coherent methodological extension 3,4 58Partial EVPI calculations are used to quantify uncertainty, identify key uncertain parameters, and inform 59the planning and prioritising of future research Some of the few published EVPI case studies have 60used slightly different computational approaches6 and many analysts, who confidently undertake PSA to 61calculate cost-effectiveness acceptability curves, still not use EVPI The aim of this paper is to 62clarify the computation of partial EVPI and encourage its use in health economic decision models 63 64The basic concepts of EVPI concern decisions on policy options under uncertainty Decision theory 65shows that a decision maker’s ‘adoption decision’ should be the policy with the greatest expected pay-off 66given current information7 In healthcare, we use monetary valuation of health (λ) to calculate a single 67payoff synthesising health and cost consequences e.g expected net benefit E(NB) = λ * E(QALYs) – 68E(Costs) In general, expected value of information (EVI) is a Bayesian approach that works by taking 69current knowledge (a prior probability distribution), adding in proposed information to be collected 70(data) and producing a posterior (synthesised probability distribution) based on all available information 71The value of the additional information is the difference between the expected payoff that would be 72achieved under posterior knowledge and the expected payoff under current (prior) knowledge On the 73basis of current information, this difference is uncertain (because the data are uncertain), so EVI is 74defined to be the expectation of the value of the information with respect to the uncertainty in the 75proposed data In defining EVPI, ‘Perfect’ information means perfectly accurate knowledge, or absolute 76certainty, about the values of some or all of the unknown parameters This can be thought of as 77obtaining an infinite sample size, producing a posterior probability distribution that is a single point, or 78alternatively, as ‘clairvoyance’ – suddenly learning the true values of the parameters Perfect 79information on all parameters implies no uncertainty about the optimal adoption decision For some 9Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 80values of the parameters the adoption decision would be revised, for others we would stick with our 81baseline adoption decision policy By investigating the pay-offs associated with different possible 82parameter values, and averaging these results, the ‘expected’ value of perfect information is quantified 83 84The expected value of obtaining perfect information on all the uncertain parameters gives ‘overall 85EVPI’, whereas ‘Partial EVPI’ is the expected value of learning the true value(s) of an individual 86parameter or of a subset of the parameters For example, we might compute the expected value of 87perfect information on efficacy parameters whilst other parameters, such as those concerned with costs, 88remain uncertain Calculations are often done per patient, and then multiplied by the number of patients 89affected over the lifetime of the decision to quantify ‘population (overall or partial) EVPI’ 90 91The limited health-based literature reveals several methods, which have been used to compute 92EVPIError: Reference source not found Early literature9,10 used stylised decision problems and 93simplifying assumptions, such as normally distributed net benefit, to calculate overall EVPI analytically 94via standard ‘unit normal loss integral’ statistical tables 11, but gave no analytic calculation method for 95partial EVPI In 1998, Felli and HazenError: Reference source not found gave a fuller exposition of 96EVPI method, setting out some mathematics using expected value notation, with a suggested general 97Monte Carlo random sampling procedure (‘MC1’) for partial EVPI calculation This procedure 98appeared to suggest that only the parameters of interest (ξI) need to be sampled but, following 99discussions with the authors of this paper, this was recently corrected 12 (both ξI and ξIC sampled), to show 100mathematical notation with nested expectations Felli and Hazen also provided a ‘shortcut’ simulation 101procedure (‘MC2’), for use when all parameters are assumed probabilistically independent and the 102payoff function is ‘multi-linear’ In the late 1990s, some UK case studies employed a different level 103algorithm to compute partial EVPI13,14,15, analysing the “expected opportunity loss remaining” if perfect 104information were obtained on a subset of parameters 105 10 11Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 106Other recent papers discuss the general value of partial EVPI, comparing either with alternative 107‘importance’ measures for sensitivity analysis16,17,18,19, or with ‘payback’ methods for prioritising 108researchError: Reference source not found, concluding that partial EVPI is the most logical, coherent 109approach without discussing the EVPI calculation methods required Very few studies examine the 110number of simulations required, and Coyle uses quadrature (taking samples at particular percentiles of 111the distribution) rather than random Monte Carlo sampling to speed up the calculation of partial EVPI 112for a single parameterError: Reference source not found Separate literature examines the case when the 113information gathering itself is the intervention of interest e.g a diagnostic test or screening strategy that 114gathers information to inform decision making concerning an individual patient 20,21 Here, the value of 115perfect information is typically the net benefit given ‘clairvoyance’ as to the true disease state of an 116individual patient EVPI methods in risk analysis literature were also recently reviewed 22 In 1999, 117building upon previous work by Gould23, Hilton24, Howard25 and Hammitt26, a case study by Hammitt 118and Shlyakhter27 set out similar mathematics to Felli and Hazen and examined the use of elicitation 119methods to quantify prior probability distributions if data were sparse 120 121Since first presenting our mathematics and algorithm Error: Reference source not found,28 a small 122number of case studies have been developed For the UK National Institute for Clinical Excellence and 123NCCHTA, Claxton et al present six such case studies 29 In Canada, Coyle at al have used a similar 124approach for the treatment of severe sepsis30 Development of the approach to calculate expected value 125of sample information (EVSI) is also ongoing31,32,33 Recent case studies include analysis of pharmaco126genetic tests in rheumatoid arthritis34 Partial EVPI of course represents an upper bound on the expected 127value of sample information for data collection on a parameter subset 128 129The objective of this paper is to examine the computation of partial EVPI We mathematically define 130partial EVPI using expected value notation, assess the alternative computation methods and algorithms, 131investigate the mathematical conditions when the alternative computation approaches may be used, and 12 13Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 132use case studies to demonstrate the accuracy or otherwise of ‘short-cut’ algorithm approximations 133Because a general level Monte-Carlo algorithm is relatively computationally intensive, we also assess 134whether relatively small numbers of iterations are inherently biased and investigate the number of 135iterations required to ensure accuracy Our overall aim is to encourage the use of partial EVPI 136calculation in health economic decision models 137 138MATHEMATICAL FORMULATION 139 140Overall EVPI Mathematics 141 142Let, 143θ be the vector of parameters in the model Since the components of θ are uncertain, 144 they have a joint probability distribution 145d denote an option out of the set of possible decisions; typically, d is the decision to adopt 146 or reimburse one treatment in preference to the others 147NB(d,θ) be the net benefit function for decision d for parameters values θ 148Overall EVPI is the value of finding out the true value of θ If we are not able to learn the value of θ, and 149must instead make a decision now, then we would evaluate each strategy in turn and choose the baseline 150adoption decision with the maximum expected net benefit Denoting this by ENB0, we have 151Expected net benefit | no additional information, [ Eθ { NB(d,θ )} ] ENB0 = max d (1) 152Notice that Eθ denotes an expectation over the full joint distribution of θ 153 154Now consider the situation where we might conduct some experiment or gain clairvoyance to learn the 155true values of the full vector of model parameters θ Then, since we now know everything, we can { NB(d,θ true )} This naturally 156choose with certainty the decision that maximises net benefit i.e max d 14 15Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 157depends on θtrue, which is unknown before the experiment, but we can consider the expectation of this 158net benefit by integrating over the uncertain θ [ { NB(d,θ )} = Eθ max d 159Expected net benefit | perfect information ] (2) 160The overall EVPI is the difference between these two (2)-(1), [ ] { NB(d,θ )} − max [ Eθ { NB(d,θ )} ] = Eθ max d d 161EVPI (3) 162It can be shown that this is always positive 163 164Partial EVPI Mathematics 165 166Now suppose that θ is divided into two subsets, θi and its complement θc, and we wish to know the 167expected value of perfect information about θi If we have to make decision now, then the expected net 168benefit is ENB0 again, but now consider the situation where we have conducted some experiment to 169learn the true values of the components of θi Now θc is still uncertain, and that uncertainty is described 170by its conditional distribution, conditional on the value of θi So we would now make the decision that 171maximises the expectation of net benefit over that distribution This is therefore NB(θi) = [ ] Eθ ** θ * { NB(d,θ )} , whose value is again unknown prior to the experiment because it depends on θi 172 max d 173Taking the expectation with respect to the distribution of θi therefore provides the relevant expected net 174benefit, ( [ EθC θI { NB(d,θ )} = EθI max d 175Expected Net benefit | perfect info only on θi ]) (4) 176and the partial EVPI for θi is the difference between (4) and ENB0, i.e 177PEVPI(θi) ( [ ]) EθC θI { NB(d,θ )} − max[ Eθ { NB(d,θ )} ] : = EθI max d d (5) 178This is necessarily positive and is also necessarily less than the overall EVPI 179 16 17Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 180The conditioning on θi in the inner expectation is significant In general, we expect that learning the 181true value of θi would also provide some information about θc Hence the correct distribution to use for 182the inner expectation is the conditional distribution that represents the remaining uncertainty in θc after 183learning θi The exception is when θi and θc are independent, allowing the unconditional (marginal) 184distribution of θc to be used in the inner expectation Although such independence is often assumed in 185economic model parameters (as we in Case Study 1), the assumption is rarely fully justified 186Equation (5) clearly shows two expectations The inner expectation evaluates the net benefit over the 187remaining uncertain parameters θc conditional on θi The outer evaluates the net benefit over the 188parameters of interest θi 189 190Residual EVPI 191 192Finally, we define the residual EVPI for θi by REVPI(θi) = EVPI – PEVPI(θc) 193REVPI(θi) [ ] ( [ { NB(d,θ )} − EθC max EθI θC { NB(d,θ )} = Eθ max d d ]) (6) 194This is a measure of the expected additional value of learning about θi, if we are already intending to 195learn about all the other parameters θc It is a measure of the residual uncertainty attributable to θi, if 196everything else were known From a decision maker’s perspective it might be interpreted as answering 197the question, ‘Can we afford not to know θi’? 198 199COMPUTATION 200 201Having explicitly set out the algebraic formulae for the different forms of EVPI, it is now possible to 202identify valid ways to compute them The key to the various approaches is how we evaluate 203expectations Notice that in (5) there are terms with two nested expectations, one with respect to the 204distribution of θi and the other with respect to the distribution of θc given θi Although this may seem to 18 19Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 205involve simply taking an expectation over all the components of θ, it is important that the two 206expectations are evaluated separately because of the need to compute a maximum between the two 207expectations It is this that makes the computation of partial EVPI complex 208 209Three techniques are commonly used in statistics to evaluate expectations 210(a) Analytic solution 211It may be possible to evaluate an expectation exactly using mathematics For instance, if X has a normal 212distribution with mean μ and variance σ2 then we can analytically evaluate various expectations such as 213E(X2) = μ2 + σ2 or E(exp(X)) = exp(μ + σ2/2) This is the ideal but is all too often not possible in practice 214For instance, if X has the normal distribution as above, there is no analytical closed-form expression for 215E((1 + X2)-1) 216(b) Quadrature 217Also known as numerical integration, quadrature is a computational technique to evaluate an integral 218Since expectations are formally integrals, quadrature is widely used to compute expectations It is 219particularly effective for low-dimensional integrals, and therefore for computing expectations with 220respect to the distribution of a small number of uncertain variables 221(c) Monte Carlo Sampling 222This is a very popular method, because it is very simple to implement in many situations To evaluate 223the expectation of some function f(X) of an uncertain quantity X, we randomly sample a large number, 224say N, of values from the probability distribution of X Denoting these by X1;X2,: : : ;XN, we then 225estimate E{f(X)} by the sample mean Eˆ { f ( X )} = N N ∑ f (X n) This estimate is unbiased and its n =1 226accuracy improves with increasing N Hence, given a large enough sample we can suppose that 227 Eˆ { f ( X )} is an essentially exact computation of E{f(X)} 228Each of these methods might be used to evaluate any of the expectations in EVPI calculations 20 10 61Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 715 716Box 1: General level Monte Carlo Algorithm for Calculation of Partial EVPI on a Parameter Subset of Interest 717 Preliminary Steps 0) Set up a decision model comparing different strategies and set up a decision rule e.g Cost per QALY is < λ 1) Characterise uncertain parameters with probability distributions e.g normal(μ,σ2), beta(a,b), gamma (a,b), triangular(a,b,c) … etc 2) Simulate (say 10,000) sample sets of uncertain parameter values (Monte Carlo) 3) Work out the baseline adoption decision given current information i.e the strategy giving (on average over the 10,000 simulations) the highest expected net benefit Partial EVPI for a parameter subset of interest The algorithm has nested loops 4) Simulate a perfect data collection exercise for your parameter subset of interest by: sampling each parameter of interest once from its prior uncertain range (outer level simulation) 5) calculate the best strategy given this new knowledge on the parameter of interest by - fixing the parameters of interest at their sampled values - simulating the other remaining uncertain parameters (say 10,000 times) allowing them to vary according to their conditional probability distribution (conditional upon the parameter of interest at its sampled value) (inner level simulation) - calculating the mean net benefit of each strategy - choosing the revised adoption decision to be the strategy which has the highest expected net benefit given the new data on the parameters of interest 6) Loop back to step and repeat steps and (say 10,000 times) and then calculate the average net benefit of the revised adoption decisions given perfect information on parameters of interest 7) The EVPI for the parameter of interest = average net benefit of revised adoption decisions given perfect information on parameters (6) minus average net benefit given current information i.e of the baseline adoption decision (3) 718Overall EVPI 719 720The algorithm for overall EVPI requires only loop 721 7228) For each of the 10,000 sampled sets of parameters from step (3) in turn, 723 - work out the optimal strategy given that particular sampled sets of parameters, 724 - record the net benefit of the optimal strategy 725 7269) With “perfect” information (i.e no uncertainty in the values of each parameter) we would always 727choose the optimal strategy 728Overall EVPI = 729 average net benefit of optimal adoption decisions given perfect information on all parameters (8) 730minus 731 average net benefit given current information i.e of the baseline adoption decision (3) 732 733 62 31 63Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 734 735Box 2: One level Monte Carlo Algorithm for Calculation of Partial EVPI on a Parameter Subset of Interest Preliminary Steps … As in Box One level Partial EVPI for a parameter subset of interest The algorithm has loop 4) Simulate a perfect data collection exercise for your parameter subset of interest by: sampling each parameter of interest once from its prior uncertain range (one level simulation) 5) calculate the best strategy given this new knowledge on the parameter of interest by - fixing the parameters of interest at their sampled values - fixing the remaining uncertain parameters of interest at their prior mean value - calculating the mean net benefit of each strategy given these parameter values - choosing the revised adoption decision to be the strategy which has the highest net benefit given the new data on the parameters of interest 6) Loop back to step and repeat steps and (say 10,000 times) and then calculate the average net benefit of the revised adoption decisions given perfect information on parameters of interest 7) The EVPI for the parameter of interest = average net benefit of revised adoption decisions given perfect information on parameters (6) minus average net benefit given current information i.e of the baseline adoption decision (3) 736 64 32 65Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 737 738Table 1: Illustration of Bias in Monte Carlo Estimates of Maxima of Several Expectations Estimate 10 11 12 13 14 15 ……… Mean of 10,000 estimates Analytic Mean Monte Carlo Estimates of Expected Net Benefit based on L=10 samples (‘000s) E(NB1) E(NB2) max{E(NB1),E(NB2)} 25.57 20.85 25.57 14.78 16.92 16.92 30.58 14.04 30.58 25.61 18.76 25.61 15.19 17.16 17.16 9.19 17.70 17.70 22.17 16.20 22.17 17.58 13.96 17.58 21.29 20.47 21.29 14.54 13.27 14.54 26.82 18.00 26.82 23.42 23.68 23.68 29.13 18.06 29.13 21.66 26.81 26.81 27.02 18.47 27.02 20.04 20 19.45 19.5 23.26 Bias 5.57 -3.08 10.58 5.61 -2.84 -2.30 2.17 -2.42 1.29 -5.46 6.82 3.68 9.13 6.81 7.02 3.26 739 740 66 33 67Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 741Table 2: Case Study Results Comparing the Algorithms  Partial EVPI Parameters Parameter No (Indexed - Overall EVPI = 100)  Utility Duration of Utility Duration of Change if Response Change if Response % Resp T0 Respond T0 T0 % Resp T1 Respond T1 T1 14 15 16 Independent Linear Cross-product model level 17 46 18 level 14 45 17 Opportunity Loss Level 10 24 10 Trial 5,14 Trial + Utility Only Utility 6,15 5,6,14,15 Overall EVPI Durations 7,16 All Case Study 16 14 12 Case Study (a) 0.1 (b) 0.2 (c) 0.3 (d) 0.6 (a) 0.1 (b) 0.2 (c) 0.3 (d) 0.6 Correlated Parameters Linear Cross-product model level 14 46 21 14 level 15 47 28 23 level 24 48 28 27 level 26 50 46 44 level 12 47 16 12 level 12 47 16 12 level 12 47 16 12 level 12 48 16 12 Case Study (a) N=3 (b) N=5 (c) N=10 (d) N=15 (e) N=20 (a) N=3 (b) N=5 (c) N=10 (d) N=15 (e) N=20 Non Linear Markov Models level level level level level level level level level level - - - 24 23 16 59 57 32 27 24 57 56 27 68 67 35 66 65 33 £ 1,352 £ 1,352 £ 1,352 19 21 22 23 20 20 20 21 56 55 70 85 56 56 57 58 26 41 47 55 20 19 17 19 58 59 68 71 52 50 47 53 60 74 81 91 62 58 55 62 65 72 76 93 62 62 64 64 £ £ £ £ £ £ £ £ 1,306 1,295 1,275 1,255 1,306 1,295 1,275 1,255 30 21 20 14 11 19 -13 -20 -26 51 69 69 44 65 64 47 25 15 79 84 58 46 55 69 52 28 17 68 76 59 82 82 54 67 75 80 81 £ £ £ £ £ £ £ £ £ £ 903 1,154 1,616 1,898 2,119 903 1,154 1,616 1,898 2,119 - - 742 68 34 69Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 743Matrix Figure 744 745 End State No Longer Responding Still Responding Start state Still Responding No Longer Responding Died 70 θ θ 20 0.60 23 0.00 0.00 θ θ 21 0.30 24 0.90 0.00 Died θ θ 22 0.10 25 0.10 1.00 35 71Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 746Figure 2: Impact of Increasing Correlation on Inaccuracy of level method to calculate partial EVPI Indexed Difference Case Study 2: level versus level Partial EVPI 40 35 30 25 20 15 10 - Average Disparity Max Disparity 0.2 0.4 0.6 0.8 Correlation 747 72 36 73Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 748 749Figure 3: Impact of Increasing Non-Linearity on Inaccuracy of level method to calculate partial EVPI Difference between level partial EVPI and level estimate Index -overall EVPI = 100 80 3e Trial parameters 60 3d 3c 40 Utility Parameters 3b 3a 20 Trial + Utility -20 750 0.8 0.85 0.9 0.95 Markov transition probability parameters Adjusted R sq < - increasing non-linearity of net benefit function 751 74 37 75Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 752Figure 4: Number of Monte Carlo Iterations and the Effect on Accuracy of EVPI estimate for Response Parameters 753 5,14 in an Independent Linear Cross-product model K (Outer level) 15 15 500 33 27 24 24 24 750 29 26 23 24 24 1000 31 30 26 27 27 2000 32 30 27 27 27 5000 30 27 24 24 24 10000 30 27 24 25 25 600 550 500 450 400 350 300 250 200 150 100 50 £334 756 757 76 K - outer level 100 10 14 500 16 750 24 1000 36 100 2000 1000 37 5000 750 36 1000 10000 500 40 500 100 44 750 10 10 10 754 755 EVPI estimates - Impact of More Samples 100 J (Inner Level J - inner level 550-600 500-550 450-500 400-450 350-400 300-350 250-300 200-250 150-200 100-150 50-100 0-50 38 77Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 758 759Table3 Standard Deviation in level partial EVPI estimates in Case Study (indexed to overall EVPI) 100 300 outer outer Parameter Subset Parameter Subset θ 5, θ 6, θ5, θ6, Case θ5, θ14 θ6, θ15 θ7, θ16 θ14, θ15 Study θ5, θ14 θ6, θ15 θ7, θ16 θ14, θ15 15.46 10.86 9.73 9.08 7.65 7.82 11.98 16.01 15.69 19.06 11.85 12.50 11.35 11.31 9.59 2.60 3.06 2.93 1.24 1.12 9.77 6.06 5.49 4.50 4.91 3.30 6.87 7.38 8.05 13.55 6.27 6.55 5.67 4.76 3.78 3a 4.36 11.04 3b 3.50 12.35 500 3c 2.31 9.73 Inner 3d 2.71 8.58 3e 2.39 6.34 760Standard deviations based on 30 runs 9.21 12.50 11.07 18.71 16.44 11.64 11.26 9.70 8.97 6.32 3.29 1.77 1.32 1.98 1.81 6.77 10.96 4.68 3.92 3.54 5.19 6.02 7.05 11.55 8.13 8.23 6.92 6.38 5.08 4.77 100 Inner 3a 3b 3c 3d 3e 5.66 4.79 3.75 3.09 2.56 761 762 78 39 79Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 763 Standard deviation in Estimate 764Figure 5: Stability of partial EVPI estimates using 100 outer and 500 inner samples Standard Deviation of 100 by 500 runs 20 15 10 0 50 100 Estimated partial EVPI 765 766 80 40 81Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 767REFERENCES 82 41 831 National Institute for Clinical Excellence Guidance for manufacturers and sponsors: Technology 84Appraisals Process Series No 2001 London, National Institute for Clinical Excellence 852 Manning WG, Fryback DG, Weinstein MC Reflecting uncertainty in cost-effectiveness analysis 86In Gold MR, Siegel JE, Russell LB, Weinstein MC, eds Cost-Effectiveness in Health and Medicine 87New York: Oxford University Press, 1996 p.247-75 883 Coyle D, Buxton MJ, O’Brien BJ Measures of importance for economic analysis based on 89decision modeling Journal of Clinical Epidemiology 2003:56;989–997 904 Felli C, Hazen GB Sensitivity analysis and the expected value of perfect information Medical 91Decision Making 1998;18:95-109 925 Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P The role of modelling in prioritising and 93planning clinical trials Health Technol Assess 2003;7(23) 946 Brennan, A., Chilcott, J., Kharroubi, S A and O'Hagan, A A two level Monte Carlo approach to 95calculating expected value of perfect information:- Resolution of the uncertainty in methods Poster 96at the 24th Annual Meeting of the Society for Medical Decision Making, October 23rd, 2002, 97Baltimore 2002 Available at http://www.shef.ac.uk/content/1/c6/03/85/60/EVPI.ppt 987 Raiffa H Decision analysis: Introductory lectures on choice under uncertainty 1968 Addison99Wesley, Reading, Mass 1008 Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR Bayesian Methods in Health Technology 101Assessment: A Review Health Technology Assessment 2000;4 1029 Claxton K,.Posnett J An economic approach to clinical trial design and research priority-setting 103Health Economics 1996;5:513-24 10410 Thompson KM, Evans JS The value of improved national exposure information for 105perchloroethylene (Perc): a case study for dry cleaners, Risk Analysis 1997;17:253-71 10611 Lapin, L L Quantitative methods for business decisions 1994 Orlando, Florida, Dryden Press 10712 Felli JC, Hazen GB Correction: Sensitivity Analysis and the Expected Value of Perfect 108Information, Medical decision Making, 2003;23:97 10913 Fenwick E, Claxton K, Sculpher M, Briggs A Improving the efficiency and relevance of health 110technology assessment: The role of iterative decision analytic modelling CHE Discussion Paper 111179, Centre for Health Economics 2000 11214 Claxton K, Neuman PJ, Araki SS, Weinstein M The value of information: an application to a 113policy model of Alzheimer's disease International Journal of Health Technology Assessment 1142001;17:38-55 11515 Payne N, Chilcott JB, McGoogan E Liquid-based cytology in cervical screening Health 116Technology Assessment 2000 11716 Felli JC, Hazen GB A Bayesian approach to sensitivity analysis Health Economics 1999;8:26311868 11917 Coyle D, Grunfeld E, Wells G The assessment of the economic return from controlled clinical 120trials A framework applied to clinical trials of colorectal cancer follow-up Eur J Health Econom 1212003;4:6-11 12218 Karnon J, Brennan A, Chilcott J Commentary on Coyle et al The Assessment of the economic 123return from controlled trials Eur J Health Econom 2003;4:239-40 12419 Meltzer D Addressing uncertainty in medical cost-effectiveness analysis—implications of 125expected utility maximization for methods to perform sensitivity analysis and the use of cost126effectiveness analysis to set priorities for medical research J Health Econ 2001;20:109-129 12720 Phelps CE, Mushlin AI Focusing technology-assessment using medical decision-theory Med 128Decis Making 1988;8:279-289 12921 Bartell SM, Ponce RA, Takaro TK, Zerbe RO, Omenn GS, Faustman EM Risk estimation and 130value-of-information analysis for three proposed genetic screening programs for chronic beryllium 131disease prevention Risk Anal 2000;20:87-99 13222 Yokota F, Thompson K Value of Information Literature Analysis: A Review of Applications in 133Health Risk Management, Medical Decision Making, 2004;24:287-98 13423 Gould J Risk,stochastic preference and the value of information, Journal of Economic Theory, 1351974;8: 64–84 13624 Hilton R Determinants of information value, Management Science, 1981;27: 57–64 13725 Howard RA Decision Analysis: Practice and Promise, Management Science, 1988;34:679–695 13826 Hammitt JK Can More Information Increase Uncertainty? Chance, 1995;8:15–17 13927 Hammitt JK, Shlyakhter AI The Expected Value of Information and the Probability of Surprise, 140Risk Analysis, 1999;19:135-152 14128 Brennan, A., Chilcott, J., Kharroubi, S A and O'Hagan, A (2002) A two level Monte Carlo 142approach to calculating expected value of perfect information:- Resolution of the uncertainty in 143methods Available at http://www.shef.ac.uk/content/1/c6/02/96/05/Brennan.doc 14429 Claxton K, Eggington S, Ginnelly L, Griffin S, McCabe C, et al A pilot study of value of 145information analysis to support research recommendations for the National Institute for Clinical 146Excellence, Society for Medical Decision Making Annual Conference, (abstract), 2004 14730 Coyle, D., McIntyre, L., Martin, C., and Herbert, P Economic analysis of the use of drotrecogin 148alfa (activated) in the treatment of severe sepsis in Canada Poster #9 presented at the 24th Annual 149Meeting of the Society for Medical Decision Making, October 20th, 2002, Washington 2002 15031 Brennan, A B., Chilcott, J B., Kharroubi, S., and O'Hagan, A A two level monte carlo approach 151to calculation expected value of sample information: how to value a research design Presented at 152the 24th Annual Meeting of the Society for Medical Decision Making, October 23rd, 2002, 153Washington 2002 Available at http://www.shef.ac.uk/content/1/c6/03/85/60/EVSI.ppt 15432 Brennan, A., Chilcott, J., Kharroubi, S A and O'Hagan, A A two level Monte Carlo approach to 155calculating expected value of sample information:- How to value a research design Discussion 156paper, 2002 Available at http://www.shef.ac.uk/content/1/c6/02/96/29/EVSI.doc 15733 Ades AE, Lu G, Claxton K Expected value of sample information calculations in medical 158decision modelling Medical Decision Making 2004 Mar-Apr;24(2):207-27 15934 Brennan, A B., Bansback, N., Martha, K., Peacock, M., and Huttner, K Methods to analyse the 160economic benefits of a pharmacogenetic test to predict response to biological therapy in rheumatoid 161arthritis and to prioritise further research Presented at the 24th Annual Meeting of the Society for 162Medical Decision Making, October 22nd, 2002, Washington 2002 16335 Oakley J Value of information for complex cost-effectiveness models Research Report No 164533/02 Department of Probability and Statistics, University of Sheffield, 2002 16536 Briggs AH, Gray A Handling uncertainty when performing economic evaluation of healthcare 166interventions Health Technology Assessment 1999;3 16737 Van Hout BA, Al MJ, Gordon GS, Rutten FF Costs, effects and C/E-ratios alongside a clinical 168trial Health Economics 1994;3:309-19 16938 http://www.shef.ac.uk/chebs/news/news1.html 17039 Briggs AH, Ades AE, Price MJ Probabilistic Sensitivity Analysis for Decision Trees with 171Multiple Branches: Use of the Dirichlet Distribution in a Bayesian Framework, Medical Decision 172Making 2003; 23:4:341-350 17340 Tappenden P, Chilcott J., Eggington, S., Oakley, J., McCabe C Methods for expected value of 174information analysis in complex health economic models: developments on the health economics of 175interferon-β and glatiramer acetate for multiple sclerosis Health Technol Assess 2004;8(27) 17641 Linville CD, Hobbs BF, Venkatesh BN Estimation of error and bias in Bayesian Monte Carlo 177decision analysis using the bootstrap, Risk Analysis, 2001;21:63-74 17842 Van Hout BA, Al MJ, Gordon GS, Rutten FF Costs, effects and C/E-ratios alongside a clinical 179trial Health Economics 1994;3:309-19 ... analysing the ? ?expected opportunity loss remaining” if perfect 10 4information were obtained on a subset of parameters 105 10 1 1Brennan et al Calculating Partial Expected Value Of Perfect Information. .. role of modelling methods 5 1in the prioritisation of clinical trials (Grant: 96/50/02) 52 7Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness Models 53INTRODUCTION... 431where we are interested in the partial EVPI of either a single parameter or a small number of parameters 36 18 3 7Brennan et al Calculating Partial Expected Value Of Perfect Information in Cost-Effectiveness