A rationale to unify measurements of effectiveness for animal health surveillance

Preventive Veterinary Medicine 120 (2015) 70–85 Contents lists available at ScienceDirect Preventive Veterinary Medicine journal homepage: www.elsevier.com/locate/prevetmed A rationale to unify measurements of effectiveness for animal health surveillance Vladimir Grosbois a,∗ , Barbara Häsler b , Marisa Peyre a , Dao Thi Hiep c , Timothée Vergne b a UPR AGIRs, Animal and Integrate Risk Management, International Research Center in Agriculture for Development (CIRAD), TA C 22/E Campus International Baillarguet, 34398 Montpellier Cedex 5, France b Veterinary Epidemiology, Economics and Public Health, The Royal Veterinary College, Hawkshead Lane, North Mymms, Hatfield, Herts AL9 7TA, United Kingdom c Center for Interdisciplinary Research on Rural Development, Vietnam National University of Agriculture, Trau Quy, Gia Lam, Hanoi, Viet Nam a r t i c l e i n f o Article history: Received July 2014 Received in revised form December 2014 Accepted 15 December 2014 Keywords: Intervention Disease surveillance Decision making Type I error Type II error a b s t r a c t Surveillance systems produce data which, once analysed and interpreted, support decisions regarding disease management While several performance measures for surveillance are in use, no theoretical framework has been proposed yet with a rationale for defining and estimating effectiveness measures of surveillance systems in a generic way An effective surveillance system is a system whose data collection, analysis and interpretation processes lead to decisions that are appropriate given the true disease status of the target population Accordingly, we developed a framework accounting for sampling, testing and data interpretation processes, to depict in a probabilistic way the direction and magnitude of the discrepancy between “decisions that would be made if the true state of a population was known” and the “decisions that are actually made upon the analysis and interpretation of surveillance data” The proposed framework provides a theoretical basis for standardised quantitative evaluation of the effectiveness of surveillance systems We illustrate such approaches using hypothetical surveillance systems aimed at monitoring the prevalence of an endemic disease and at detecting an emerging disease as early as possible and with an empirical case study on a passive surveillance system aiming at detecting cases of Highly Pathogenic Avian Influenza cases in Vietnamese poultry © 2015 Elsevier B.V All rights reserved Introduction The past 20 years have seen wide-reaching economic, social and political impact of large-scale animal disease outbreaks such as bovine spongiform encephalopathy, foot and mouth disease or avian influenza (Caspari et al., 2007; Knight-Jones and Rushton, 2013; Otte et al., 2004) These ∗ Corresponding author Tel.: +33 467593833; fax: +33 467593799 E-mail address: Vladimir.grosbois@cirad.fr (V Grosbois) http://dx.doi.org/10.1016/j.prevetmed.2014.12.014 0167-5877/© 2015 Elsevier B.V All rights reserved shockwaves emphasize the need for well-developed and adequately resourced health systems, including animal health surveillance (Rushton and Upton, 2006) Moreover, there are various endemic diseases that not get the same attention as large, unexpected outbreaks, but that cause continuous losses for society in terms of human disease, decreased productivity in animals and negative consequences for animal welfare (Otte et al., 2004; KnightJones and Rushton, 2013) Importantly, to combat animal disease, resources must be allocated to surveillance, prevention and intervention efforts that could otherwise be V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 used for alternative purposes (Häsler et al., 2011; Howe et al., 2013) While the need for effective animal health surveillance is widely recognized for the management of animal health threats, investment is being constrained due to financial budget restrictions Therefore, there is strong demand for frameworks that allow assessing the economic value of surveillance programmes that inform decision about investments for surveillance Surveillance has been defined as the systematic measurement, collection, collation, analysis, interpretation, and timely dissemination of animal-health and -welfare data from defined populations essential for describing health-hazard occurrence and to contribute to the planning, implementation, and evaluation of risk-mitigation actions (Hoinville et al., 2013) In other words, surveillance provides information for decisions regarding the implementation of interventions Together surveillance and intervention achieve loss avoidance through the process of making the effects of disease less severe by avoiding, containing, reducing or removing it – the outcome decision-makers are ultimately interested in (Häsler et al., 2011) If surveillance information shows that the disease situation is not of concern, then a decision may be taken not to anything In reality, the decision to implement an intervention does not only depend on the disease situation and the information provided by the surveillance system, but also on multiple other factors such as social expectations, political will, or practical considerations This article focuses on one decision-factor only, namely the quality of information provided by the surveillance system, while fully acknowledging the multi-factorial complexity of decision-making Keeping other factors constant, this article aims to provide a rationale for measurement of the effectiveness of information produced by animal health surveillance that is used to make a decision on disease management Surveillance data are generated through reporting, diagnosing, sampling and testing processes Often reporting and/or sampling are not exhaustive, and sometimes can be non-representative Moreover diagnostic and/or sample testing procedures usually misclassify a fraction of the examined units and tested samples The data generated by surveillance systems are thus most of the time non-exhaustive, partially distorted and sometimes nonrepresentative Decisions regarding the implementation of intervention measures nonetheless rely on the assessment through the analysis and interpretation of such imperfect data of the epidemiological status of target populations or of focal units in target populations (Häsler et al., 2011; Howe et al., 2013) Consequently, even with perfectly tailored response mechanisms, ineffective surveillance can result in misjudging an epidemiological situation and adopting inappropriate intervention measures Thus, although surveillance systems produce imperfect data, they should provide information that is reliable enough for suitable decisions on intervention measures to be made The challenge for surveillance systems is therefore to maximise the reliability of the information it produces through the optimization of the data generation and interpretation processes For doing so, attributes reflecting information reliability need to be assessed and variation 71 in such attributes with regard to the characteristics of data generation and interpretation processes need to be investigated So far, numerous attributes such as sensitivity, specificity, negative predictive value, positive predictive value, bias, precision and timeliness have been proposed to quantify such reliability (German et al., 2001; Hendrikx et al., 2011; Drewe et al., 2012, 2015; Hoinville et al., 2013) Moreover, effectiveness evaluations often aim at optimizing a specific aspect of the surveillance process which differs according to the objectives of the surveillance system considered Evaluations of the effectiveness of surveillance systems aiming at demonstrating freedom from disease most often focused on the sampling process (random vs risk-based, sample size, e.g Martin et al., 2007a) Such evaluation for systems aiming at detecting early the introduction of an emerging pathogen commonly focused on the comparison of the timeliness or componentlevel sensitivity of distinct surveillance components (e.g Yamamoto et al., 2008; Knight-Jones et al., 2010) Finally, evaluations of the effectiveness of syndromic surveillance mainly focused on statistical algorithms for the detection of anomalies in time series (e.g Dórea et al., 2013) None of these studies explored the meaning of performance attributes in general and did not establish a generic theoretical foundation for the measurement of effectiveness independent of the surveillance objective or approach used Consequently, there is little guidance available about what the common denominator is of the performance measures listed above and what differentiates them This can not only lead to confusion among users, but also limit standardisation and comparison of studies aiming to assess surveillance performance Here we present a rationale which can be used to assess effectiveness whatever is the objective of the surveillance system considered It is assumed that the primary effectiveness criterion is the ability of a surveillance system to provide information that is reliable enough for decision makers to implement mitigation measures similar to those they would implement given a perfect knowledge of an epidemiological situation This rationale allows developing optimization studies for any aspect of the surveillance process and forms an important basis for economic evaluation of surveillance General overview of the rationale The rationale we propose to assess the effectiveness of a surveillance system relies on the principle that the decisions that are made based on the information produced by surveillance should not differ greatly from the decisions on interventions that would be made given perfect knowledge of the epidemiological situation (i.e of the epidemiological state of the target population and of its components) It requires reviewing several aspects of mitigation strategies and processes, as detailed below and highlighted in Fig 2.1 Defining relevant epidemiological scales and state variables The epidemiological scale and the state variable(s) that are relevant with regard to the objectives of surveillance 72 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 SURVEILLANCE DATA Non-exhausƟve, non-representaƟve, parƟally distorted Data generaƟon process Sampling, reporƟng, diagnosing, tesƟng Data analysis and interpretaƟon TRUE EPIDEMIOLOGICAL SITUATION ASSESSMENT EPIDEMIOLOGICAL SITUATION IntervenƟon strategy Defined based on epidemiological modelling and cost-effecƟveness and/or cost-benefit analyses PREVENTION/CONTROL MEASURES That would be implemented given a perfect knowledge of the epidemiological situaƟon Decision making process Surveillance EffecƟveness EFFECTIVENESS AND ECONOMIC EFFICIENCY of the prevenƟon/control measures that would be implemented given a perfect knowledge of the epidemiological situaƟon PREVENTION/CONTROL MEASURES That are actually implemented (modaliƟes/intensity) EFFECTIVENESS AND ECONOMIC EFFICIENCY of the risk prevenƟon/control measures that are actually implemented Fig Proposed approach for the evaluation of the effectiveness of a surveillance system Table Examples of simple intervention strategies for various surveillance objectives Surveillance objective Scale State variable S− I− S+ I+ Monitoring prevalence Country/region Yearly prevalence of a disease (Prev) Prev ≤ Threshold Do nothing Prev > Threshold Disease case detection Herd Disease status Do nothing Demonstrate freedom from disease Early detection of an emerging disease Country/region Yearly prevalence of a disease (Prev) Instantaneous incidence rate (IIR) No infected animal in the herd Prev ≤ Threshold ≥1 infected animal in the herd Prev > Threshold Implement systematic testing in slaughterhouses before products are put on the market Cull the herd IIR = Do nothing Country/region Allow exportations IIR > Ban exportations Launch intensive surveillance and in depth case investigation Limit movements S− , S+ : epidemiological states for which the “no intervention” and “intervention” options, respectively, are required; I− , I+ : description of actions associated with to the “no intervention” and “intervention” options form the basis for an intervention decision The relevant epidemiological scale is the scale at which decisions are being made about implementing an intervention Such decisions can be for example to start vaccinating animals in the target population if the disease prevalence crosses a defined threshold or not to anything if surveillance to document freedom from disease delivers the expected result (i.e freedom) The scale can be animal, herd, country, V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 regional or global level The relevant state variable is a variable, such as prevalence or incidence, that reflects the current epidemiological situation, and which value determines the intervention measures considered as appropriate by stakeholders and decision makers Table provides examples of relevant epidemiological scales and associated state variables for surveillance systems with distinct objectives 2.2 Describing the intervention strategy The proposed rationale relies on the comparison of the decisions likely to be made based on the information produced by surveillance with the decisions that would be made given a perfect knowledge of the epidemiological situation Consequently, the decisions that would be considered as appropriate by stakeholders and decision makers for a set of possible epidemiological situations need to be described Planning response mechanisms to potential epidemiological situations constitutes an important measure to improve preparedness towards threats posed by animal diseases (Rushton and Upton, 2006) Epidemiological modelling and analysis in combination with economic evaluation produce the scientific evidence for planning intervention strategies Such approaches have been widely used for the definition of national and international preparedness plans Predefined intervention strategies should thus in most instances exist and can be described (Tomassen et al., 2002) Intervention strategies can be described through the relationship between the value(s) of the epidemiological state variable(s)that characterize an epidemiological situation and the intervention measures considered as appropriate for that epidemiological situation Usually, the possible values of the epidemiological state variable(s) are classified into ordered categories of increasingly harmful sanitary and economic consequences In Table 1, contrived examples of simple intervention strategies are presented for distinct surveillance objectives In these strategies, the possible values of the relevant state variable are classified according to two subsets referred to as S+ and S− Each of these subsets is associated with a pre-defined intervention option (I+ and I− , respectively) considered as appropriate by stakeholders and decision makers S+ is the subset of values of the state variable that requires the implementation of intervention measures (i.e intervention option I+ ) and S− is the subset of values of the state variable that requires no intervention (i.e intervention option I− ) 2.3 Describing the data generation and interpretation processes Once the epidemiological scale, the state variable and the intervention strategy are defined, it is necessary to describe the surveillance data generation and interpretation processes that produce the information upon which decision makers rely for the implementation of intervention measures (Fig 1) Surveillance data generation processes include reporting (e.g underreporting rate and the factors influencing it), diagnostic (e.g case 73 definition), sampling (e.g coverage, stratification, intensity, frequency), and sample testing (e.g sensitivity and specificity of the tests used) Surveillance data interpretation involves in most instances the computation of statistics that provide an assessment of the current epidemiological situation and inform decisions regarding intervention Considering the potential intervention strategies presented in Table where two subsets of values of the focal state variable are considered, two subsets of values for such a statistic can be defined (Table 2) A+ is the subset for which the focal epidemiological state variable is assessed as falling into the category requiring the implementation of intervention measures (I+ ) A− is the subset for which the focal epidemiological state variable is assessed as falling into the category requiring no intervention measure (I− ) 2.4 Effectiveness criteria With S+ , S− , A+ , A− determined, it is possible to define two types of errors, namely Type I and Type II errors, analogously to the types of error used in statistical or diagnostic tests (Table 3) Type I error occurs when a surveillance system produces information which results in the implementation of intervention measures while the true state of the population would not require it Type I errors imply that costly mitigation measures are unnecessarily activated Type II error occurs when a surveillance system produces information which results in no implementation of mitigation measures while the true state of the population would require it Type II errors result in increased risks of failure to control a genuine disease threat or may lead to a delayed response The effectiveness of a surveillance system can be assessed by estimating for that system the probabilities of Type I errors P(A+ |S− ) and the probabilities of Type II errors P(A− |S+ ) Using the information on intervention strategies as well as on data generation, analysis and interpretation processes leading to decisions, probabilities of Type I and Type II errors can be assessed either analytically using sampling and probability theories or through simulations This is illustrated in the following section with two hypothetical and a real examples Illustrations of effectiveness assessment three contrived surveillance system examples and an empirical case study In this section, the proposed rationale is further developed for four types of surveillance systems with the objectives of: (1) demonstrating freedom from a disease, (2) monitoring the prevalence of an endemic disease, (3) detecting as early as possible the presence of an emerging disease, and (4) detecting cases of a disease 3.1 Demonstrating freedom from a disease The case of surveillance systems aiming at demonstrating a territory as free from a disease is interesting because the rationale proposed here has already been applied to assess effectiveness of such surveillance systems The state 74 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 Table Examples of decision making rules relying on the analysis and interpretation of surveillance data The decision rules correspond to the mitigation strategies presented in Table Surveillance objective Scale Statistics used to assess epidemiological status A− Decision I− A+ Decision I+ Monitoring prevalence Country/region Proportion of positive tests in the samples collected over a year P(+) P(+) ≤ Threshold Do nothing P(+) > Threshold Case detection of disease Demonstrate freedom from disease Herd Result of a pooled test Proportion of positive tests in the samples collected over a year P(+) Case reporting Negative test result P(+) ≤ Threshold Do nothing Positive test result P(+) > Threshold Implement systematic testing in slaughterhouses before products are put on the market Cull the herd No case reported Do nothing Early detection of an emerging disease Country/region Country/region Allow exportations ≥1 case reported Ban exportations Launch intensive surveillance and in depth case investigation Limit movements A− , A+ : assessments of epidemiological state for which the “no intervention” and “intervention” options, respectively, are implemented; I− , I+ : description of actions associated with the “no intervention” and “intervention” options Table The two types of error used as effectiveness criteria True epidemiological status S+ intervention required S− intervention not required Assessment of the epidemiological status resulting from the generation, analysis and interpretation of surveillance data A+ intervention implemented Type I error A− intervention not implemented Type II error S− , S+ : epidemiological states for which the “no intervention” and “intervention” options, respectively, are required; A− , A+ : assessments of epidemiological state for which the “no intervention” and “intervention” options, respectively, are implemented variable which conditions decisions in terms of prevention/intervention measures is usually the prevalence of the disease in the focal population The prevalence categories considered as requiring distinct intervention options are determined according to the so called “design prevalence” Whenever the prevalence in the population is below the design prevalence, the territory is considered “as free from the disease” (S− ) and no measure to limit its spread is implemented (for instance no limitations to animal trading: I− ) whereas whenever the prevalence in the population is above the design prevalence, measures to limit its spread are implemented (for instance animal trading is restricted: I+ ) The crucial aspect of the mitigation strategy is the determination of the design prevalence It can be chosen based on the relative likelihood of prevalence levels given the presence of the disease on the territory or by considering how the magnitude of sanitary and economic consequences of the presence of the disease vary as a function of the prevalence of that disease So the design prevalence can be the minimum expected prevalence of the disease provided it is present on the territory or the maximum prevalence at which the sanitary and economic consequences of the presence of the disease are considered as negligible The statistics used to assess the epidemiological situation from surveillance data is usually the binary variable reflecting whether at least one case has been detected (A+ ) or no case has been detected (A− ) In the numerous papers in which this approach has been used to assess the effectiveness of surveillance systems aiming at demonstrating the freedom of a territory from a disease (e.g Martin et al., 2007a; Martin, 2008; Frössling et al., 2009; Hood et al., 2009; Christensen et al., 2011), the effectiveness criterion used is the probability of a Type II error P(A− |S+ ) which is the probability that the territory is qualified as free of a disease while the prevalence of the disease is above the design prevalence The method applied to compute this probability is usually scenario tree modelling (Martin et al., 2007b) although other methods have been proposed (Hood et al., 2009) These published effectiveness assessments for systems aiming at demonstrating freedom from disease already follow the rationale proposed here and support the logic outlined V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 75 Table Information for assessing the effectiveness of a contrived surveillance system aiming at monitoring the prevalence of an endemic disease Surveillance objective Relevant scale Relevant epidemiological variable Intervention strategy Surveillance data generation process Statistics computed from surveillance data Decision rule (test performances not accounted for) Decision rule (test performances accounted for) Knowing how prevalent is an endemic disease to inform decisions about vaccination strategy Country (population of 100,000 animals) Individual level prevalence (p) S− p ≤ 0.1 I− no vaccination S+ 0.1 < p ≤ 0.2 I+ vaccination is implemented only in high risk areas S++ p > 0.2 I++ vaccination is implemented in all areas A+ 0.1 * n < np ≤ 0.2 * n I+ targeted vaccination is implemented A+ (0.1 * Se + (1 − 0.1) * (1 − Sp)) * n 0.2 * n I++ vaccination is implemented in all areas A++ np > (0.2*Se + (1 − 0.2)*(1 − Sp)) * n n = 100 randomly chosen individuals are sampled over a month period (coverage = 0.1%) Each sample is tested using a test with sensitivity Se = 0.90 and specificity Sp = 0.95 Number of sampled units testing positive (np ) A− np ≤ 0.1 * n I− no vaccination A− np ≤ (0.1 * Se + (1 − 0.1) * (1 − Sp)) * n I− no vaccination I++ vaccination is implemented in all areas S− , S+ , S++ : epidemiological states for which the “no intervention”, “low intensity intervention” and “high intensity intervention” options, respectively, are required; I− , I+ , I++ : description of actions associated with the “no intervention”, “low intensity intervention” and “high intensity intervention” options, respectively; A− , A+ , A++ : assessments of epidemiological state for which the “no intervention”, “low intensity intervention” and “high intensity intervention” options, respectively, are implemented 3.2 Monitoring the prevalence of an endemic disease This section presents a contrived example an active surveillance system aiming at monitoring prevalence of a cattle disease to inform decision-makers on which vaccination strategy to implement at the national level 3.2.1 Information required for assessing effectiveness Table summarises the information required to assess the effectiveness of such a surveillance system Potential epidemiological situations are categorised according to three prevalence levels: at low prevalence, it is considered that vaccination is not necessary; at intermediate prevalence, it is considered that targeted vaccination should be implemented around detected outbreaks; at high prevalence, it is considered that vaccination should be implemented in all areas of the country Surveillance data are assumed to be generated through random sampling of individuals (which ensures homogeneous coverage of the population) and assessment of individual disease status with a test of known sensitivity and specificity (Table 4) The data interpretation process consists in comparing the number of samples testing positive with the expected numbers of diseased individuals in the sample for the prevalence thresholds defined in the intervention strategy (Table 4) 3.2.2 Assessment of effectiveness The effectiveness of this surveillance system is determined by estimating the probabilities that the information produced by the surveillance system leads to the implementation of inappropriate intervention measures Using the notations of Table 4, Pr(A− |S+ ), Pr(A− | S++ ) and Pr(A+ |S++ ) represent probabilities of more or less severely under-sizing the intervention measures given the true epidemiological situation (i.e more or less severe Type II errors) while, Pr(A+ |S− ), Pr(A++ |S+ ) and Pr(A++ |S− ) represent probabilities of more or less severely over-sizing 76 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 Fig Probabilities that data generation, analysis and interpretation processes result in the implementation of different intervention options as a function of the true epidemiological state n: sample size; Se: sensitivity of the test; Sp: specificity of the test; S− : vaccination is not required; S+ : targeted vaccination is required; S++ : mass vaccination is required the intervention measures given the true epidemiological situation (i.e more or less severe Type I errors) The probability distribution of the statistics examined to make decisions about intervention measures (i.e the number of samples testing positive, np ) is known: it is a binomial distribution where the number of trials parameter is the sample size (n) and the probability parameter is a function of the real prevalence of the disease in the population and of the test performance parameters (pSe + (1 − p)(1 − Sp)) So Pr(X < np < Y|p, n, Se, Sp) can be computed for any value of X, Y, p, n, Se and Sp Fig illustrates the surveillance effectiveness of this hypothetical example This figure displays for any given value of true disease prevalence (p), the probabilities that decision makers implement the “no vaccination” (I− ),“targeted vaccination” (I+ )or “mass vaccination” (I++ ) options according to decision rule in Table It also shows the ranges for true prevalence (p) requiring distinct intervention measures to be implemented as defined by decision-makers (S− : “no vaccination”, S+ : “targeted vaccination”, S++ : “mass vaccination”) When true prevalence is just above 0.2 (thus where mass vaccination would be required), it is estimated that given the sample size, the diagnostic test characteristics, and the decision rule used, the probability of actually implementing mass vaccination is 0.65, the probability of implementing only targeted vaccination is 0.35 (moderate Type II error) and the probability of not implementing vaccination is (severe Type II error) When true prevalence is just above 0.05 (thus no vaccination would be required), it is estimated that the probability of nonetheless implementing mass vaccination (severe Type I error) is 0, the probability of implementing targeted vaccination is 0.32 (moderate Type I error) and the probability of not implementing vaccination is around 0.68 3.2.3 Sensitivity of effectiveness to the characteristics of data generation and interpretation processes The proposed approach allows assessing how probabilities of Type I and Type II errors change when reporting, diagnostic, sampling, sample testing or data interpretation procedures are modified In Fig 3, modifications of the surveillance process in terms of sample size and performance of the diagnostic test used to detect the disease in each sampled unit are illustrated Increasing sample sizes and improving test performances results in reducing the probabilities of Type I and Type II errors In Fig the sample size is 100, Se is 0.6 and Sp is 0.8 in the two panels but the surveillance data interpretation process differs between the two panels In Fig 4a the decision regarding intervention relies on an assessment of the population epidemiological status (i.e the prevalence level) that does not account for the fact that the test used to assess individual disease status is imperfect (decision rule in Table 4) In Fig 4b test sensitivity and specificity are accounted for in the assessment of the population epidemiological status (decision rule in Table 4) This figure illustrates that changes in the data interpretation process can improve dramatically the performance of a surveillance system 3.3 Early detection In the case of surveillance systems aiming at detecting the introduction of a pathogen in a territory or a population as early as possible, the state variables which condition decisions regarding intervention measures are the binary variable that indicates whether or not the pathogen infects at least one unit in the focal host population and the time elapsed since the occurrence of the index infection(s) in the focal population The latter underlies a number of other V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 77 Fig Sensitivity of surveillance effectiveness to changes in sampling and sample testing procedures n: sample size; Se: sensitivity of the test; Sp: specificity of the test Fig Sensitivity of surveillance effectiveness to changes in data analysis and interpretation procedures n: sample size; Se: sensitivity of the test; Sp: specificity of the test 78 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 Table Information for assessing the effectiveness of a contrived surveillance system aiming at detecting an emerging or exotic disease early Surveillance objective Relevant scale Relevant epidemiological variables Intervention strategy Surveillance data generation process Statistics computed from surveillance data Decision rule Detecting an emerging disease following its introduction in a territory as soon as possible Country (population of 10,000 animals) Cumulative incidence (correlated with time elapsed since introduction and spatial spread) S− The disease has not yet been introduced I− Keep low intensity surveillance with 50 individuals sampled daily S+ The disease has been introduced but cumulative incidence is Rt) the probability of case reporting when mortality is above the reporting threshold and Se the sensitivity of the confirmatory test The probability of a Type II error can be computed as: P A− |S + = − P(M > Rt) × P(R|M > Rt) × Se (1) Although it is known that the transmission of HPAI viruses in poultry flocks is very fast and that mortality rates from 0.5 to can be reached within 3–4 days from the onset of the first clinical signs (Swayne, 2009), it is difficult to obtain precise information on mortality rates over days in villages or holding infected by HPAI However mortality rates over days of 0.33 in poultry holdings infected by H5N1 viruses in Thailand have been reported in Tiensin et al (2007) Furthermore the analysis of unpublished data on mortality monitoring in poultry flocks infected by H9N2 viruses in an Egyptian holding revealed that maximum mortality over days in these flocks ranged between 0.06 and 0.24 This information was used to define a distribution for maximum mortality over days in infected units This distribution was assumed to be a Beta distribution of parameters ˛ = 1.85 and ˇ = 4.41 The 5th percentile of this distribution is 0.06 and its median is 0.20 The probability P(M > Rt) of maximum mortality over days being higher than the reporting threshold is easily computed using the cumulative density function of this distribution For a reporting threshold of 0.05, this probability equals 0.96 Assuming that the recommendations of Vietnamese state veterinary services are strictly respected, the probability that a unit is reported as suspicion is when maximum mortality over days in that unit exceeds 0.05 and when maximum mortality over days in that unit does not exceed 0.05 Thus, P(R|M > Rt) = 82 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 Table Information for assessing the effectiveness of the passive surveillance system aiming at detecting cases of Highly Pathogenic Avian Influenza (HPAI) in the Vietnam poultry population Surveillance objective Relevant scale Relevant epidemiological variable Intervention strategy Surveillance data generation process Statistics computed from surveillance data Decision rule Find villages/holdings infected by HPAI Industrial holding/village Epidemiological status (infected/uninfected) of the unit (holding or village) S− The unit is not infected (no infected animal in the unit) I− Do nothing A unit is reported as suspicious whenever mortality rate over 2-day exceeds 5% in that unit and specific symptoms are observed A confirmatory tests which sensitivity is Se = 0.93 and specificity is Sp = 0.98 is applied to all the units reported as suspicious Infectious status of the unit as perceived through the reporting and testing processes A− The unit is not reported as suspicious The unit is reported as suspicious but the confirmatory tests have not detected HPAI I− Do nothing S+ The unit is infected (at least one infected animal in the unit) I+ Cull the unit and implement ring vaccination A+ The unit is reported as suspicious and the confirmatory tests have detected HPAI I+ Cull the unit and implement ring vaccination S− , S+ : epidemiological states for which the “no intervention” and “intervention” options are respectively required; I− , I+ : description of actions associated respectively to the “no intervention” and “intervention” options; A− , A+ : assessments of epidemiological state for which the “no intervention” and “intervention” options are respectively implemented Finally, given that the sensitivity of the confirmatory test is 0.93, the probability of a Type II error according to Eq (1) is P(A− |S+ ) = − 0.93 × × 0.96 = 0.11 The above evaluation of effectiveness suggests that HPAI passive surveillance in Vietnam is quite effective (the probabilities of Type I and Type II errors are low) However, assumptions such as strict adherence to the recommendations of Vietnamese state veterinary services or maximum mortality following a Beta (1.85, 4.41) distribution should be questioned and could be easily relaxed in a sensitivity analysis such as the one presented in Section 3.2 For instance, unpublished data obtained from participatory investigations of the reporting behaviour of backyard poultry owners suggest that suspicions are actually not reported when mortality over days is lower than 0.3, so that the reporting threshold would be 0.3 rather than 0.05 According to Eq (1), the probability of a Type II error in that situation would be equal to 0.58 Finally, a thorough evaluation of effectiveness of this surveillance system should also consider that mortality and reporting patterns differ between types of units (industrial holdings and villages) Discussion This paper shows how the effectiveness of a surveillance system can be evaluated in terms of discrepancy between the modalities and intensity of prevention and/or control measures that would be implemented given a perfect knowledge of the true epidemiological status of a population and of its components and the modalities and intensity of prevention and/or control measures that are likely to be actually implemented based on the analysis and interpretation of the data produced by a surveillance system We have also shown that this rationale can be used to conduct sensitivity analyses to establish which changes in the surveillance system allow improved effectiveness Importantly, it appears that information on data generation processes alone does not allow thorough evaluations of surveillance effectiveness Indeed, information on planned mitigation strategies, on the processes through which surveillance data are analysed and interpreted and on the decision-making process leading to the implementation of mitigation strategies are also crucial 4.1 Links with previously proposed effectiveness criteria It is important to note that probabilities of Type I and Type II errors have already been used as effectiveness attributes for surveillance systems or components However they are most often referred to as component-level or system-level sensitivity (which is the complement to of the probability of a Type II error) and false alarm rate (which is the probability of a Type I error) We argue that system-level sensitivity and false alarm rate are relevant effectiveness criteria for any surveillance system All the other previously proposed effectiveness attributes matter in that they influence system-level sensitivity and false alarm rate V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 The rationale proposed here identifies the analysis and interpretation of surveillance data as an important aspect of surveillance The analysis and interpretation of surveillance data often imply estimations of epidemiological variables such as incidence or prevalence The accuracy (or bias) and precision of such estimations have been considered as important surveillance effectiveness attributes (e.g Drewe et al., 2015; Hoinville, 2013) However, their importance relies in that they ultimately influence probabilities of making decisions that differ from those that would be made given a perfect knowledge of the true value of the focal epidemiological variables We thus argue that accuracy and precision are in some instances important quantities for computing probabilities of Type I and Type II errors which, ultimately, are the two most relevant attributes for assessing effectiveness The positive predictive value (PPV) and negative predictive value (NPV) computed at system level have also been proposed as relevant statistics for the evaluation of surveillance systems (Drewe et al., 2012) As pointed out by Martin et al (2007b) PPV and NPV are related to probabilities of a focal population being in a given epidemiological state given an assessment of the epidemiological state of that population through the analysis and interpretation of surveillance data (i.e P(S|A) using the notation introduced above) Indeed, the PPV is P(S+ |A+ ) (or − P(S− |A+ )) and the NPV is P(S− |A− ) (or − P(S+ |A− )) NPV and PPV are important quantities that inform decision makers about the risk taken when making a decision and therefore constitute critical information in decision making (Martin et al., 2007b) P(S+ |A− ) (i.e − NPV) informs the decision maker on the probability that the true epidemiological situation would require the implementation of mitigation measures in situations where surveillance evidence suggests that no mitigation measures should be implemented (for example the probability to declare a territory free of a disease although the disease is present with a prevalence higher than the design prevalence (Martin, 2008; Frössling et al., 2009)) P(S− |A+ ) (i.e − PPV) informs the decision maker on the probability that the true epidemiological status would not require the implementation of mitigation measures in situations where surveillance evidence suggests that mitigation measures should be implemented (for example the probability not to declare a territory free of a disease although the disease is absent or present with a prevalence lower than the design prevalence) PPV and NPV are thus useful quantities for interpreting data produced by surveillance rather than for evaluating the effectiveness of a surveillance system P(S+ |A− ) (i.e − NPV) and P(S− |A+ ) (i.e − PPV) are related to the probability of Type II error (P(A− |S+ )) and the probability of Type I error (P(A+ |S− )) through the Bayes formula for conditional probabilities (Martin, 2008) Timeliness is considered as an important attribute for the effectiveness of surveillance systems aiming at detecting the introduction or the emergence of a pathogen because the later the implementation of intervention measures (relative to the time of occurrence of the index case) the larger are the losses already generated by the disease and the costs of intervention measures required to control it Thus early detection is important because it 83 insures detection before the disease has spread widely in the population The hypothetical example of a surveillance system aiming at detecting a disease as early as possible presented above illustrates this point by considering that decision makers plan their intervention strategy in relation with cumulative incidence rather than with time since introduction It also shows that when intervention strategies are elaborated in this way timeliness can be incorporated in the evaluation of Type I and Type II by looking at how probabilities of such errors change as time since introduction increases 4.2 Limitations 4.2.1 Collecting information on mitigation strategies The proposed rationale requires the characterization of potential epidemiological situations in terms of categories considered by stakeholders and decision makers as requiring distinct responses This is a pre-requisite for the computation of the probabilities of Type I and Type II errors It is possible that in some instances, intervention strategies are not defined yet and that responses to threats posed by animal diseases follow conventional outbreak investigation activities Depending on the technical possibilities, interventions may then be implemented to contain the disease However, in some cases there are either no technical intervention measures available, or the disease is too widespread or not considered important enough to warrant a reaction Planning an intervention for any type of unknown hazards poses a considerable challenge for animal health services, because no information is available about the nature of any such hazard, the population it affects, or its transmission and physiological characteristics The EU Animal Health Strategy for 2007–2013 (available at http://ec.europa.eu/food/animal/ diseases/strategy/index en.htm) advocated the precautionary principle “where proportionate provisional measures should be taken to ensure a high level of health protection pending further scientific information clarifying the extent of the risk” But in the absence of information about what type of hazard emergence is to be expected, the formulation of appropriate strategies and therefore the assessment of early warning surveillance for emerging diseases are severely constrained More epidemiological research is needed to estimate the likelihood of different categories of hazards, which then allows gathering information on likely consequences and the necessary response The availability of such structured approaches to support decision-making are critical to direct resources towards hazards identified based on latest scientific evidence, which will avoid ‘fishing in the dark’ Participatory approaches involving stakeholders and decision makers (e.g using companion modelling) could for instance be used to determine which management measures are considered as appropriate for different epidemiological scenarios (regarding the status of the focal population and/or its components) In conclusion, the description of an intervention strategy might not always be available but is an essential pre-requisite to assess surveillance effectiveness through the approach presented here 84 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 4.3 Further developments 4.3.1 Considering more complex mitigation strategies For the sake of clarity we have only considered mitigation strategies in which the possible values of the epidemiological state variable were categorized into subsets each of which was associated with a specific mitigation option Mitigation strategies in which the modalities and intensity of mitigation measures vary continuously as a function of the value of the epidemiological state variable will be considered in further developments Surveillance-control is most often an adaptive management process in which surveillance informs decision makers not only on the current epidemiological state of a focal population but also on the effectiveness of the intervention measures implemented The implementation of an intervention measure is thus likely to be motivated by a change in the epidemiological state of the population resulting from previous interventions rather than by its current epidemiological state It is thus necessary to extend the proposed rationale to situations where surveillance iteratively produce information on the evolution of an epidemiological situation that determine sequences of intervention measures 4.3.2 Integration of an epidemiological component As exemplified in the hypothetical surveillance system aiming at monitoring an endemic disease presented above, the proposed approach allows evaluating the effectiveness of surveillance for a given epidemiological situation One possible limitation is thus that estimations of the probabilities of Type I and Type II errors are relevant with regard only to this unique situation A question that will shortly be addressed is that of the integration of probabilities of Type I and Type II errors over sets of potential epidemiological situations Epidemiological models could be integrated in our framework to derive relative probabilities of occurrence the potential situations in such sets Such relative probabilities could be used to compute weighted average for probabilities of Type I and Type II errors 4.3.3 Economic analysis and decision-making With the probabilities of Type I and Type II errors established, the next step is to assess what the economic consequences are of each type of error This would allow assessing discrepancy between the cost, the effectiveness and the benefits of intervention measures that would be implemented given a perfect knowledge of the epidemiological situation and the cost, the effectiveness and the benefits of intervention measures that are likely to be actually implemented based on the analysis and interpretation of the data produced by a surveillance system Evaluating economic consequences of Type I and Type II errors implies estimating the economic consequences of either implementing costly interventions unnecessarily or not implementing interventions when they would be required because of disease presence This includes valuating (1) the production losses that occur due to morbidity and mortality in animals by for example multiplying physical losses such as reduction in litres of milk produced in dairy cows by price coefficients; (2) all the financial and other resources used for intervention measures (e.g vaccines, veterinary services, drugs); and (3) wider impacts including human health effects, spill-over to other sectors (e.g disruption to tourism), and impacts on downstream and upstream businesses (e.g breeders, feed and drug producers, slaughterhouses), and multiplying them by the probability of the error in question Consequently, it is indispensable to have an idea of the number of holdings or animals affected (or in other words of the spread of the disease) as well as the activities comprised in mitigation activities In the absence of empirical data, epidemiological modelling techniques that capture the dynamics and complexity of disease in populations can be used to generate these data Such models are often used to deliver important input data for economic analyses (Perry and Randolph, 2004) Ideally, the estimations of costs are made based on a continuous function of, for example, the cumulative incidence and the probability of the errors as well as the surveillance costs associated with this error Like this, the economic consequences of a Type I or II error can be compared directly to the investment in surveillance needed to reduce the probability of this error and the target or target range can be determined where overall costs are minimised Consequently, such a target would be an epidemiologically and economically efficient one Hence, the rationale proposed is suitable both to establish the value of the surveillance in relation to an already defined target, as well as to refine a decision rule taking into account the socio-economic consequences of Type I or II errors and the additional surveillance costs that would accrue to reduce these errors Conflicts of interest There are no conflicts of interest Acknowledgments The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007–2013) under grant agreement no 310806 (RISKSUR project) We warmly thank all the participants of RISKSUR project without whom the completion of this work would not have been possible References Caspari, C., Christodoulou, M., Monti, E., 2007 Prevention and control of animal diseases worldwide Economic Analysis – Prevention Versus Outbreak costs Final Report Part I The World Organisation for Animal Health, pp 198, Available at: http://www.oie.int/doc/ged/D4309.PDF Christensen, J., Stryhn, H., Vallières, A., Allaki, F.E., 2011 A scenario tree model for the Canadian Notifiable Avian Influenza Surveillance System and its application to estimation of probability of freedom and sample size determination Prev Vet Med 99, 161–175 Dórea, F.C., McEwen, B.J., McNab, W.B., Revie, C.W., Sanchez, J., 2013 Syndromic surveillance using veterinary laboratory data: data preprocessing and algorithm performance evaluation J R Soc Interface 10, 20130114 Drewe, J., Hoinville, L., Cook, A., Floyd, T., Stärk, K., 2012 Evaluation of animal and public health surveillance systems: a systematic review Epidemiol Infect 140, 575–590 Drewe, J., Hoinville, L., Cook, A., Floyd, T., Gunn, G., Stärk, K., 2015 SERVAL: a new framework for the evaluation of animal health surveillance Transbound Emerg Dis 62, 33–45 V Grosbois et al / Preventive Veterinary Medicine 120 (2015) 70–85 Frössling, J., Ågren, E.C.C., Eliasson-Selling, L., Lewerin, S.S., 2009 Probability of freedom from disease after the first detection and eradication of PRRS in Sweden: scenario-tree modelling of the surveillance system Prev Vet Med 91, 137–145 German, R.R., Lee, L., Horan, J., Milstein, R., Pertowski, C., Waller, M., 2001 Updated guidelines for evaluating public health surveillance systems MMWR Recomm Rep 50, 1–35 Häsler, B., Howe, K.S., Stärk, K.D., 2011 Conceptualising the technical relationship of animal disease surveillance to intervention and mitigation as a basis for economic analysis BMC Health Serv Res 11, 225 Hendrikx, P., Gay, E., Chazel, M., Moutou, F., Danan, C., Richomme, C., Boue, F., Souillard, R., Gauchard, F., Dufour, B., 2011 OASIS: an assessment tool of epidemiological surveillance systems in animal health and food safety Epidemiol Infect 139, 1486–1496 Hoinville, L., 2013 Animal Health Surveillance Terminology Final Report from Pre-ICAHS Workshop, Available at: http://www.defra.gov.uk/ ahvla-en/files/icahs-workshop-report.pdf Hoinville, L., Alban, L., Drewe, J., Gibbens, J., Gustafson, L., Häsler, B., Saegerman, C., Salman, M., Stärk, K., 2013 Proposed terms and concepts for describing and evaluating animal-health surveillance systems Prev Vet Med 112, 1–12 Hood, G., Barry, S., Martin, P., 2009 Alternative methods for computing the sensitivity of complex surveillance systems Risk Anal 29, 1686–1698 Howe, K., Häsler, B., Stärk, K., 2013 Economic principles for resource allocation decisions at national level to mitigate the effects of disease in farm animal populations Epidemiol Infect 141, 91–101 Knight-Jones, T., Rushton, J., 2013 The economic impacts of foot and mouth disease – what are they, how big are they and where they occur? Prev Vet Med 112, 161–173 Knight-Jones, T.J., Hauser, R., Matthes, D., Stärk, K.D., 2010 Evaluation of effectiveness and efficiency of wild bird surveillance for avian influenza Vet Res 41, 50 MARD, MOH, 2011 Vietnam Integrated National Operational Program on Avian Influenza, Pandemic Preparedness and Emerging Infectious Diseases, 2011–2015 Ministry of Agriculture and Rural Development and Ministry of Health, Hanoi, Vietnam, Available at: http:// animaldiseaseintelligence.com/wordpress/wp-content/uploads/ 2011/12/AIPED.pdf Martin, P., 2008 Current value of historical and ongoing surveillance for disease freedom: surveillance for bovine Johne’s disease in Western Australia Prev Vet Med 84, 291–309 85 Martin, P., Cameron, A., Barfod, K., Sergeant, E., Greiner, M., 2007a Demonstrating freedom from disease using multiple complex data sources: 2: Case study – classical swine fever in Denmark Prev Vet Med 79, 98–115 Martin, P., Cameron, A., Greiner, M., 2007b Demonstrating freedom from disease using multiple complex data sources: 1: A new methodology based on scenario trees Prev Vet Med 79, 71–97 Minh, P.Q., Morris, R.S., Schauer, B., Stevenson, M., Benschop, J., Nam, H.V., Jackson, R., 2009 Spatio-temporal epidemiology of highly pathogenic avian influenza outbreaks in the two deltas of Vietnam during 2003–2007 Prev Vet Med 89, 16–24 Otte, M., Nugent, R., McLeod, A., 2004 Transboundary Animal Diseases: Assessment of Socio-Economic Impacts and Institutional Responses, at: http://www.fao.org/ag/againfo/resources/en/ Available publications/sector discuss/pp nr9 final.pdf Perry, B.D., Randolph, T.F., 2004 Integrated epidemiology and economics modelling for the management of animal health Dev Biol (Basel) 119, 389–402 Peyre, M., Fusheng, G., Desvaux, S., Roger, F., 2009 Avian influenza vaccines: a practical review in relation to their application in the field with a focus on the Asian experience Epidemiol Infect 137, 1–21 Rushton, J., Upton, M., 2006 Investment in preventing and preparing for biological emergencies and disasters: social and economic costs of disasters versus costs of surveillance and response preparedness Rev Sci Tech 25, 375–388 Swayne, D.E., 2009 Avian Influenza John Wiley & Sons Tiensin, T., Nielen, M., Vernooij, H., Songserm, T., Kalpravidh, W., Chotiprasatintara, S., Chaisingh, A., Wongkasemjit, S., Chanachai, K., Thanapongtham, W., 2007 Transmission of the highly pathogenic avian influenza virus H5N1 within flocks during the 2004 epidemic in Thailand J Infect Dis 196, 1679–1684 Tomassen, F., De Koeijer, A., Mourits, M., Dekker, A., Bouma, A., Huirne, R., 2002 A decision-tree to optimise control measures during the early stage of a foot-and-mouth disease epidemic Prev Vet Med 54, 301–324 Yamamoto, T., Tsutsui, T., Nishiguchi, A., Kobayashi, S., 2008 Evaluation of surveillance strategies for bovine brucellosis in Japan using a simulation model Prev Vet Med 86, 57–74  ... the analysis and interpretation of surveillance data as an important aspect of surveillance The analysis and interpretation of surveillance data often imply estimations of epidemiological variables... enough to warrant a reaction Planning an intervention for any type of unknown hazards poses a considerable challenge for animal health services, because no information is available about the nature... multi-factorial complexity of decision-making Keeping other factors constant, this article aims to provide a rationale for measurement of the effectiveness of information produced by animal health surveillance