Evaluation of information systems in health care: a framework and its application Hannu Salmela and Pekka Turunen Turku School of Economics and Business Administration Rehtorinpellonkatu 3, 20500 Turku FINLAND The objective of this study is to develop a framework for assessing the costs and benefits of information systems in health care The framework combines views from different disciplines, such as information systems evaluation, medical informatics, and health economics It suggests that the impact of health care information system should be assessed on multiple levels: the quality of medical information, the quality of diagnostic decisions and, ultimately, the quality of health care services To illustrate the application of the framework, an evaluation plan is developed for an information system called Computer Assisted Notification of Drug Effects on Laboratory Tests (CANDELA) We assume that the framework and the evaluation plan benefits researchers and practitioners in the evaluation of similar systems Introduction While information technology (IT) expenditure in hospitals is increasing, the effects of these investments on health care services have not been extensively studied (Ovid MEDILINE 1966-1997, Ingasoll et al 1990; van der Loo et al 1995) Although individual studies have suggested a positive relationship between the level of IS investments and the productivity of health care services (c.f Menon et al 1996), the overall results of IT investment profitability studies have been inconclusive (Mitra & Karim 1996 p 29-31) On the other hand, general IT investment productivity does not guarantee the productivity of a single health care information system In many ways, the evaluation of information systems in health care faces similar challenges as the evaluation of IS in other types of organizations Costs are often indirect and difficult to measure Organizational impacts and benefits, on the other hand, are often intangible and their realisation may take a long time (Saarinen 1993; see also Ives et al 1983) Hence, the key principles of the framework presented in this paper are derived from IS evaluation literature Both the improvements in information, improvements in individual decisions/actions, as well as improvements in organization level can indicate IS effectiveness and success (DeLone & McLean 1992) The selection between these measures is to a large degree dependent on the values and objectives driving the evaluation The type of the systems and the economic considerations of gathering the evaluation data need to be considered as well There are, however, additional challenges in IS evaluation in health care context Most notably, information systems can leverage improved treatment and consequently contribute to patients’ health Since the health impacts are difficult to evaluate in monetary units, the costs of the system have to be contrasted with improved "utility" or "outcome", rather than with monetary benefits To illustrate the application of evaluation measures, an evaluation plan is developed for an information system called Computer Assisted Notification of Drug Effects on Laboratory Tests (CANDELA) The system encodes and links pharmacological drug interference information into a laboratory information system (Grönroos et al 1995 a; Grönroos et al 1995 b; Grönroos et al 1997) It assists physicians in interpreting laboratory analysis results and has potential both to reduce costs and to improve the quality of health care services The system is currently being implemented in the University Hospital of Turku Its systematic evaluation will start in the fall 1997 Evaluation of information systems The evaluation of the effectiveness of an information system constitutes one of the key issues in information systems research In research, well-defined outcome measures are needed to ensure that the results from different studies are comparable They are a prerequisite for information systems research to make a contribution to IS practice In practice, success measures are needed to evaluate IS practice, policies and procedures (DeLone & McLean 1992) While a single measure of IS success or IS effectiveness would certainly be desirable, it seems unlikely that such a measure could be found Instead, research has provided taxonomies of success variables, which can be applied in different situations (DeLone & McLean 1992; Grover et al 1996) In general, the success of an information system can be evaluated through 1) The quality of information provided to the users, 2) The impact of IS on users’ thinking, decisions or actions, and 3) The impact of IS on organization level costs and benefits 2.1 User sastisfaction Due to the difficulty to assess IS impacts on individuals or organizations measures based on user perceptions have become prominent within IS literature (Galletta & Lederer 1989) User information satisfaction (UIS) is probably the most widely used single measure of IS success The original user satisfaction instrument contained a set of 39 factors (Bailey & Pearson 1983) The satisfaction of users was calculated as the sum of user’s positive and negative attitudes to these factors Other researchers have later developed shortened and modified versions of UIS (Ives et al 1983; Doll & Torkzadeh 1988; Saarinen 1993) so much that in 1989, Miller identified 12 different UIS instruments (Miller 1989) Some researchers consider UIS as being related to higher systems use, which in turn is related to higher individual and organizational performance (comp DeLone & McLean 1994; Grover et al 1996; Scott 1994) However, user satisfaction should be seen as a signal of acceptance of users rather than a measure of organizational outcomes While organizational effectiveness measures focus on actual outcomes, user satisfaction focuses on process Thus UIS is likely to be more useful in finding critical problems of IS implementation or use process than in evaluating the organizational outcomes 2.2 Individual impact Some researchers have tried to develop more direct measures of the impact of an IS on users' learning and decision making Ultimately, individual impact should be measured on the basis of whether information causes the receiver to change his or her behaviour (Mason 1978) Although questionnaires can be used here as well, many studies have relied on laboratory tests where the impact of IS on decision processes can be directly observed (Dickson et al 1977; O'Keefe 1989) A limitation of laboratory tests is that they fail to take into account the real world environment where the information system will be used (O'Keefe 1989) 2.3 Organizational outcome Measures of IS costs and benefits have been more common in practice than in research Academic researchers have tended to avoid organizational performance measures because of the difficulty of isolating the effect of IS effort from other effects which influence organizational performance (Grover et al 1996) In practice, ex-ante evaluations have been more frequently while academics have preferred ex-post evaluation (Parker et al 1988) For research purposes, evaluation of IS cost-effectiveness is costly and inhibits comparisons between different studies (Ives et al 1983 p 785-786; Saarinen 1993) Thus, standard IS evaluation methods are needed in IS research Expressing impacts in monetary terms places additional challenges on evaluation Some of the organizational impacts of information systems, such as improvements in products and services or improved management are often intangible Traditional accounting systems rarely provide the information needed to evaluate the costs and benefits associated with a particular IS (Matlin 1979) Hence, many researchers have proposed methodologies to estimate the actual contribution of IS on firm performance (see Grover et al 1996) DeLone and McLean (1992) conclude, however, that much work still needs to be done in this area It appears that a comprehensive evaluation of an information system requires multiple measures (DeLone & McLean 1992; Saarinen 1993 p 51) Both the improvements in information, improvements in individual decisions/actions, as well as improvements in organization level can indicate IS effectiveness and success The selection between these measures is to a large degree dependent on the values and objectives driving the evaluation The type of the systems and the economic considerations of gathering the evaluation data need to be considered as well Evaluation of information systems in health care In many ways, evaluation of information systems in health care is no different from that in other types of organizations There are, however, additional challenges in IS evaluation in health care context Most notably, information systems can leverage improved treatment and consequently contribute to patients' health Because of the potential impacts on patients' lives more strict measurements are used to evaluate health care information systems 3.1 User satisfaction User satisfaction has been used in evaluating health care information systems (for example Gardner & Lundsgaarde 1994) A number of studies have also applied UIS instruments developed in information systems science Pearson’s original user satisfaction measure has been applied in the evaluation of Hospital information systems (HIS) in 160 Veterans Administration Medical Centers (Bailey 1990 p 51) That study is further adapted in the Clinical Computerised Information System (Pugh & Tan 1994) Bailey and Pearson’s (1983) user satisfaction measure has been applied into HIS DSS (Dupuits & Hasman 1995) Chin and McClure (1995 p 717-721) have used Doll and Torkzadeh’s UIS instrument to evaluate clinical information systems The more strict approach to evaluation is, however, reflected in the role that UIS measures have in health care IS evaluation The role of user satisfaction has not been as prominent as in general IS research In health care context, only about % of the studies used user satisfaction effect measure, whereas in IS research the fraction is 20% (van der Loo et al 1994; Grover et al 1996) Hence, although UIS is seen as providing insights into the usefulness of the system as perceived by users, in health care it is not very widely used as a surrogate measure for systems effectiveness Instead, more direct measures about the impacts on patients’ health are used Surprisingly, user satisfaction measures have not been used in the evaluation of supporting and auxiliary type of information system either (see van der Loo et al 1994, p 50) Such systems are used in making appointments, and in managing documentation, financial transactions and personnel information They support patient’s welfare only extremely indirectly (van der Loo et al 1994 p 47 and 50) 3.2 Individual impact The impact of an information system on decision making, particularly on diagnostic and treatment decisions, has been a common basis for evaluation (for example Maria et al 1994; Wagner & Cooper 1995) A typical evaluation design is based on two groups where one group uses the information system while the other doesn’t A recent study that classified evaluative studies in health care found that about 64 % of evaluations were based on two group comparative study (van der Loo et al p 49) 3.3 Organizational outcome In many cases, a demonstrated impact on treatment effectiveness is considered as a sufficient criteria for IS success It is seen as adequate justification for widespread systems implementation and use The value of health outcomes should, however, be assessed in order to ensure economic use of hospital resources and optimal health care services for patients In health economics, several concepts and methods have been developed to assist in evaluating the value of improved health outcomes The three main types of cost and utility measures used in health economics are: cost-effectiveness, cost-utility and cost-benefit analysis In cost-effectiveness analysis the patient outcomes are measured in the most appropriate natural or physical units such as: life years gained, disabilitydays saved, points of blood pressure reduction Analysis is appropriate when a treatment has only one effect on patient’s health Such analysis can be supported with ratios such as cost per life-year gained or reverse life-years gained per dollar spent (Drummond et al 1987) In cost-utility analysis the patient’s health effects are expressed as quality adjusted life-years In costutility analysis even multiple health effects of treatment, not necessarily common to both alternatives, can be compared In cost-benefit analysis both the costs and health outcomes are estimated in monetary units (Drummond et al 1987) Cost-benefit studies have been more frequent in the evaluation of IS in health care than in IS evaluation in general In health care context, 13 % of the evaluative studies used cost-benefit analysis This is still considerably more than in IS research, where the fraction is only 4% (van der Loo et al 1994 and Grover et al 1996) The cost-benefit type of analysis used in information system science have had only a limited affect on this type of evaluation in health care There are some studies in the evaluation of health care information systems where cost-benefit and cost-effectiveness terms have used from point of view of information systems science (Zielstroff 1985) In information systems science intangible effects and costs are considered as a difficult problem (Saarinen 1993), but in health economics those things are not seen as insurmountable problems In information system science intangible costs and different types of effects are listed in evaluation reports (Johnston & Vitale 1988) But there are only few attempts to quantify intangible benefits and translate them into measurable comparable values (like Money et al 1988) In this area more models could be adopted from the cost-utility analysis in health economics They could facilitate comparison of several information systems Such analyses have not been really common in information science One difference between information system science and health economics is that the cost and utility analysis for information systems are more often exante than in health care In health care physicians want to develop and give best possible treatments The associated costs are thought only afterwards A framework for evaluating medical alert systems A medical alert system is defined as an information system which supports physicians in making diagnostic and treatment decisions by providing alerts based on information in clinical databases (e.g patient information, patients medication profile, drug information, treatment information) The core of such system is a knowledge base, which contains medical research results about various interactions that need to be taken into account in diagnostic or treatment decisions The basic reason for using such systems is that an individual physician can rarely master all possible information of different interactions while making routine clinical decisions The evaluation of an alert system is a challenging task The following framework attempts to provide a comprehensive view of the costs and impacts of such system Thus, it assists hospital management in evaluating the costs and benefits of implementing alert systems It also assists the developers of alert systems in evaluating system impacts and thus in improving the system and justifying its widespread use 4.1 Evaluating the costs of medical alert systems Figure represents the framework for evaluating the costs associated with developing, implementing, using and maintaining an alert system It is assumed that these processes use four different types of resources: 1) IS personnel time, 2) hospital personnel time, 3) infrastructure and 4) capital In the case of medical alert systems, the IS personnel cost and the hospital personnel cost are fairly easy to estimate The development and maintenance of software is usually done by external medical software vendors The associated cost is the amount that a hospital pays for the software vendor as software licenses and maintenance fees In the future, some hospitals specialise in maintaing knowledge databases in specific areas Other hospitals can then purchase new versions of knowledge database against predefined fee Thus, the hospital personnel costs as well can be easily evaluated It is important to note, however, that the implementation and use of an alert system in a hospital is likely to require some time of physicians Obviously, the time of physicians should not be considered as a free resource RESOURCE IS personnel time The development and maintenance of alert systems COSTS IS personnel wages, vendor payments Hospital personnel time Hospital personnel wages Hardware; supplies; office space Overhead costs Capital employed in IS/IT Interest costs Figure 1: Overall costs of developing and maintaining alert information systems Electronic databases are a prerequisite for the use of alert systems Investments needed in developing and maintaing electronic patient databases, electronic laboratory test databases, etc far exceed the direct costs of developing alert systems These costs are infrastructural in the sense that there are many other applications that use the databases The way these infrastructural costs should be taken into account in evaluation is likely vary from one hospital to another Finally, if the initial investment is significant, the cost of capital employed in the project should also be considered 4.2 Evaluating the benefits of medical alert systems Figure represents the model for analysing the impact of alert system on hospital operations The model assumes that an alert information system can reduce the time of physician in analysing potential interactions Alternatively, it can improve the quality of interaction information, which leads to better clinical decisions and thus to cost savings or improved health outcomes Finally, statistics about the frequency and impacts of different alerts can result to individual and organizational learning INFORMATION IMPACT In the absence of alert systems, physicians use their personal experience and judgement in evaluating the possible drug and medication inferences They consult colleagues, laboratory physicians, books and articles to consider the potential inferences while making clinical decisions Since an alert may reduce the need for such consultation, it has the potential to save physician time in making clinical decisions The main objective in implementing alert systems is to improve the quality of clinical decisions The evaluation should demonstrate that the implementation of an alert system has resulted to a permanent change in clinical decision making It should also demonstrate a change in the use of laboratory analyses and medication when diagnosing or treating a particular illness The evaluation of whether the alert system impacts the quality of clinical decisions can be made in a number of ways Subjective assessment of physicians is one alternative Another approach is to use laboratory tests using two groups of physicians to solve a number of clinical decision problems, one group using the alert system while the other one relying on their own judgement Ultimately, however, the evaluation should be based on the analysis of hospital records about actual clinical decisions DECISION IMPACT No information impact Alert information system Improved medical information Improved diagnosis / treatment decisions Improved statistics Figure 2: Evaluation of alert/reminder system impacts ORGANISATIONAL IMPACT BENEFIT Resource savings in processing medical data Cost savings (information processing) Resource savings in operations Cost savings in hospital operations Improved treatment Health outcomes/ patient utility Improved evaluation of treatment Individual/ organizational learning Reduced need for laboratory experiments or medication can lead to cost savings and/or improved health outcomes In general, cost savings can be expressed in monetary terms Most hospitals already have cost estimates for a laboratory test, a particular medication or a patient day in the hospital In evaluating the impact on health outcomes or utility for patients, subjective evaluation of the physicians is perhaps most common Methods from health economics could, however, be applied to get more objective evaluation measures Evaluation plan information system Finally, it is possible that alert systems can constitute an effective means to foster individual and organizational learning about different interactions Thus, physicians may learn about new interactions and their influence on clinical decisions In the long run, this may be an important outcome of implementing an alert system Evaluation is also expected facilitate systems implementation Because of the diversity of alerts, it seems necessary that the evaluation process leads to a clear view about which of the CANDELA alerts are most valuable By doing so, evaluation assists in withholding some of the less valuable alerts from laboratory reports and thus in reducing information overflow CANDELA information system CANDELA is an information system that automatically gives alerts of important drug effects and drug interactions (other than therapeutic) on laboratory tests It plays an active role in assisting physicians in interpreting laboratory analysis results and has potential both to reduce costs and to improve the quality of health care services CANDELA is based on a database that contains a large number of rules about how different drugs interact on laboratory tests The system is connected to an electronic patient database that contains information about the medication of individual patients Thus, for each laboratory test the system automatically checks whether patients medication profile could interfere with the test results The ward physician responsible for the treatment immediately evaluates on-line alarms of potential drug interactions (Grönroos et al 1995 a; Grönroos et al 1995 b; Grönroos et al 1997) In its current form, CANDELA generates alerts that are automatically printed to laboratory test reports Hence, the physicians are not direct users of the system, and they can not directly ask for explanations about the alerts If a physician wants further information, he or she can use the system interactively from a terminal In general, CANDELA represents a fairly new type of clinical information system A somewhat similar system (HELP) has been in use in a hospital in Salt Lake City, Utah User attitudes towards HELP have also been evaluated (Gardner & Lundsgaarde 1994) As more and more information about patients, drugs, laboratory tests etc is converted to electronic form, developing alert and reminder systems will become easier Hence, the ability to evaluate such systems becomes increasingly important for CANDELA The CANDELA information system will be implemented in Turku University Central Hospital (TUCH) during the Fall 1997 The evaluation of CANDELA is projected to take place during Fall 1997 and Spring 1998 The objective of evaluation is to identify and demonstrate the cost-effectiveness of CANDELA This is considered as an important prerequisite for its widespread use in other hospitals Based on the framework presented in figure 2, the essential questions for CANDELA evaluation can now be stated as follows: does CANDELA improve the quality of information about drug interactions on laboratory tests? does CANDELA have an impact on clinical decisions that are based on laboratory tests? does the benefits of using CANDELA exceed the costs associated with it? Hence, the objective is to evaluate the CANDELA system on multiple levels In the following, a plan for the evaluation is outlined 6.1 User satisfaction The impact of CANDELA on information quality will be evaluated using Doll and Torkzadeh's (1988) UIS instrument This particular instrument was selected because its reliability has been tested in previous studies (Torkzadeh & Doll 1991; Hendrickson et al 1994; Torkzadeh et al 1994) The instrument contains key questions of user satisfaction on information and its quality The developers of the instrument consider it as a potential surrogate measure for utility in decision-making (Doll & Torkzadeh 1988) The questionnaire will be sent to physicians few months after the system has been taken to use The analysis aims at identifying the departments that were most satisfied with the information The impact of variables such as age, job experience, specialisation, etc on the perceived value of alert information is analysed 6.2 Invidual impact The impact of CANDELA on clinical decisions will be analysed using two methods To get qualitative data about the impact of alerts, some physicians are asked to re-evaluate their previous clinical decisions (that they had made prior systems implementation) The question is, whether their decision would have been different if CANDELA alerts had been available at the time they made the decision The second analysis relies on perceptions of physicians as they use the alerts for interpreting laboratory tests For each alert, physicians are asked to evaluate the degree to which the alert was relevant and the degree to which it changed their clinical decision This analysis assists in identifying the alerts that are most influential in decision making To improve the validity of analysis, the results concerning different alerts are later evaluated and discussed with physicians, who are considered as best experts in their field 6.3 Organizational outcome The evaluation of organizational outcomes is based on the analysis of historical data about clinical decisions The hospital has history records about laboratory tests, patients’ medical profiles and associated clinical decisions from the past two years The analysis of this data can reveal how frequently physicians have misinterpreted laboratory tests Hence, it assists in estimating the potential impact that CANDELA can have on clinical decisions implementation of CANDELA will reduce the frequency of inaccurate clinical decisions in certain medical areas In this study, the primary objective is to validate these impacts through history data Comparisons with other hospitals are not planned to be made in the first round of evaluation CANDELA information system is telling to physicians responsible for treatment what is true about patient What is true about patient, can theoretically be made without consideration of cost and benefit (comp Shortliffe 1987 p 62-64) But it would be more useful if information from cost and effects would be in future build into Computer Assisted Notification of Drug Effects on Laboratory Tests information system Conclusions and future research The framework for evaluating medical alert systems benefits researchers Its contribution is as a new means to help them study the impacts of such systems in clinical decision making The framework presented here can also be used in explaining why some alert systems provide more value in terms of cost savings and health outcomes than others Practitioners can use the framework to review the potential benefits of alert systems in their own hospital They can establish objectives for implementing such systems And they can use the framework as an investigative tool in analysing reasons for low organizational impact of an implemented alert system The analysis of improved health outcomes relies largely on the expert opinion of physicians Some of the drug interactions on laboratory results may have lead to inconclusive diagnosis and to further experiments and medication Unnecessary experiments or medication is likely influence patients’ health The value of avoiding these tests can be truly significant for an individual patient The evaluation of health impacts is based on expert opinion of physicians Health economics evaluation methods will be applied as well This paper describes the evaluation plan for CANDELA Its systematic evaluation will start simultaneously The first preliminary evaluation results will be available in October 1997 The full evaluation of the system is to be completed by May 1998 Based on the experiences gained in CANDELA evaluation the framework and methods will be improved and then applied in other medical information systems projects Evaluating the costs of CANDELA is primarily based on the license prices and maintenance fees for the software and for the drug interaction database Subjective evaluation of the physicians will be used to estimate whether the use of the system has increased the time they need for analysing laboratory results and making clinical decisions The development costs of electronic patient database and electronic laboratory database are considered as infrastructural It seems difficult to assign any monetary value for the use of the databases This study is part of larger research program "Road ahead in Medical Informatics" We would like to express our gratitude to the Academy of Finland for their financial support to this program and to our research In general, the evaluation of CANDELA is based on historical comparisons It is believed that the Acknowledgements References Bailey J E (1990) Development of an Instrument for the Management of Computer User Attitudes in Hospitals Methods of Information in Medicine 1990:1, 51-56 Bailey J.E - Pearson S.W (1983) Development of a tool for measuring and analyzing computer user satisfaction Management Science 1983:5, 530-545 Chin, H.L - McClure, P (1995) Evaluating a Comprehensive Outpatient Clinical Information System: A Case Study and Model for System Evaluation Proceedings Nighteenth Annual Symposium on Computer Applications in Medical Care JAMIA 1995 Lousiana DeLone, W.H - McLean, E.R (1992) Information Systems Success: The Quest for the Dependent Variable Information Systems Research 1992:1, 6095 Dickson, G - Chervany N.- Senn J (1977) Research in MIS: The Minnesota Experiments Management Science 1977:5, 913-923 Doll, W.J - Torkzadeh, G (1988) The Measurement of End-User Computing Satisfaction MIS quarterly 1988:2, 259-273 Drummond, M.F - Stoddart, G.L - Torrance, G.W (1987) Methods for the Economic Evaluation of Health Care Programmes Oxford University Press: Oxford Dupuits, F.M.H.M - Hasman, A (1995) User satisfaction of general practioners with HIOS+, a medical decision support system Computer Methods and Programs in Biomedicine 1995:2, 183-188 Galetta, D.F - Lederer, A.L (1989) Some Cautions on the Measurement of User Information Satisfaction Decision Sciences 1989:3, 419-439 Gardner, R.M - Lundsgaarde, H.P (1994) Evaluation of User acceptance of a Clinical Expert System JAMIA 1994:6, 428-438 Grönroos, P - Irjala, K - Forström, J.J (1995a) Coding Drug Effets on Laboratory tests for Health Care Information Systems JAMIA Symposium Supplement, SCAMC Proceedings 1995, 449-453 Philadelphia Grönroos, P - Irjala, K - Heiskanen, J - Torniainen, K - Forström J.J (1995b) Using computerized individiual medication data to detect drug effects on clinical laboratory tests Scandinavian journal of clinical & laboratory investigation 1995:55, 31-36 (Suppl 222) Grönroos, P.E - Irjala, K.M - Huupponen, R.K Scheinin, H - Forström, J - Forström, J.J (1997) A Medication Database - A Tool for Detecting Drug Interactions in Hospital European Journal of Clinical Pharmalogy (in press) Grover, V - Jeong, S.R - Segars, A.H (1996) Information systems effectiveness: The construct space and patterns of application Information & Management 1996:4, 117-191 Hendrickson, A.R - Glorfeld, K - Cronan T.P (1994) On the Repated Test-Retest Reliability of the End-User Computing Satisfaction Instrument: A Comment Decision Science 1994:4, 655-665 Ingasol, G L - Hoffart, N -Schultz, A W (1990) Health services research in nursing: current status and future directions Nursing Economics 1990:8, 229-238 Ives, B - Olson, M.H - Baroudi, J.J (1983) The Measurement of User Information Satisfaction Communications of the ACM 1983:10, 785-793 Johnston, R.H - Vitale, M.R (1988) Creating Competitive Advantage With Interorganizational Information Systems MIS Quarterly 1988:2, 153165 van der Loo, R.P - van Gennip, E.M.S.J - Baker, A.R (1995) Evaluation of automated information systems in health care: an approach to classifying evaluative studies Computer Methods and Programs in Biomedicine 1995:48, 45-52 Maria B.L - Lambay, F.A - Dankel II, D Chakravarthy, S - Tufekci S - Marcus, R -Kedar, A (1994) XNEOr: Development and Evaluation of an Expert System to Improve the quality and Cost of Decision-Making in Neuro-Oncology Proceedings Eighteenth Annual Symposium on Computer Applications in Medical Care JAMIA 1994, 678683 Washington, DC Mason, R.O (1978) Measuring Information Output: A Communication System Approach Information & Management 1978:5, 219234 Matlin, G (1979) What Is the Value of Investment in Information Systems? MIS Quarterly 1979:3, 5-34 Menon, N.M - Lee, B - Eldenberg, L (1996) Information Technology Productivity in the Health Care Industry Proceedings of he Seventeenth International Conference on Information Systems 1996, 477 Cleveland Miller J (1989) Information systems effectiveness: The fit between business Needs and system capabilities Proceedings of the Tenth Internatinal Conference on Information Systems 1989, 273-288 Boston Massachusetts (edirors DeGross J.I., Henderson J.C and Konsynski B.R.) Mitra, S - Karim, C.A (1996) Analyzing Costeffectiveness of organizations: The Impact of Information Technology Spending Journal of Management Information Systems 1996:2, 29-57 Money A - Tromp D - Wegner T (1988) The Quantification of Decision Support Benefits Within The Context of Value Analysis MIS Quarterly 1988:2, 223-236 O’Keefe, R.M (1989) The Evaluation of DecisionAiding Systems: Guidelines and Methods Information & Management 1989:17, 217-226 Ovid MEDILINE (1966-1997) Parker, M.M - Benson, R.J - Trainor, H.E (1988) Information ecomics Prentice-Hall Inc: New Jersey Pugh G E - Tan J K H (1994) Conputerized Databases for Emergency Care: What Impact on Patient Care? Methods of Information in Medicine 1994:5, 503-517 Saarinen T (1993) Success of information systems Evaluation of Development Projects and the Choice of Procurement and Implementation Strategies Acta Academiae Oeconomicae Helsingiensis Dissertation A:88:1993 Helskinki Sanders, L.G - Courtney, J.F (1985) A Field Study of Organizatinal Factors Influencing DSS Success MIS Quarterly.1985:1, 77-93 Scott, J.E (1994) The Measurement of Information Systems Effectiveness: Evaluating A Measuring Instrument Proceedings of the 15th ICIS 111-128 Vancouver Shortliffe E.H (1987) Computer Programs to Support Clinical Decision Making JAMA 1987:1, 61-66 Torkzadeh, G - Doll, W.J (1991) Test-retest reliability of the end-user computing satisfaction instrument Decision Sciences 1991:1, 26-37 Torkzadeh, G - Xia, W - Doll, W.J (1994) A Confirmatory Factor Analysis of the End-User Computing Satisfaction Instrument MIS Quarterly 1994:4, 453-461 Wagner, M.M - Cooper, G.F (1995) Evaluation of a Belief-Network-Based Reminder System that Learns from Utility Feedback Proceedings of the Nineteenth Annual Symposium on Computer Applications in Medical Care JAMIA 1995, 666672 Lousiana Zielstroff R.D (1985) Cost Effectiveness of Computerization in Nursing Practice and Administration The Journal of Nursing Administration 1985:2, 22-26 ... Proceedings of the 15 th ICIS 11 1 -12 8 Vancouver Shortliffe E.H (19 87) Computer Programs to Support Clinical Decision Making JAMA 19 87 :1, 61- 66 Torkzadeh, G - Doll, W.J (19 91) Test-retest reliability... References Bailey J E (19 90) Development of an Instrument for the Management of Computer User Attitudes in Hospitals Methods of Information in Medicine 19 90 :1, 51- 56 Bailey J.E - Pearson S.W (19 83)... Segars, A.H (19 96) Information systems effectiveness: The construct space and patterns of application Information & Management 19 96:4, 11 7 -19 1 Hendrickson, A.R - Glorfeld, K - Cronan T.P (19 94) On