Analysis of impacts of cardinality and diversity on collective performance

WROCŁAW UNIVERSITY OF SCIENCE AND TECHNOLOGY FACULTY OF COMPUTER SCIENCE AND MANAGEMENT Ph.D Dissertation ANALYSIS OF THE IMPACTS OF CARDINALITY AND DIVERSITY ON COLLECTIVE PERFORMANCE Van Du Nguyen Supervisor: Prof Dr Sc Ph.D Ngoc Thanh Nguyen Co-supervisor: Assoc Prof., Ph.D Mercedes G Merayo (Universidad Complutense de Madrid, Spain) Wrocław 2018 ACKNOWLEDGMENTS This thesis is the end of the 8-year journey of studying abroad at Wrocław University of Science and Technology, Poland (including master study) Indeed, it was an interesting journey for an international student like me Beyond new knowledge that I have been learning, I have also found opportunities to explore new cultures, European cultures In this 8-year journey, first, I would like to express my sincere gratitude to my supervisor Professor Ngoc Thanh Nguyen for the continuous support of my study and related research, for his patience, motivation, and immense knowledge It could be said that I have had a great opportunity to work with him Second, I would also like to thank Associate Professor Mercedes G Merayo, my cosupervisor from Universidad Complutense de Madrid, Spain for the support of my thesis I also sincerely thank Professor Manuel Núñez, Universidad Complutense de Madrid, Spain for giving me an opportunity to participate the project S2013/ICE-3006 SICOMORo-CM Third, I also sincerely thank the colleagues in the Department of Information Systems, Faculty of Computer Science and Management for all their helpful comments on my thesis Next, I would also like to thank Professor Dosam Hwang and the colleagues in Knowledge Engineering Laboratory, Department of Computer Engineering Yeungnam University, the Republic of Korea for their invitation and support during 2-month working in the BK21+ project Besides, I also sincerely thank my friends, who have encouraged and supported me during the time of Ph.D study Finally, I cordially thank my family, especially my Mom and my Dad, for encouraging and believing me during the years of studying abroad i “It may be, in the end, that a good society is defined more by how people treat strangers than by how they treat those they know.” ― James Surowiecki (2005), The Wisdom of Crowds “Being different is as important as being good.” ― Scott E Page (2007), The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies "Great things in business are never done by one person They are done by a team of people." ― Steve Jobs, Ex-Apple CEO “If you want to have good ideas you must have many ideas.” ― Linus Pauling, An American Quantum Chemist and Biochemist “Talent wins games, but teamwork and intelligence wins championships.” ― Michael Jordan, An American retired professional basketball player, businessman, and principal owner and chairman of the Charlotte Hornets of the National Basketball Association(NBA) This thesis is dedicated to my parents, Thi Thien Le & Van Lo Nguyen For their endless love, support, and encouragement From their son, Van Du Nguyen ii INDEX ACKNOWLEDGMENTS i INDEX iii LIST OF FIGURES vii LIST OF TABLES x CHAPTER INTRODUCTION 1.1 Collective Intelligence 1.2 Criteria of intelligent collectives 1.2.1 Diversity 1.2.2 Independence 1.2.3 Decentralization 1.2.4 Aggregation 1.3 Applications of Collective Intelligence 1.3.1 Prediction Markets 1.3.2 Delphi Method 10 1.4 Motivation of thesis 12 1.5 Research problem 13 1.5.1 Aim of the thesis 13 1.5.2 Hypotheses 14 1.5.3 Objectives of the thesis 14 1.6 Contributions of thesis 14 1.7 Thesis organization 17 CHAPTER LITERATURE REVIEW 19 2.1 Basic notions 19 2.1.1 Collective 19 2.1.2 Collective prediction determination 21 iii 2.1.3 The measures of collective performance 25 2.1.3.1 Based on the accuracy of collective prediction 25 2.1.3.2 Based on win ratio 26 2.1.3.3 Based on the quotient of collective error and individual errors 26 2.1.4 The measures of diversity 29 2.2 Relationship between collective prediction and individual predictions 33 2.3 The impact of cardinality on collective performance 34 2.4 The impact of diversity on collective performance 35 2.4.1 Diversity in the composition of collective members 35 2.4.2 Diversity of individual predictions 38 CHAPTER THE IMPACT OF CARDINALITY ON COLLECTIVE PERFORMANCE 40 3.1 Introduction 40 3.2 Research model and hypothesis 41 3.3 Simulation experiments and their evaluation 43 3.3.1 Simulation design 43 3.3.2 Experimental results 44 3.4 Analysis of collective performance 53 3.5 Determining collective predictions of big collectives 60 3.5.1 Introduction 61 3.5.2 Two-stage integration approach to collective prediction determination 63 3.5.3 Experiments and their evaluation 68 3.5.3.1 Datasets 68 3.5.3.2 Experimental results and their evaluation 69 CHAPTER THE IMPACT OF DIVERSITY ON COLLECTIVE PERFORMANCE 75 4.1 Introduction 75 4.2 General research model 76 4.3 Collectives with the same cardinality 78 iv 4.3.1 Simulation design 79 4.3.2 Simulation results 81 4.3.2.1 Collectives with the same cardinality but with different diversity levels 85 4.3.2.2 Collectives with the same diversity level but with different cardinalities 87 4.4 Collectives with different cardinalities 90 4.4.1 Research model and hypotheses 90 4.4.2 Simulation design 96 4.4.3 Simulation experiments and their evaluation 96 4.5 Analysis of collective performance 101 4.6 Modifying individual predictions and collective performance 113 4.6.1 Research model 113 4.6.2 Simulation design 117 4.6.3 Simulation results and their evaluation 117 4.6.3.1 Updating individual predictions 118 4.6.3.2 Removing individual predictions 124 CHAPTER EVALUATING THE INTELLIGENCE DEGREE OF A COLLECTIVE 131 5.1 Introduction 131 5.2 The measure 134 5.3 Impact of diversity on the intelligence degree of a collective 137 5.3.1 Research model 137 5.3.2 Simulation design 138 5.3.3 Simulation results 140 5.3.4 Statistical analysis 144 5.3.4.1 Scenario 1: collectives varying in diversity 144 5.3.4.2 Scenario 2: collectives varying in cardinality 148 5.4 Collective performance and the intelligence degree of a collective 150 5.4.1 Datasets 151 v 5.4.2 Experimental results and their evaluation 152 5.5 Analysis of the intelligence degree of a collective 155 CHAPTER CONCLUSIONS 164 6.1 Conclusions 164 6.2 Limitations 166 CHAPTER RECOMMENDATIONS FOR FUTURE RESEARCH 168 REFERENCES 172 vi LIST OF FIGURES Fig 1.1 General research model of Collective Intelligence Fig 2.1 Diff measure 26 Fig 2.2 The measures of diversity 30 Fig 2.3 Example of diversity measure 32 Fig 3.1 General model 43 Fig 3.2 Diff with cardinality ranging from to 250 [one-dimensional vector] 44 Fig 3.3 WR with cardinality ranging from to 250 [one-dimensional vector] 45 Fig 3.4 QIC with cardinality ranging from to 250 [one-dimensional vector] 45 Fig 3.5 Diff with some selected cardinalities [one-dimensional vector] 46 Fig 3.6 WR with some selected cardinalities [one-dimensional vector] 47 Fig 3.7 QIC with some selected cardinalities [one-dimensional vector] 47 Fig 3.8 Diff with cardinality ranging from to 250 [two-dimensional vector] 48 Fig 3.9 WR with cardinality ranging from to 250 [two-dimensional vector] 48 Fig 3.10 QIC with cardinality ranging from to 250 [two-dimensional vector] 49 Fig 3.11 Diff with some selected cardinality values [two-dimensional vector] 50 Fig 3.12 WR with some selected cardinality values [two-dimensional vector] 50 Fig 3.13 QIC with some selected cardinality values [two-dimensional vector] 51 Fig 3.14 Two-stage integration of collective prediction 63 Fig 3.15 The results in the second stage [O1-based collective prediction] 73 Fig 3.16 The results in the second stage [O2-based collective prediction] 74 Fig 4.1 General research model 77 Fig 4.2 Collective performance with cardinality of [c 3-based diversity] 81 Fig 4.3 Collective performance with cardinality of [c5-based diversity] 82 Fig 4.4 Collective performance with cardinality of 109 [c 3-based diversity] 82 Fig 4.5 Collective performance with cardinality of 109 [c 5-based diversity] 83 Fig 4.6 Collective performance with cardinality of 209 [c 3-based diversity] 83 Fig 4.7 Collective performance with cardinality of 209 [c 5-based diversity] 84 Fig 4.8 Diff with cardinalities of 9, 109, 209 [c3-based diversity] 87 Fig 4.9 QIC with cardinalities of 9, 109, 209 [c3-based diversity] 88 Fig 4.10 Diff with cardinalities of 9, 109, 209 [c5-based diversity] 89 Fig 4.11 QIC with cardinalities of 9, 109, 209 [c5-based diversity] 90 vii Fig.4.12 Research model of expanding cardinality 91 Fig 4.13 Collective performances with cardinalities ranging from to 209 [c 3-based diversity, diversity up] 97 Fig 4.14 Collective performances with cardinalities ranging from to 209 [c 3-based diversity, diversity down] 97 Fig 4.15 Collective performances with cardinalities ranging from to 209 [c 5-based diversity, diversity up] 98 Fig 4.16 Collective performances with cardinalities ranging from to 209 [c 5-based diversity, diversity down] 98 Fig 4.17 An example of the relationship between diversity and collective performance 102 Fig 4.18 Example of adding/removing a prediction to/in a collective 110 Fig 4.19 General research model 114 Fig 4.20 Results of updating predictions and cardinality of 59 [c3-based diversity] 118 Fig 4.21 Results of updating predictions and cardinality of 109 [c 3-based diversity] 119 Fig 4.22 Results of updating predictions and cardinality of 209 [c 3-based diversity] 119 Fig 4.23 Results of updating predictions and cardinality of 59 [c 5-based diversity] 121 Fig 4.24 Results of updating predictions and cardinality of 109 [c 5-based diversity] 122 Fig 4.25 Results of updating predictions and cardinality of 209 [c 5-based diversity] 122 Fig 4.26 Results of removing predictions and cardinality of 59 [c 3-based diversity] 125 Fig 4.27 Results of removing predictions and cardinality of 109 [c 3-based diversity] 125 Fig 4.28 Results of removing predictions and cardinality of 209 [c 3-based diversity] 125 Fig 4.29 Results of removing predictions and cardinality of 59 [c 5-based diversity] 127 Fig 4.30 Results of removing predictions and cardinality of 109 [c 5-based diversity] 128 Fig 4.31 Results of removing predictions and cardinality of 209 [c 5-based diversity] 128 Fig 5.1 An example of intelligence measures 132 Fig 5.2 General research model 137 Fig 5.3 Impact of diversity on the intelligence degree of a collective [β=0.5, c3-based diversity] 140 Fig 5.4 Impact of diversity on the intelligence degree of a collective [β=0.7, c3-based diversity] 141 Fig 5.5 Impact of diversity on the intelligence degree of a collective [β=0.9, c3-based diversity] 142 Fig 5.6 Impact of diversity on the intelligence degree of a collective [β=0.5, c5-based diversity] 142 viii Fig 5.7 Impact of diversity on the intelligence degree of a collective [β=0.7, c5-based diversity] 143 Fig 5.8 Impact of diversity on the intelligence degree of a collective [β=0.9, c5-based diversity] 143 Fig 5.9 Computational results on Cereal Vase 153 Fig 5.10 Computational results on Cow Weight 154 Fig 5.11 Computational results on Coin Jar 154 Fig 5.12 Computational results on Temp 155 ix Conclusions Then, we have investigated another aspect of the impact of diversity on collective performance by taking into consideration some modifications within a collective [84] as follows:  Updating predictions within collectives to reach higher levels of consistency (less diversity levels);  Removing predictions that causing the consistency levels of collectives to be higher (less diversity levels) By means of computational experiments, the modifications (such as by updating or removing predictions) causing the diversity level of a collective to be decreased will not lead to a better collective prediction In such situations, the collective performance decreases as the predictions of the collective become more consistent (less diverse) 4) To evaluate the intelligence degree of a collective: We have proposed a function that can be used for measuring the so-called the intelligence degree of a collective [82] The measure is not only based on the accuracy fo collective prediction but also based on the accuracies of individual predictions A comparison between collective performance and the proposed measure is also reported with some selected real datasets on guessing the total pieces of cereal in a weirdlooking glass vase, the weight of a cow, the value of coins in a huge jar, and the current temperature in a classroom Also, we have investigated the impact of diversity on the intelligence degree of a collective by taking into account cardinality By means of computational experiments, the diversity is positively associated with the intelligence degree of a collective Some theorems concerning the influence of adding or removing predictions on the intelligence degree of a collective have also formally proved 6.2 Limitations 166 Conclusions In general, we have found the significant impacts of cardinality and diversity on collective performance However, there are many aspects, which have not been taken into consideration as follows: 1) We have only assumed two representations of individual predictions including one-dimensional vector and two-dimensional vector However, in some situations, interval values can be used as predictions on a given problem 2) Most results are based simulation experiments There exists a need to conduct more comprehensive experiments in which collective members can be groups of people or members of crowdsourcing systems like Amazon Mechanical Turk 3) Diversity in the composition of collective members was not taken into account Although, it has been argued that the diversity of individual predictions can be enhanced by the existence of the diversity in the composition of collective members 4) We did not propose a threshold of Int value in which a collective is called intelligent For example, a collective can be considered intelligent if its Int value is equal to 0.5 This research problem will be extensively investigated to find out that a threshold 167 CHAPTER RECOMMENDATIONS FOR FUTURE RESEARCH First, this chapter briefly presents recent research on applying Collective Intelligence with other research fields such as Machine Learning and Social Networks Then we will discuss possible further research According to the definition by Pierre Levy [62], it can be noted that Collective Intelligence is mainly based on the notion of knowledge collectively becoming more than the sum of individual knowledge With this assumption, by using Crowdsourcing for solving some problems in the real world, one can tap into the Collective Intelligence of crowds According to [20], Crowdsourcing, which is an online, distributed problem-solving and production model [17], can be considered as a special case of Collective Intelligence The term Crowdsourcing was introduced by Jeff Howe and Mark Robinson (editors at Wired) in 2005 Firstly, it aims at describing how businesses use the Internet to "outsource work to the crowd." To date, there exist many definitions of Crowdsourcing in the scientific and popular literature However, the most cited one is as follows [48]: "The act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call." Besides, it has been proved that Crowdsourcing is helpful in leveraging the distributed knowledge found in crowds and changing the way groups of people produce knowledge, generate ideas, and make them actionable [18] In recent years, advances in information technologies such as virtual communities, social web, and information sharing have given an additional source of information referred as the wisdom of crowds to Information Retrieval and Machine Learning systems Indeed, intelligent systems have significantly benefited by exploiting big data to incorporate massive volumes of data to improve their solutions to given problems in 168 Recommendations for future research the real world Having combined with Machine Learning, Collective Intelligence can be considered as an element in conducting research In particular, Crowdsourcing systems like Amazon Mechanical Turk (AMT) can be used for creating corpora in computational linguistics or for evaluating results in information retrieval By this way, Crowdsourcing is considered as a crowd-mediated for harnessing such collective intelligence Referring to [100], the authors have demonstrated the effectiveness of using AMT for a variety of natural language annotation tasks The findings have revealed that non-expert labelers can provide reliable natural language annotations In particular, on average the label quality by using four non-expert labels per item is equal to an expert annotator does Even in [60], the authors have found that by using Crowdsourcing market-oriented predictions yielded can be equal to (or even better) than those generated by the incumbent approaches In Computer Vision, the authors in [19] have investigated the use of crowds in supervised learning approaches to the problem of object classification to reduce label uncertainty Similarly, crowds can be utilized to highlight portions of an image corresponding to a given annotation [109] Furthermore, recommender systems can be improved their product recommendations by applying existing approached in Machine Learning to mining consumer behaviors and product review sentiments The concept of Collective Intelligence has been shed light by the emergence of social web [35] Beyond aggregating individual potentials, in the case of Social Networks, Collective Intelligence often aims at maximizing the potential of a group [10] For such an assumption, the criteria worked out by Surowiecki are often taken into account to form groups of problem solvers According to [122], in order to be not influenced by others in making individual solutions to a given problem, the social influence of each other must be minimal However, in [14], the authors have worked out general network conditions such that social influence is positively associated with the accuracies of collective predictions, even as individuals have similar beliefs In particular, the information exchange in decentralized communication networks will lead to more accurate collective predictions Taking advantage of the Collective Intelligence 169 Recommendations for future research of crowds on the Internet to find relevant information, Question and Answer (Q&A) websites such as Quora and Yahoo!Answers provide platforms in which users can post questions and receive answers Li and colleagues have studied Collective Intelligence in the Yahoo!Answers of online social network (OSN) by taking into account the structure of OSN, behavior, and knowledge of users, and the knowledge base in a user’s social network [63] The obtained results can be considered as a foundation for the evolution of further research on applying Collective Intelligence to Q&A systems From these recent findings, we intend to conduct the empirical experiments by using Crowdsourcing environment or using social networks as a crowd-meditated Furthermore, it can be seen that anonymous members who are often diverse in their backgrounds or knowledge bases Thus, in some sense, their solutions to a given problem can be unreliable or biased This phenomenon leads to the need of how to properly aggregate the solutions of the diverse members, and how to reduce the bias (to increase the accuracy of collective prediction) such as by providing a small amount of expert knowledge For this aim, belief propagation - a message-passing algorithm [121] for performing inference probabilistic graphical models, which provide a powerful framework for aggregating multiple sources of information and reasoning over large numbers of variables, will be used [121] Moreover, choosing members for solving a given problem also plays an important role in Collective Intelligence applications Beyond criteria proposed by Surowiecki, Simons and colleagues have argued that for collectives to be intelligent they should satisfy knowledge, motivation, diversity, and independence [98] The knowledge is essential because members whose are knowledge of the given problem will reduce unreliable or biased solutions provided by anonymous members For such tasks, Machine Learning methods such as Learning-to-Rank, InfluenceRank can be used for forming a collective with predefined requirements These research problems will be the subject of future works Beyond these research problems, we also intend to take into account interval value as a representation of individual predictions It is due to most research results on the use 170 Recommendations for future research of the wisdom of crowds are based on the so-called point estimates - single values (in the form of one-dimensional vector) provided as estimates of unknown quantities or the outcomes of future events However, in some situations such as predicting how much sales will grow in the next year, it is easy to provide a prediction in the form of an interval value For example, one can report that the sale will grow from 3% to 5% For simplicity, we called this case as interval estimates Recent research on interval estimates has mainly focused on proposing efficient methods for aggregating individual predictions [67, 89] The main idea of these proposed methods is based on the un-weighted (or weighted) midpoints of intervals Others are concentrated on issues related to the so-called overconfidence - a factor that may lead to suboptimal solutions [37] According to [102, 104], the phenomenon of overconfidence can be reduced by asking for the lower and upper bounds of the intervals separately or asking individuals for assigning their confidence levels to the intervals However, whether the wisdom of crowds effect exists in the case of interval estimates? If yes, how some characteristics of crowds such as diversity, independence, decentralization, and cardinality affect crowd performance? These research problems should be the subject of future works 171 REFERENCES [1] B.N Adebambo, B Bliss, The value of crowdsourcing: Evidence from earnings forecasts, Working Paper, 2015 [2] L.R Anderson, C.A Holt, Information Cascades in the Laboratory, The American Economic Review, 87 (1997) 847-862 [3] O Arazy, N Halfon, D Malkinson, Forecasting Rain Events - Meteorological Models or Collective Intelligence?, in Proc of EGU General Assembly Conference 17, 2015, pp 15611-15614 [4] O.M Arazy, W.; and Patterson, R., Wisdom of the crowds: Decentralized knowledge construction in Wikipedia, in Proc of WITS 2006, pp 79-84 [5] L Argote, R Devadas, N Melone, The base-rate fallacy: Contrasting processes and outcomes of group and individual judgment, Organizational Behavior and Human Decision Processes, 46 (1990) 296-310 [6] J.S Armstrong, Combining forecasts, Principles of forecasting 30 (2001), pp 417439 [7] J.S Armstrong, How to Make Better Forecasts and Decisions: Avoid Face-to-face Meetings, Foresight—The International Journal of Applied Forecasting (2006) 3-8 [8] K.J Arrow, Social choice and individual values, Wiley New York, 1963 [9] K.J Arrow, et al., The Promise of Prediction Markets, Science, 320 (2008) 877878 [10] G.K Awal, K.K Bharadwaj, Team formation in social networks based on collective intelligence - an evolutionary approach, Appl Intell., 41 (2014) 627-648 [11] J.A Baars, C.F Mass, Performance of National Weather Service forecasts compared to operational, consensus, and weighted model output statistics, Weather and Forecasting, 20 (2005) 1034-1047 [12] J.P Barthelemy, A Guenoche, O Hudry, Median linear orders: Heuristics and a branch and bound algorithm, European Journal of Operational Research, 42 (1989) 313-325 [13] A Bassamboo, R Cui, A Moreno, The Wisdom of Crowds in Operations: Forecasting Using Prediction Markets, DOI http://dx.doi.org/10.2139/ssrn.2679663, 2015 [14] J Becker, D Brackbill, D Centola, Network dynamics of social influence in the wisdom of crowds, Proceedings of the National Academy of Sciences of the United States of America, 114 (2017) E5070-E5076 172 References [15] J Berg, R Forsythe, F Nelson, T Rietz, Results from a Dozen Years of Election Futures Markets Research, in Handbook of Experimental Economics Results, 2008, pp 742-751 [16] E Birnbaum, E.L Lozinskii, Consistent subsets of inconsistent systems: structure and behaviour, Journal of Experimental & Theoretical Artificial Intelligence 15 (2003) 25-46 [17] D.C Brabham, Crowdsourcing as a model for problem solving: An introduction and cases, Convergence 14 (2008) 75-90 [18] D.C Brabham, Crowdsourcing The MIT press essential knowledge series, Cambridge: Massachusetts Institute of Technology, 2013 [19] S Branson, C Wah, F Schroff, B Babenko, P Welinder, P Perona, S Belongie, Visual Recognition with Humans in the Loop, in Proc of ECCV 2010, pp 438-451 [20] T Buecheler, J.H Sieg, R.M Füchslin, R Pfeifer, Crowdsourcing, Open Innovation and Collective Intelligence in the Scientific Method-A Research Agenda and Operational Framework, in Proc of ALIFE 2010, pp 679-686 [21] K Campbell, A Mínguez-Vera, Gender Diversity in the Boardroom and Firm Financial Performance, Journal of Business Ethics 83 (2008) 435-451 [22] J.A.R Castillo, A Silvescu, D Caragea, J Pathak, V.G Honavar, Information extraction and integration from heterogeneous, distributed, autonomous information sources-a federated ontology-driven query-centric approach, in Proc of IRI 2003, pp 183-191 [23] K.S Cheruvelil, P.A Soranno, K.C Weathers, P.C Hanson, S.J Goring, C.T Filstrup, E.K Read, Creating and maintaining high-performing collaborative research teams: the importance of diversity and interpersonal skills, Frontiers in Ecology and the Environment 12 (2014) 31-38 [24] R.T Clemen, Combining forecasts: A review and annotated bibliography, International Journal of Forecasting (1989) 559-583 [25] L Conradt, Collective behaviour: When it pays to share decisions, Nature 471 (2011) 40-41 [26] B Cowgill, J Wolfers, E Zitzewitz, Using Prediction Markets to Track Information Flows: Evidence from Google, Working Paper, Dartmouth University, 2009 173 References [27] R Cui, S Gallino, A Moreno, D.J Zhang, The Operational Value of Social Media Information, Available at SSRN http://dx.doi.org/10.2139/ssrn.2702151, (2015) [28] C.P Davis-Stober, D.V Budescu, S.B Broomell, J Dana, The Composition of Optimally Wise Crowds, Decision Analysis 12 (2015) 130-143 [29] W.H.E Day, The Consensus Methods as Tools for Data Analysis, in Proc ofIFC'87: Classifications and related methods of data analysis 1988, pp 317-324 [30] J Dean, S Ghemawat, MapReduce: simplified data processing on large clusters, in Proc of the 6th conference on Symposium on Opearting Systems Design & Implementation 2004, pp 10-10 [31] P.C Fishburn, Condorcet Social Choice Functions, SIAM Journal on Applied Mathematics, 33 (1977) 469-489 [32] R Forsythe, T.A Rietz, T.W Ross, Wishes, expectations and actions: a survey on price formation in election stock markets, Journal of Economic Behavior & Organization, 39 (1999) 83-110 [33] F Galton, Vox populi (The wisdom of crowds), Nature, 75 (1907) 450-451 [34] V Gaur, S Kesavan, A Raman, M.L Fisher, Estimating Demand Uncertainty Using Judgmental Forecasts, Manufacturing & Service Operations Management, (2007) 480-491 [35] B Gholami, R Safavi, Ieee, Harnessing Collective Intelligence: Wiki and Social Network from End-user Perspective, in Proc of IC4E 2010, pp 242-246 [36] D Gigone, R Hastie, Proper analysis of the accuracy of group judgments, Psychological Bulletin, 121 (1997) 149-167 [37] M Glaser, T Langer, M Weber, True Overconfidence in Interval Estimates: Evidence Based on a New Measure of Miscalibration, Journal of Behavioral Decision Making, 26 (2013) 405-417 [38] B Golub, M.O Jackson, Naïve Learning in Social Networks and the Wisdom of Crowds, American Economic Journal: Microeconomics, (2010) 112-149 [39] K Gordon, Group Judgments in the Field of Lifted Weights, Journal of Experimental Psychology, (1924) 398-400 [40] A Graefe, J.S Armstrong, Comparing Face-to-Face Meetings, Nominal Groups, Delphi and Prediction Markets on an Estimation Task, International Journal of Forecasting, 27 (2011) 183-195 174 References [41] A Graefe, S Luckner, C Weinhardt, Prediction markets for foresight, Futures, 42 (2010) 394-404 [42] K Green, J Armstrong, A Graefe, Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared, Foresight: The International Journal of Applied Forecasting, (2007) 17-20 [43] A Hecker, Knowledge Beyond the Individual? Making Sense of a Notion of Collective Knowledge in Organization Theory, Organization Studies, 33 (2012) 423445 [44] E Herrera-Viedma, F.J Cabrerizo, J Kacprzyk, W Pedrycz, A review of soft consensus models in a fuzzy environment, Information Fusion, 17 (2014) 4-13 [45] H Hong, Q Du, G Wang, W Fan, D Xu, Crowd Wisdom: The Impact of Opinion Diversity and Participant Independence on Crowd Performance, in Proc of AMCIS 2016 [46] L Hong, S.E Page, Groups of diverse problem solvers can outperform groups of high-ability problem solvers, Proceedings of the National Academy of Sciences of the United States of America, 101 (2004) 16385-16389 [47] L Hong, S.E Page, Groups of diverse problem solvers can outperform groups of high-ability problem solvers, Proceedings of the National Academy of Sciences, 101 (2004) 16385-16389 [48] J Howe, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, Crown Publishing Group, 2008 [49] I.L Janis, Groupthink: psychological studies of policy decisions and fiascoes, Houghton Mifflin, 1982 [50] J Kacprzyk, S Zadrozny, Soft computing and Web intelligence for supporting consensus reaching, Soft Comput., 14 (2010) 833-846 [51] D Kahneman, Thinking, fast and slow, Farrar, Straus and Giroux, New York, 2011 [52] T Kanungo, D.M Mount, N.S Netanyahu, C.D Piatko, R Silverman, A.Y Wu, An Efficient k-Means Clustering Algorithm: Analysis and Implementation, IEEE Trans Pattern Anal Mach Intell., 24 (2002) 881-892 [53] H Kawamura, A Ohuchi, Evolutionary emergence of collective intelligence with artificial pheromone communication, in Proc of 26th Annual Conference of the IEEE Industrial Electronics Society 2000, pp 2831-2836 175 References [54] T.L Kelley, The applicability of the Spearman-Brown formula for the measurement of reliability, Journal of Educational Psychology, 16 (1925) 300-303 [55] J.A Kline, Orientation and group consensus, Central States Speech Journal, 23 (1972) 44-47 [56] S Kloker, T Straub, C Weinhardt, Designing a Crowd Forecasting Tool to Combine Prediction Markets and Real-Time Delphi, in Lecture Notes in Computer Science (10243), 2017, pp 468-473 [57] K Knight, Measuring Inconsistency, Journal of Philosophical Logic, 31 (2002) 77-98 [58] R.H Kurvers, et al S.M Herzog, R Hertwig, J Krause, P.A Carney, A Bogart, G Argenziano, I Zalaudek, M Wolf, Boosting medical diagnostics by pooling independent judgments, Proceedings of the National Academy of Sciences, (2016) 8777-8782 [59] R.H Kurvers, J Krause, G Argenziano, I Zalaudek, M Wolf, Detection accuracy of collective intelligence assessments for skin cancer diagnosis, JAMA dermatology, 151 (2015) 1346-1353 [60] M Lang, N Bharadwaj, C.A Di Benedetto, How crowdsourcing improves prediction of market-oriented outcomes, Journal of Business Research, 69 (2016) 4168-4176 [61] R.P Larrick, J.B Soll, Intuitions About Combining Opinions: Misappreciation of the Averaging Principle, Management Science, 52 (2006) 111-127 [62] P Levy, Collective Intelligence: Mankind's Emerging World in Cyberspace, Perseus Books, 1997 [63] Z Li, H Shen, J.E Grant, Collective intelligence in the online social network of yahoo!answers and its implications, in Proc of ACM CIKM 2012, pp 455-464 [64] H.A Linstone, M Turoff, The Delphi method: Techniques and applications, Addison-Wesley Reading, MA, 1975 [65] J Lorenz, H Rauhut, F Schweitzer, D Helbing, How social influence can undermine the wisdom of crowd effect, Proceedings of the National Academy of Sciences, 108 (2011) 9020-9025 [66] I Lorge, D Fox, J Davitz, M Brenner, A survey of studies contrasting the quality of group performance and individual performance, Psychological Bulletin, 55 (1958) 337-372 176 References [67] A Lyon, B.C Wintle, M Burgman, Collective wisdom: Methods of confidence interval aggregation, Journal of Business Research, 68 (2015) 1759-1767 [68] S Maitrey, C.K Jha, MapReduce: Simplified Data Analysis of Big Data, Procedia Computer Science, 57 (2015) 563-571 [69] M Maleszka, N.T Nguyen, Integration computing and collective intelligence, Expert Syst Appl., 42 (2015) 332-340 [70] T.W Malone, M.S Bernstein, Handbook of Collective Intelligence, The MIT Press, 2015 [71] R.P Mann, D Helbing, Optimal incentives for collective intelligence, Proceedings of the National Academy of Sciences, 114 (2017) 5077-5082 [72] A Newell, Unified theories of cognition, Harvard University Press, 1990 [73] N.T Nguyen, Using consensus methods for determining the representation of expert information in distributed systems, Artificial Intelligence: Methodology, Systems, and Applications 2000, pp 11-20 [74] N.T Nguyen, Consensus-based Timestamps in Distributed Temporal Databases, The Computer Journal, 44 (2001) 398-409 [75] N.T Nguyen, Using consensus for solving conflict situations in fault-tolerant distributed systems, in Proc of Cluster Computing and the Grid 2001, pp 379-384 [76] N.T Nguyen, Methods for consensus choice and their applications in conflict resolving in distributed systems, Wroclaw University of Technology Press, 2002 [77] N.T Nguyen, Advanced Methods for Inconsistent Knowledge Management, Springer-Verlag, London, 2008 [78] N.T Nguyen, Inconsistency of knowledge and collective intelligence, Cybernetics and Systems, 39 (2008) 542-562 [79] N.T Nguyen, C Danilowicz, Deriving Consensus for Conflict Data in WebBased Systems, in Developments in Applied Artificial Intelligence, Springer Berlin Heidelberg, 2003, pp 254-263 [80] N.T Nguyen, V.D Nguyen, D Hwang, An influence analysis of the number of members on the quality of knowledge in a collective, Journal of Intelligent and Fuzzy Systems, 32 (2017) 1217-1228 [81] V.D Nguyen, M.G Merayo, Intelligent collective: some issues with collective cardinality, Journal of Information and Telecommunication, (2017) 1-14 [82] V.D Nguyen, M.G Merayo, N.T Nguyen, Intelligent Collective: The Role of Diversity and Collective Cardinality, in Proc of ICCCI 2017, pp 83-92 177 References [83] V.D Nguyen, N.T Nguyen, A Two-Stage Consensus-Based Approach for Determining Collective Knowledge, in Proc of ICCSAMA 2015, pp 301-310 [84] V.D Nguyen, N.T Nguyen, The Impact of Diversity on the Quality of Collective Prediction, in Proc of INISTA, 2017, pp 149-154 [85] V.D Nguyen, N.T Nguyen, An influence analysis of diversity and collective cardinality on collective performance, Information Sciences, 430 (2018) 487-503 [86] V.D Nguyen, N.T Nguyen, D Hwang, An Improvement of the Two-Stage Consensus-Based Approach for Determining the Knowledge of a Collective, in Proc of ICCCI 2016, pp 108-118 [87] M Nofer, O Hinz, Are crowds on the internet wiser than experts? The case of a stock prediction community, Journal of Business Economics, 84 (2014) 303-338 [88] S.E Page, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies, Princeton University Press, Princeton, NJ, 2007 [89] S Park, D.V Budescu, Aggregating multiple probability intervals to improve calibration, Judgment and Decision Making, 10 (2015) 130-143 [90] C.R Plott, J Wit, W.C Yang, Parimutuel betting markets as information aggregation devices: experimental results, Economic Theory, 22 (2003) 311-351 [91] T Prokesch, H.A von der Gracht, H Wohlenberg, Integrating prediction market and Delphi methodology into a foresight support system — Insights from an online game, Technological Forecasting and Social Change, 97 (2015) 47-64 [92] L Robert, D.M Romero, Crowd Size, Diversity and Performance, in Proc of the 33rd Annual ACM Conference on Human Factors in Computing Systems 2015, pp 1379-1382 [93] G Rowe, A Guide to Delphi, Foresight: The International Journal of Applied Forecasting (2007) 11-16 [94] G Rowe, G Wright, The Delphi technique as a forecasting tool: issues and analysis, International Journal of Forecasting, 15 (1999) 353-375 [95] P Russell, The Global Brain Awakens: Our Next Evolutionary Leap, Element, 2000 [96] S Saint, J.R Lawson, Rules for Reaching Consensus: A Modern Approach to Decision Making, Jossey-Bass, 1994 [97] E Servan‐Schreiber, J Wolfers, D.M Pennock, B Galebach, Prediction Markets: Does Money Matter?, Electronic Markets, 14 (2004) 243-251 178 References [98] J.P Simmons, L.D Nelson, J Galak, S Frederick, Intuitive Biases in Choice versus Estimation: Implications for the Wisdom of Crowds, Journal of Consumer Research, 38 (2011) 1-15 [99] A.M Simons, Many wrongs: the advantage of group navigation, Trends in Ecology & Evolution, 19 (2004) 453-455 [100] R Snow, B O'Connor, D Jurafsky, A.Y Ng, Cheap and fast -but is it good?: evaluating non-expert annotations for natural language tasks, in Proc of EMNLP 2008, pp 254-263 [101] E Snowberg, J Wolfers, E Zitzewitz, Prediction Markets for Economic Forecasting, in Handbook of Economic Forecasting 2013, pp 657-687 [102] J.B Soll, J Klayman, Overconfidence in interval estimates, Journal of Experimental Psychology: Learning, Memory, and Cognition, 30 (2004) 299–314 [103] M Spann, B Skiera, Internet-Based Virtual Stock Markets for Business Forecasting, Management Science, 49 (2003) 1310-1326 [104] A Speirs-Bridge, F Fidler, M McBride, L Flander, G Cumming, M Burgman, Reducing Overconfidence in the Interval Judgments of Experts, Risk Analysis, 30 (2010) 512-523 [105] Sugha P., G R., A Survey Paper on Map Reduce in Big Data, International Journal of Science and Research, (2016) 1103-1107 [106] C.R Sunstein, Infotopia: How Many Minds Produce Knowledge, Oxford University Press, Inc., 2006 [107] J Surowiecki, The wisdom of crowds, Doubleday/Anchor, New York, 2005 [108] G Tziralis, I Tatsiopoulos, Prediction markets: An extended literature review, The journal of prediction markets, (2012) 75-91 [109] S Vijayanarasimhan, K Grauman, Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds, International Journal of Computer Vision, 108 (2014) 97-114 [110] G Vossen, Big data as the new enabler in business and other intelligence, Vietnam J Comput Sci, (2013) 1-12 [111] V.H Vroom, Leadership and the decision-making process, Organizational Dynamics, 28 (2000) 82-94 [112] C Wagner, A Suh, The role of task difficulty in the effectiveness of collective intelligence, in Proc of 7th IEEE International Conference on Digital Ecosystems and Technologies (DEST) 2013, pp 90-95 179 References [113] C Wagner, A Suh, The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance, in Proc of 47th Hawaii International Conference on System Sciences, 2014, pp 594-603 [114] C Wagner, T Vinaimont, Evaluating the wisdom of crowds, Journal of Computer Information Systems, 11 (2010) 724-732 [115] H Wang, M Song, Ckmeans.1d.dp: Optimal k-means clustering in one dimension by dynamic programming, The R Journal, (2011) 29-33 [116] J.H Watkins, Prediction markets as an aggregation mechanism for collective intelligence, in Proc of UCLA Lake Arrowhead Conference on Human Complex Systems 2007, pp 1-9 [117] J Wolfers, E Zitzewitz, Prediction markets, The Journal of Economic Perspectives, 18 (2004) 107-126 [118] A.W Woolley, C.F Chabris, A Pentland, N Hashmi, T.W Malone, Evidence for a collective intelligence factor in the performance of human groups, Science, 330 (2010) 686-688 [119] Z Wu, J Xu, Consensus reaching models of linguistic preference relations based on distance functions, Soft Comput, 16 (2012) 577-589 [120] Y Xu, K.W Li, H Wang, Distance-based consensus models for fuzzy and multiplicative preference relations, Information Sciences, 253 (2013) 56-73 [121] J.S Yedidia, W.T Freeman, Y Weiss, Understanding belief propagation and its generalizations, in: Exploring artificial intelligence in the new millennium, Morgan Kaufmann Publishers Inc., 2003, pp 239-269 [122] H Yin, B Cui, Y Huang, Finding a Wise Group of Experts in Social Networks, in Proc of ADMA 2011, 2011, pp 381-394 [123] H.P Young, An axiomatization of Borda's rule, Journal of Economic Theory, (1974) 43-52 [124] K Zettsu, Y Kiyoki, Towards Knowledge Management Based on Harnessing Collective Intelligence on the Web, in Lecture Notes in Computer Science, 2006, pp 350-357 180 ... the impact of cardinality on collective performance; To analyze the impact of diversity on collective performance; To analyze the impact of both diversity and cardinality on collective performance; ... we only focus on the diversity of individual predictions On analyzing the impacts of cardinality and diversity on collective performance, we address two research questions: "Does a larger collective. .. 2.2 Relationship between collective prediction and individual predictions 33 2.3 The impact of cardinality on collective performance 34 2.4 The impact of diversity on collective performance