Personalized information retrieval based on novelty feeback

A NOVELTY-BASED APPROACH TO PERSONALIZED INFORMATION RETRIEVAL YIN HAINAN (B.Eng., Shanghai Jiao Tong University) A THESIS SUBMITTED FOR THE DEGREE OF MASTER OF SCIENCE DEPARTMENT OF INFORMATION SYSTEMS NATIONAL UNIVERSITY OF SINGAPORE 2005 ACKNOWLEDGEMENTS I would like to express my deepest gratitude and thanks to all those who have helped me with this work First of all, I am greatly indebted to my supervisor Dr Xu Yunjie, who provided me with professional guidance and personal example during my master study at NUS His invaluable motivation, advice and comments have the largest immediate influence on this thesis Secondly, I would like to thank A/P Danny POO and Dr Kan Min-Yen Their valuable suggestions definitely helped me a lot in the improvement of my research In addition, many people have facilitated my research by providing suggestions in system development, data analysis, and thesis writing Among all the people there who have given me helpful assistance, I would like to present my special thanks to Mr Liu Chengliang, Mr Ji Yong, Mr Zhang Xinhua, Mr Cui Hang, Mr Wang Gang, Mr Chen Zhiwei and Mr Chen Xi I gratefully acknowledge the financial support of National University of Singapore in the form of my research scholarship Besides, I would also like to express my gratitude for the excellent environment and facilities provided by NUS My heartfelt thanks go to my friends for their constant love and support Miss Shen Wei, Mr Chin Yee Yung, Miss Qian Bo and Miss Shi Yijing have each given me years of friendship and have done more for me than I could ever hope to repay Also, I would like i to thank my lab-mates in the Knowledge Management Lab, such as Miss Teoh Say Yen, Mr Kong Wei-Chang, Ms Chen Junwen, Mr Qian Zhijiang, Mr Cai Shun and Miss Yang Li Every day being with them was really enjoyable Lastly, I would like to express my sincerest thanks to my parents Their love and understanding are my impetus to perform research during my postgraduate studies ii TABLE OF CONTENTS ACKNOWLEDGEMENTS i TABLE OF CONTENTS iii SUMMARY v LIST OF TABLES .vii LIST OF FIGURES viii Chapter Introduction Chapter Related Works .5 2.1 Subjective Relevance, Topicality and Novelty 2.2 Relevance in Personalized Information Retrieval Studies 2.3 Novelty in System-Centered Information Retrieval Studies 10 2.4 Integration Rule in Relevance Judgment 13 Chapter Novelty-Augmented Systems 17 3.1 Topicality Profile and Judgment .18 3.1.1 Topicality Profile 18 3.1.2 Topicality Profile Updating Strategy 18 3.1.3 Topicality Judgment .19 3.2 Novelty Profile and Relevance Judgment 19 3.2.1 Novelty Profile Type I 19 3.2.2 Novelty Profile Type II 21 Chapter Experiment Design 27 4.1 Testing Task 27 4.2 Testing Corpus 28 iii 4.3 Experimental Procedure .29 Chapter Result and Analysis .32 5.1 Performance Test 33 5.2 Test of Learning Assumption and Judgment Criteria Integration .38 5.3 Simulations and Sensitivity Analysis 40 5.3.1 Simulation of Relevance Feedback 42 5.3.2 Novelty Profile Updating Speed 43 5.3.3 Novelty Weight .43 Chapter Discussion and Conclusion 46 Bibliography 50 Appendix A 59 Appendix B 62 iv SUMMARY Information overload becomes an immediate issue with the rapid progress of information technology, especially the WWW In order to help users better find their desired information, it is important to tailor information retrieval systems to meet individual preference However, the performances of most personalized information retrieval systems are still far from satisfactory One potential problem as pointed by user-centered studies is that the relevance measures in information retrieval systems are biased towards topicality and fail to capture the multidimensionality of users’ relevance judgment Furthermore, it has been also found by user-centered studies that novelty perception is the next most important factor of user’s relevance judgment besides topicality Building on past user studies, this thesis proposes a novelty-based approach to personalized information retrieval which incorporates both topicality and novelty as relevance criteria More specifically, we propose a set of hypotheses regarding topicality and novelty in relevance judgment and test the validity of such hypotheses with real users using systems designed based on the hypotheses Particularly, we hypothesize that (1) novelty perception is a value-added criterion to improve personalized information retrieval, (2) relevance measures in past system-centered personalized information retrieval studies are biased towards topicality, (3) user’s novelty judgment standard is directed toward a subtopic and is slowly changing because user’s learning of document content in retrieval process is incomplete, and (4) relevance judgment of a document starts with topicality judgment followed by novelty judgment in a stepwise fashion A set of personalized information retrieval systems has been designed to implement these v propositions Our user test supports these hypotheses except for last one which might be insignificant because of the specific nature of the testing corpus vi LIST OF TABLES Table 1: Example – simple retrieval example based on vector space model Table 2: Example – vector space model with relevance feedback 10 Table 3: Summary of system-centered studies on novelty 12 Table 4: Potential PIR models that incorporate both topicality and novelty .16 Table 5: Novelty and topicality precision 38 Table 6: Missing evaluation analysis in simulations 62 vii LIST OF FIGURES Figure 1: System interface 30 Figure 2: Raw relevance precision .34 Figure 3: Adjusted relevance precision 36 Figure 4: Adjusted relevance precision by round 37 Figure 5: Interaction effect of learning assumption and judgment criteria integration rule 40 Figure 6: Simulation for traditional relevance feedback 42 Figure 7: Sensitivity of IL-Add and IL-Step to novelty updating speed .43 Figure 8: Sensitivity of MMR-Add5 to redundancy parameter 44 Figure 9: Sensitivity of IL-Add to novelty weight .45 viii Chapter Introduction With the rapid progress of information technology especially the prosperity of WWW, the amount of information in the form of documents and web pages increases dramatically, which arouses an acute need for information retrieval (IR) systems to help users exploit such an extremely valuable resource However, one severe problem of most IR systems such as search engines is that they are not tailored to meet individual preference Pretschner and Gauch (1999) noted that almost half of the documents returned by search engines are deemed irrelevant by their users An IR system typically treats a user only by the text query submitted by the user, and generates the same search results regardless of who submitted the query In order to discriminate the different information needs of the users, the learning ability and personalization of the IR systems is critical to achieve a satisfactory retrieval performance Therefore, personalized information retrieval (PIR) has been a very active research field in the past years Typical PIR techniques are based on relevance feedback and its variants (Ide, 1971; Ide and Salton, 1971; Rocchio, 1971; Salton & Buckley, 1990), which can be considered to be learning user’s interest model in a single search session Such techniques try to capture the context of a user’s query from extra feedback Furthermore, the application of relevance feedback technique to long-term personalization can be seen as a kind of user profiling In the IR domain, Another proposition of this study - that the relevance measure in system-centered PIR studies is biased toward topicality - is supported by our simulation of traditional relevance feedback The existence of such bias further supports the theoretical validity of introducing novelty besides system level impact The sensitivity analysis of the systems also helps to confirm us that the above conclusions are not artifacts of parameter choice Finally, although not intentionally designed, the data analysis shows that individual difference as reflected in round document evaluation is a good predictor a user’s relevance judgment in the later rounds In comparison, the different designs of system contribute relatively less On one hand, this implies much should be done to improve the algorithms; on the other hand, it implies that capturing subjectivity and individual difference in IR is really a profitable area of research although its potential is far from being realized However, this study has a number of limitations that may threat on the validity of the conclusions drawn above First, the sample size in the user study is still small, which causes substantial difference in the initial condition across systems If the impact of initial condition on later document evaluation is nonlinear, then the above conclusion might be in question Second, it is desirable to test the traditional relevance feedback model and parameter sensitivity with real user groups rather than simulation Third, 48 the use of small pre-filtered corpus might introduce bias in the conclusion Fourth, fine tuning of parameters are not explored in this study Finally, it is desirable to compare the system performance in a realistic setting rather than in an experimental setting because user’s learning behavior could be very different in a real setting With all these limitation in mind, this study represents a first step toward incorporating findings from user-centered IR studies into PIR systems We have explored the possibility to quantify relevance perception as a multidimensional construct Particularly, we have explored the two most important dimensions of relevance: topicality and novelty We have confirmed some propositions drawn from user-centered IR studies and offer them an empirical grounding in system studies We have also provided a set of practical implementation methods Our preliminary findings open the way for better quantification methods to be proposed to describe user topicality and novelty profile in future studies 49 Bibliography Allan, J., Wade, C., & Bolivar, A (2003) Retrieval and novelty detection at the sentence level In the Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, Canada, July 28-August 1, 20 2003 314-321 Baeza-Yates, R.A., & Ribeiro-Neto, B.A., (1999) Modern Information Retrieval ACM Press /Addison-Wesley Barry, C., & Schamber, L (1998) Users’ criteria for relevance evaluation: A cross-situational comparison Information Processing & Management, 34, 219–236 Barry, C.L (1994) User-defined relevance criteria: An exploratory study Journal of the American Society for Information Science, 45(3), 149-159 Bateman, J (1998) Changes in relevance criteria: A longitudinal study Proceedings of the 61st Annual Meeting of the American Society for Information Science, 35, 23–32 Berkley, University of California (2001) Web Term Document Frequency Form http://elib.cs.berkeley.edu/docfreq Accessed on June 19, 2005 50 Bettman, James R and Park, C Whan (1980) Effects of Prior Knowledge and Experience and Phase of the Choice Process on Consumer Decision Processes: A Protocol Analysis Journal of Consumer Research, (December), 234-248 Bookstein, A (1979) Relevance Journal of the American Society for Information Science, 30(5), 269-273 Borlund, P (2003) The concept of relevance in IR Journal of the American Society for information Science and Technology, 54(10), 913-925 Boyce, B (1982) Beyond topicality: A two stage view of relevance and the retrieval process Information Processing & Management, 18(3), 105-109 Brants, T., Chen, F., & Farahat, A (2003) A system for new event detection In the Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, Canada, July 28 - August 1, 2003 330-337 Carbonell, J., & Goldstein, J (1998) The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries Proceedings of SIGIR 1998 Cosijn, E., & Ingwersen, P (2000) Dimensions of relevance Information Processing 51 & Management, 36(4), 533-550 Croft, W.B., Cronen-Townsend, S., & Lavrenko, V (2001) Relevance Feedback and Personalization: A language Modeling Perspective In the Proceedings of the DELOS-NSF Workshop on Personalization and Recommender Systems in Digital Libraries, Dublin, Ireland, June 18-20, 2001 Eirinaki, M & Vazirgiannis, M (2003) Web Mining for Web Personalization ACM Transactions on Internet Technology, Vol 3, No 1, February 2003, Pages 1-27 Fitzgerald, M.A., & Galloway, C (2001) Relevance judging, evaluation, and decision making in virtual library: A descriptive study Journal of the American Society for Information Science and Technology, 52(12), 989-1010 Froehlich, T.J (1994) Relevance reconsidered: Towards an agenda for the 21st century: Introduction to special topic issue on relevance research Journal of the American Society for Information Science, 45, 124 –134 Google, Corp (2005) http://www.google.com Accessed on June 19, 2005 Greisdorf, H (2003) Relevance thresholds: A multi-stage predictive model of how users evaluate information Information Processing & Management, 39, 403-423 52 Harter, S.P (1992) Psychological relevance and information science Journal of the American Society for information Science, 43(9), 602-615 Hirschman, E.C (1980) Innovativeness, novelty seeking, and consumer creativity Journal of Consumer Research, 7, 283-295 Ide, E (1971) New experiments in relevance feedback The SMART retrieval system - experiments in automatic document processing, (G Salton ed.), Chapter 16, pp 337-354 Ide, E & Salton, G (1971) Interactive search strategies and dynamic file organization in information retrieval The SMART retrieval system - experiments in automatic document processing, (G Salton ed.), Chapter 18, pp 373-393 Kekäläinen, J., & Järvelin, K (2002) Using graded relevance assessments in IR evaluation Journal of the American Society for Information Science and Technology, 53(13):1120-1129 Kumaran, G., & Allan, J (2004) Text classification and named entities for new event detection SIGIR’04, July 25-29, 2004, Sheffield, South Yorkshire, UK 297-304 53 Liu, F., Yu, C., & Meng, W (2002) Personalized Web Search by Mapping User Queries to Categories Proceedings of CIKM2002 Maglaughlin, K.L., & Sonnewald, H (2002) User Perspective on relevance criteria: A comparison among relevance, partially relevance, and not-relevance Journal of the American Society for Information Science and Technology, 53(5), 327-342 Middleton, S.E., Shadbolt, N R., & Roure, D C De (2004) Ontological User Profiling in Recommender Systems ACM Transaction on Information Systems, Vol.22, No.1, Pages 54-88 Mizzaro, S (1997) Relevance: The whole history Journal of the American Society for Information Science, 48(9), 810–832 Mostafa, J., Mukhopadhyay, S., & Palakal, M (2003) Simulation Studies of Different Dimensions of Users’ Interests and their Impact on User Modeling and Information Filtering Information Retrieval, 6, 199-223 Netscape, Corp (1998) The Open Directory Project (ODP), http://dmoz.org Accessed on June 19, 2005 Park, H (1997) Relevance of science information: origins and dimensions of 54 relevance and their implications to information retrieval Information Processing & Management, 33(3), 339-352 Park, T.K (1993) The nature of relevance in information retrieval: An empirical study Library Quarterly, 63, 318–351 Payne, J.W (1976) Task complexity and contingent processing in decision making: An information search and protocol analysis Organizational Behavior and Human Performance, 16:366-387 Pretschner, A & Gauch, S (1999) Ontology based personalized search In the Proceedings of the Eleventh IEEE International conference on Tools with Artificial Intelligence Robertson, S E., & Sparck-Jones, K (1976) Relevance weighting of search terms Journal of the American Society for Information Science, 27: 129-146 Rocchio, J J (1971) Relevance feedback in information retrieval The SMART retrieval system -experiments in automatic document processing, (G Salton ed.), Chapter 14, pp 313-323 Salton, G., & Buckley, C (1990) Improving retrieval performance by relevance 55 feedback Journal of the American Society for Information Science, 41(4), 288-297 Salton, G & McGill, M.J (1983) Introduction to Modern Information Retrieval McGraw Hill Saracevic, T (1970) The concept of “relevance” in information science: A historical review In Saracevic, T., Introduction to Information Science, 111-151, New York: R.R Bowker Schamber, L (1991) Users' criteria for evaluation in a multimedia environment In J.M Griffiths (ed.), Proceedings of the 54th Annual Meeting of the American Society for Information Science, 28, 126-133 Schamber, L (1994) Relevance and information behavior In M E Williams (ed.), Annual Review of Information Science and Technology, 29, 33-48 Schamber, L., Eisenberg, M.B., & Nilan, M S (1990) A re-examination of relevance: Toward a dynamic, situational definition Information Processing & Management, 26(6), 755-776 Slovic, P (1972) Psychological study of human judgment: Implications for investment decision making The Journal of Finance, 27(4), 779-799 56 Speretta, M & Gauch, S (2004) Personalizing Search Based on User Search Histories Proceedings of CIKM 2004 Spink, A., Greisdorf, H., & Bateman, J (1998) From highly relevant to not relevant: Examining different regions of relevance Information Processing & Management, 34, 599–621 Statistic Singapore (2005) http://www.singstat.gov.sg/keystats/annual/indicators.html Accessed on June 19, 2005 Vakkari, Pertti (2003) Task-based information search Annual Review of Information Science and Technology, 37, (B Cronin, Ed.) Information Today: Medford, NJ, 2002, 413-464 Wang, P & Soergel, D (1998) A cognitive model of document use during a research project Study I Document selection Journal of the American Society for Information Science, 49(2), 115–133 Widyantoro, D H., Ioerger, T R & Yen, J (1999) An Adaptive Algorithm for Learning Changes in User Interests Proceedings of CIKM1999 Widyantoro, D H., Ioerger, T R & Yen, J (2001) Learning User Interest Dynamics 57 with a Three-Descriptor Representation Journal of the American Society of Information Science and Technology, Vol 52, No 3, pp 212-225 Xu, Y., & Chen, Z (2005) Relevance Judgment – What Do Information Users Consider beyond Topicality? Journal of the American Society for Information Science and Technology Forthcoming Yang, Y., Zhang, J., Carbonell, J., & Jin, C (2002) Topic-conditioned novelty detection In the Proceedings of ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), Edmonton, Alberta, Canada 2002 688-693 Zhai, C., Cohen, W.W., & Lafferty, J (2003) Beyond independent relevance: methods and evaluation metrics for subtopic retrieval In the Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, Canada, July 28 - August 1, 2003 10-17 Zhang, Y., Callan, J., & Minka, T (2002) Novelty and redundancy detection in adaptive filtering In the Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, August 11-15, 2002 81-88 58 Appendix A Experiment Instruction and Survey Form Purpose: The purpose of this study is to study Online Searching Behavior Part 1: Search Topic You will be asked to search documents on the “relationship between mobile phone radiation and health.” Documents addressing the following questions are considered relevant: • • • Does use of mobile phone pose radiation threats to the users’ health? Why are there such or no such radiation threats to health? What is the proper way to use mobile phone to protect your health from radiation? Strongly Disagree 1) I know this topic area very well 2) I am able to tell others much about this topic 3) I am very confident in my knowledge about this topic Strongly Agree 7 4A) How long have you been using the Search Engine? Years 4B) How long have you been using the Mobile Phone? Years Part 2: Online Search Mobile Phone Radiation and Health Task Description Assume you are taking a health education class and the final examination which accounts for 50% of total grade is to search and study online documents on “mobile phone radiation and health” Again, the relevance of the document depends on how much it addresses the above-mentioned issues: • • • Does use of mobile phone pose radiation threats to the users’ health? Why are there such or no such radiation threats to health? What is the proper way to use mobile phone to protect your health from radiation? 59 You need to search for documents with the provided search engine After a list of documents (60 documents in pages) is returned by the search engine, please read the documents ONE BY ONE in order, and evaluate them in terms of whether it is on-topic, it is novel (provides new knowledge) to you, and it is overall useful Moreover, you will be asked to take the online exam about the topic “mobile phone radiation and health” after the search A document is on-topic if it talks about something related to your information need But an on-topic document can have as little or as much content related to your information need You may assign it a score ranging from (totally off-topic), (marginally on-topic) to (highly on-topic) A document is novel if it provides NEW knowledge to you You may assign it a score ranging from (nothing new), (very little new knowledge) to (very much new knowledge) based on how much it is novel A document is overall useful if it makes major contribution to your information need and you expect it to substantially contribute to your quiz grade and you try to memorize its content You may assign it a score ranging from (No Contribution), (Very Low Probability / Very Low Contribution) to (Very High Probability / Very High Contribution) Start Online Search Only after being allowed by the experiment administrator, can you log into our search engine (with Internet Explorer) by http://172.18.179.46:8080/PIRWeb/loginLRN.jsp You should use the default search terms “mobile phone health” to start searching Please answer the questions in Part only after you have searched and evaluated all documents online Part 3: Survey Questions Now that you have read and evaluated all articles during the searching process Regarding your knowledge on mobile phone radiation and health, after reading the documents, you would say: 5) I know this topic area very well now 6) I am able to tell others much about this topic now 7) I am very confident in my knowledge about this topic now Strongly Strongly Disagree Agree 7 8) Is the number of documents enough for my task? a) More Than Enough b) Enough c) Not Enough 9) How necessary it is to read further for my task? a) Necessary b) Not Necessary 10) How much you think you have learned from the articles you have just read? a) Very Much b) Much c) Medium d) Very Little 60 11) What is your MAJOR criterion when you judge whether a document is “Novel”? (Choose one) a) A document is novel if its content is NOT similar to the ones I have already read during this experiment b) A document is novel if it provides me with knowledge I didn’t have before I come for this experiment c) Other reason _ 12) Exam Score: Part 4: Personal Information 1) Your gender: a) Male b) Female 2) Your age: 3) Education Level: a) Undergraduate b) Postgraduate 4) Faculty or Department: 5) Email address: 6) Nationality: 61 Appendix B Missing Evaluation Analysis in Simulations Table Missing evaluation analysis in simulations System of Data Source System to Simulate Parameter Setting Total Evaluation per Round Missing Evaluation in Round Missing Evaluation in Round Missing Evaluation in Round TF RF Relevance Feedback 260 2 MMR-Add5 MMR-Add α=0.6 260 16 55 67 MMR-Add5 MMR-Add α=0.7 260 19 66 95 MMR-Add5 MMR-Add α=0.8 260 19 77 116 MMR-Add5 MMR-Add α=0.9 260 25 99 122 MMR-Add5 MMR-Add α=1.0 260 34 104 116 IL-Add IL-Add γ= 0.0 270 59 65 107 IL-Add IL-Add γ= 0.3 270 28 28 46 IL-Add IL-Add γ= 0.7 270 22 36 52 IL-Add IL-Add β= 0.6 270 12 IL-Add IL-Add β=0.1 270 IL-Step IL-Step β=0.6 280 IL-Step IL-Step β=0.1 280 62 [...]... relationship and novelty based on complete learning (e.g., Zhang et al., 2002), with the exception of compensatory relationship and novelty based complete learning (Zhai et al., 2003) However, no exploration has been done based on incomplete learning hypothesis Neither is the comparison of system performance based on different hypotheses Table 4 Potential PIR models that incorporate both topicality and novelty. .. document can contribute to both the novel and the non-novel set proportionally For example, if the maximum novelty score is 7, then a document with novelty score 5 contributes 5/7 document to the novel document set and 2/7 document to the non-novel document set After such conversion, now rj is the sum of novelty “fractions” of only those documents containing term tj, R is the sum of novelty scores... topicality, novelty, understandability, reliability, and scope Topicality and novelty are regarded as two key dimensions of relevance Given the identified importance of novelty in relevance judgment, our first proposition is that novelty is value-added criterion to improve personalized information retrieval performance when it is incorporated into system design If relevance is multidimensional, then... topicality and novelty are considered, non-compensatory integration rule is a better approximation of users’ relevance judgment 15 If we cross-tabulate the assumption of the compensation relationship between topicality and novelty and the assumption of learning, we have four quadrants of possible PIR models (Table 4) The studies done in the system-centered IR have focused the combination of non-compensatory... et al (2002) recognize the limitation of 10 traditional relevance measure “A common complaint about information filtering systems is that they do not distinguish between documents that contain new relevant information and document documents that contain information that is relevant but already known” (p81) They notice that it is unrealistic to expect a single component (i.e., user profile and judgment... and novelty Novelty is regarded as a value added measure to topicality Novelty is also regarded as order-dependent When documents are evaluated in different orders, their novelty measure should change; therefore novelty is dependent on what has been seen before They classify documents into i) not relevant, ii) relevant but contains no new information, and iii) relevant and contains new information In... definition of novelty is different from the user-centered perspective Novelty is not personal and subjective It is based on a historical document set If two people start with the same document set, they have to regard next document as of same novelty Second, novelty is reduced to redundancy If a document or sentence is similar to ones seen before, it is non-novel However, such simplification prerequisites... as well as testing the validity of such propositions with real users using PIR systems designed based on the propositions In particular, we propose that (1) novelty perception is a value-added criterion to improve personalize information retrieval, (2) relevance measures in past system-centered PIR studies are biased towards topicality, (3) user’s 3 novelty judgment standard is directed toward a subtopic... algorithm modification Based on the findings of user-centered IR studies on relevance judgment, the purpose of this thesis is to propose a novelty -based approach to PIR which incorporates both topicality and novelty as relevance criteria More specifically, we would like to propose a set of propositions regarding users’ novelty perception and the way topicality and novelty perceptions are integrated... severe threat to her son when she buys a mobile phone for him She goes to an online search engine and submits a query Q which consists of three terms: ‘mobile phone’ (a bi-gram), ‘radiation’, and ‘child’ Document D1 and D2 are returned (Table 1) In Q, ‘mobile phone’ is a topicality term; 8 ‘child’ is a novelty term; ‘radiation’ is on- topic but non-novel Each term has its corresponding term frequency ... criterion to improve personalized information retrieval, (2) relevance measures in past system-centered personalized information retrieval studies are biased towards topicality, (3) user’s novelty. .. Relevance, Topicality and Novelty 2.2 Relevance in Personalized Information Retrieval Studies 2.3 Novelty in System-Centered Information Retrieval Studies 10 2.4 Integration Rule in Relevance... besides topicality Building on past user studies, this thesis proposes a novelty -based approach to personalized information retrieval which incorporates both topicality and novelty as relevance criteria