Query processing in peer to peer based data management system

Query Processing in Peer-to-Peer Based Data Management System Sai Wu A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE 2011 Query Processing in Peer-to-Peer Based Data Management System Sai Wu School of Computing National University of Singapore Summary In last ten years, we have witnessed the success of P2P (Peer-to-Peer) network. It facilitates the information sharing to an unprecedented scale. Some popular applications, such as Skype and Emule, are deployed to serve millions of users. Although well recognised for its scalability, current P2P network lacks of state-of-art data management system, especially for enterprise applications. To address this problem, database community attempts to integrate database technologies into P2P networks and various PDMSs (Peer-based Data Management Systems) are proposed. In this thesis, we design an efficient processing framework for the PDMS. The framework consists of a query optimizer and three processing approaches tailored for different types of queries. • For simple OLTP queries, the optimizer applies the distributed index to process it. To reduce the maintenance cost of indexes, we propose a just-in-time indexing approach. Instead of indexing the whole dataset, we selectively publish the data based on the query patterns. • For multi-way join queries, the optimizer adopts an adaptive join strategy. It first generates an initial query plan based on the distributed histograms. Since the histograms only provide a coarse estimation, the optimizer will periodically adjust the plan by exploiting the real-time query results. i • When a small amount of inaccuracy can be tolerated, the optimizer switches to an approximate OLAP query processing algorithm. The algorithm continuously retrieves random samples from PDMS. And approximate results are generated and refined based on the samples. The query optimizer select the corresponding processing scheme and exploits the distributed histograms to optimize the query plan. The proposed approaches in this thesis are evaluated on a real distributed platform, PlanetLab. We used TPC-H queries and dataset in our benchmark. Keywords: P2P, PDMS, BATON, Just-in-time, Indexing, Adaptive Query Processing, Approximate Query Processing, Sampling, Online Aggregation. ii Acknowledgments This thesis would not have been accomplished without the support, advice and encouragement of so many people. It is my pleasure to thank those people who made this thesis possible. I would like to express my deep and sincere gratitude to my supervisor, Prof Beng Chin Ooi. When I started my Ph.D journey, I was a layman to the research. It is him who guides me to the correct path. He teaches me the right working style and basic research skills. He shares with me his insightful thoughts, some of which motivate the work in this thesis. Moreover, his attitudes towards research and philosophy of life are great gifts for me. I would like to thank Prof. Kian-Lee Tan and Prof. Anthony K. H. Tung, who give me valuable advice for research and instruct my writing skills. And I owe my most sincere gratitude to Dr. Kun-Lung Wu, who offers me an internship at IBM TJ-Watson Research Lab. During this work I have collaborated with many colleagues, who have helped me build up systems and discuss technical problems, I would like to thank all of them. And I really enjoy the basketball games with “Clouder” team. Particular thanks to all the team members. Lastly, and most importantly, I would like to thank my wife, Ji Zhou, for her encouragement and understanding. During my thesis writing, she gave birth to our lovely baby, Chenze Wu, and undertook most housework to support my research. To her, I dedicate this thesis. iii Contents Introduction 1.1 PDMS (Peer-based Data Management System) . . . . . . . . . . . 1.2 Query Processing in PDMS . . . . . . . . . . . . . . . . . . . . . . 1.2.1 OLTP Queries . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Multi-way Join . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Aggregate Query . . . . . . . . . . . . . . . . . . . . . . . Outline of The Thesis . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Literature Review 2.1 11 P2P Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Unstructured Overlays . . . . . . . . . . . . . . . . . . . . 12 2.1.2 Structured Overlays . . . . . . . . . . . . . . . . . . . . . 14 2.1.3 Comparison of Overlay Networks . . . . . . . . . . . . . . 17 PDMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.1 Schema Mapping . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.2 Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.3 Query Processing . . . . . . . . . . . . . . . . . . . . . . . 20 2.3 Adaptive Query Processing . . . . . . . . . . . . . . . . . . . . . . 22 2.4 Approximate Query Processing . . . . . . . . . . . . . . . . . . . . 23 2.2 A PDMS For Enterprise Applications 27 3.1 Example: A PDMS-based Supply-Chain Management System . . . 28 3.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3 BATON, A BAlanced Tree Overlay Network . . . . . . . . . . . . 34 iv CONTENTS 3.4 Query Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.5 Implementation of MetaStore . . . . . . . . . . . . . . . . . . . . . 38 3.5.1 Histograms for PDMS . . . . . . . . . . . . . . . . . . . . 38 3.5.2 Monitoring Network Status . . . . . . . . . . . . . . . . . . 45 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.6 Just-In-Time Query Retrieval Over Partially Indexed Data in PDMS 52 4.1 Basic Tuple-Level Indexing Strategy . . . . . . . . . . . . . . . . . 53 4.2 PISCES Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.1 Cost of a PDMS . . . . . . . . . . . . . . . . . . . . . . . 55 4.2.2 Approximate Range Index . . . . . . . . . . . . . . . . . . 58 4.2.3 Tuning the Partial Index . . . . . . . . . . . . . . . . . . . 70 4.2.4 Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . 73 4.2.5 Implementation in Other Overlays . . . . . . . . . . . . . . 74 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . 76 4.3.1 Effect of Query Distribution . . . . . . . . . . . . . . . . . 78 4.3.2 Average Setup Time . . . . . . . . . . . . . . . . . . . . . 80 4.3.3 Effect of Different Network Configurations . . . . . . . . . 81 4.3.4 Convergence of Iterative Sampling . . . . . . . . . . . . . . 83 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.3 4.4 Adaptive Query Processing for Multi-Join 85 5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.2 Adaptive Multi-Join Processing in P2P Network . . . . . . . . . . . 88 5.2.1 Generating an Initial Query Plan . . . . . . . . . . . . . . . 88 5.2.2 Adaptive Query Processing . . . . . . . . . . . . . . . . . . 95 v CONTENTS 5.2.3 5.3 5.4 Distributed Mechanism . . . . . . . . . . . . . . . . . . . . 103 Experimental Study . . . . . . . . . . . . . . . . . . . . . . . . . . 104 5.3.1 Effect of varying α . . . . . . . . . . . . . . . . . . . . . . 106 5.3.2 Effect of varying β . . . . . . . . . . . . . . . . . . . . . . 107 5.3.3 Effect of varying γ . . . . . . . . . . . . . . . . . . . . . . 108 5.3.4 Effect of varying δ . . . . . . . . . . . . . . . . . . . . . . 110 5.3.5 Effect of varying ǫ . . . . . . . . . . . . . . . . . . . . . . 111 5.3.6 Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . 111 5.3.7 Performance of Heavy Join Operations . . . . . . . . . . . 113 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Approximate Query Processing in PDMS 6.1 Distributed Online Aggregation . . . . . . . . . . . . . . . . . . . . 117 6.1.1 6.2 6.3 6.4 115 System Overview . . . . . . . . . . . . . . . . . . . . . . . 117 Adaptive Random Sampling . . . . . . . . . . . . . . . . . . . . . 119 6.2.1 Local Sampling . . . . . . . . . . . . . . . . . . . . . . . . 119 6.2.2 Distributed Sampling . . . . . . . . . . . . . . . . . . . . . 120 Online Aggregate Query Processing . . . . . . . . . . . . . . . . . 123 6.3.1 Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.3.2 Samples Dissemination . . . . . . . . . . . . . . . . . . . . 127 6.3.3 Optimizing the Bucket Size . . . . . . . . . . . . . . . . . 129 6.3.4 Query Processing . . . . . . . . . . . . . . . . . . . . . . . 133 Maintaining Samples as a Precomputed Synopsis . . . . . . . . . . 139 6.4.1 Sample Replacement . . . . . . . . . . . . . . . . . . . . . 140 6.4.2 Synopsis Update . . . . . . . . . . . . . . . . . . . . . . . 141 vi CONTENTS 6.5 6.6 Experiment Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 143 6.5.1 Optimal Bucket Size . . . . . . . . . . . . . . . . . . . . . 145 6.5.2 Effect of Data Size . . . . . . . . . . . . . . . . . . . . . . 146 6.5.3 Effect of Error Bound . . . . . . . . . . . . . . . . . . . . . 148 6.5.4 Effect of Confidence c . . . . . . . . . . . . . . . . . . . . 149 6.5.5 Precision of Estimation . . . . . . . . . . . . . . . . . . . . 150 6.5.6 Effect of Insertion . . . . . . . . . . . . . . . . . . . . . . 150 6.5.7 Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . 151 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 Conclusion 153 7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 vii List of Figures 2.1 Unstructured Network . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Structured Network (Chord) . . . . . . . . . . . . . . . . . . . . . 15 2.3 Adaptive Query Processing . . . . . . . . . . . . . . . . . . . . . . 22 2.4 Demonstration of Online Aggregation . . . . . . . . . . . . . . . . 24 3.1 PDMS for Supply-Chain Management . . . . . . . . . . . . . . . . 29 3.2 Architecture of PDMS . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3 Work Flow of Query Engine . . . . . . . . . . . . . . . . . . . . . 32 3.4 BATON Overlay . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.5 Histogram indexing . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.6 Sampling Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.1 Comparison of Different False Positive Factor . . . . . . . . . . . . 60 4.2 Shrunk Index Range . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.3 Maintenance Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.4 A Variant of BATON Tree . . . . . . . . . . . . . . . . . . . . . . 69 4.5 Range Indexing in CAN . . . . . . . . . . . . . . . . . . . . . . . 74 4.6 Schema of TPC-H . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.7 Effect of Query Distribution . . . . . . . . . . . . . . . . . . . . . 79 4.8 Statistics of AOL . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.9 Cells’ Benefit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.10 Average Setup Time of Node . . . . . . . . . . . . . . . . . . . . . 80 4.11 Effect of Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.12 Effect of Churn . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 viii 7.1. CONCLUSION distributed systems are more economic, where the participants are commodity machines, connected via high-speed network links. As the first widely deployed distributed systems, P2P system shows its attractive features, such as better scalability, higher reliability and lower maintenance overhead. Most P2P systems are designed as a community system, but to support enterprise applications, the P2P systems should be enhanced to integrate with relational data model. PDMS (Peer-based Data Management System) is a hybrid system by combining the P2P technique with the DBMS (DataBase Management System) design. In PDMS, as data are distributed over the participating nodes, query processing is very challenging. In this thesis, we address the query processing issues in PDMSs. We start our discussion by formalizing the semantics of query processing in the PDMS. And then, we discuss how to maintain distributed histograms to facilitate query processing. Finally, three new processing schemes are proposed for different types of queries in the PDMS. • To support high-selectivity OLTP queries, a Just-in-time indexing scheme is adopted. The approach is motivated by the observation that queries always follow a skewed distribution. For example, 80% of queries are focusing on 20% of data. Therefore, instead of indexing every tuple in the database, which is costly, we selectively build index for the popular data. In this way, we improve the query performance, as most queries can be directly answered by the index. And we greatly reduce the maintenance cost, as fewer tuples are published in the network. The queries that involve the unindexed data can be processed via an approximate index (range index). To catch the query patterns, the index is dynamically tuned, where popular data are continuously published 154 7.1. CONCLUSION for indexing and rarely queried data are discarded from the index. • Most work on modern query optimizer is focusing on generating a good plan for multi-way join, because different join plans will affect the performance significantly. In PDMS, processing join incurs data transfer between nodes, which is much more expensive than local I/Os. And, different from centralized systems, where precise statistics are available, in PDMS, query plans are generated based on estimations. Hence, it is both important and challenging to optimize multi-way join queries in PDMS. The thesis tries to address the above problem by using an adaptive optimization approach. Specifically, an initial query plan is generated based on the approximate estimation. And the query is processed based on the plan. As query is being processed, real-time statistics are collected and a new plan is produced. The new plan is used to replace the old one and the query continues via the new plan. The query plan adjustment is invoked periodically until the query is fully processed. The adaptive approach can effectively avoid the bad plan generated by the incorrect statistics. • Aggregate queries are very useful in decision making applications. Compared to other queries, aggregate queries are always expensive to process. Therefore, we adopt an approximate query processing approach to reduce the processing cost. The approach is based on the assumption that approximate results with error bounds and confidences are acceptable for most aggregate queries. The approximate approach continuously retrieves samples from local databases. And an approximate result is generated and refined. To process queries in parallel, linear hash function is adopted to disseminate samples among the processing nodes. We dynamically adjust the number of processing nodes for 155 7.2. FUTURE WORK a query to improve the overall throughput. If the user does not require precise results, the approximate processing scheme can improve the performance by an order of magnitude. Our proposed schemes are evaluated on an open distributed platform, PlanetLab [11]. And a widely used database benchmark, TPC-H [14], is employed as our test dataset. The results show the efficiency and effectiveness of above schemes. 7.2 Future Work P2P technique is evolving after several years of development. Some P2P systems change their design philosophy to embrace the new Cloud wave. PDMS is one of such systems. In PDMS for enterprise applications, the network is stable, because the nodes join the systems for sharing information and improving the productivity of the whole industry. The nodes will not leave or join the systems frequently, which is different from the other P2P networks. Moreover, an administrator is required to manage the membership and coordinate between the participants. Therefore, PDMS for enterprise applications is more like a Cloud system, rather than a conventional P2P system. In the future, we would like to apply our approaches to the Cloud system and work on the following directions. We plan to propose a new indexing framework for the Cloud systems by exploiting the routing protocols of P2P networks. In our previous work, BATON is employed as the underlying overlay for the PDMS. As BATON keeps the localities for one-dimensional data, we can easily build an index to support range queries. But it is challenging to implement a multi-dimensional index in BATON. In other words, the index approach in current PDMS is limited by the underlying overlay. We can 156 7.2. FUTURE WORK only design the index that is supported by the overlay partitioning and routing protocols. To address this problem, a new indexing framework, which is independent of the underlying overlay, is required. As a matter of fact, most P2P overlays follow the principles of Cayley graph [72]. The new indexing framework can be designed based on this observation. Namely, we can define a Cayley graph and generate other virtual overlays, when necessary (e.g. for building index). In this way, different types of indexes can be supported and we avoid maintaining multiple overlays. Another lesson learned from our implementation experience is that, in largescale distributed systems, parallel processing is more important than other optimizations. For example, in a multi-way join query R1 ⊲⊳R1 .a=R2 .a R2 ⊲⊳R2 .a=R3 .a R3 , if R1 ⊲⊳R1 .a=R2 .a R2 only generates a few results, the optimizer will generate a plan by joining R1 with R2 first and delaying the join with R3 . However, if all three relations are partitioned by the join attribute a, a better approach is to perform the multi-way join at the involved nodes in parallel. Even the processing cost is higher, by exploiting more computation nodes, the parallel plan leads to a lower response time than the former plan. Therefore, the cost-based optimizer is not enough in the PDMS. We should consider the parallelism when generating query plans. The goal of the new optimizer is to reduce the processing costs and at the same time, maximizes the level of parallelism. We need to achieve a good trade-off between the costs and parallelism. In the thesis, we try to optimize a single query in most cases. But in some systems, multiple concurrent queries may be submitted to the system for processing simultaneously. As queries may share some common expressions, multi-query optimization may be more efficient than optimizing the query individually. We can process the common sub-queries and maintain their results in temporary tables, which 157 7.2. FUTURE WORK are used as inputs for multiple queries. We reduce the processing costs by avoiding repeatedly evaluating a sub-query. Multi-query optimization has been studied in centralized systems. In our scenario, as data are partitioned among the participants and queries are processed by different processing nodes, optimizing multiple queries together is more difficult. The nodes need to collaborate each other to share their partial results. A possible solution is to employ a master server to manage the query optimization. 158 Bibliography [1] http://data.aolsearchlogs.com. [2] http://news.techworld.com/operating-systems/3477/moores-law-is-deadsays-gordon-moore. [3] http://peercast.org. [4] http://shareaza.sourceforge.net. [5] http://www.bestpeer.com. [6] http://www.gnu.org/philosophy/gnutella.htm. [7] http://www.kazaa.com. [8] http://www.limewire.com. [9] http://www.macnn.com/news/27315. [10] http://www.napster.com. [11] http://www.planet-lab.org. [12] http://www.pps.tv. [13] http://www.skype.com. [14] http://www.tpc.org/tpch. [15] K. Aberer. P-grid: A self-organizing access structure for p2p information systems. In CooplS, pages 179–194, 2001. 159 BIBLIOGRAPHY [16] S. Acharya, P. B. Gibbons, and V. Poosala. Congressional samples for approximate answering of group-by queries. In SIGMOD, pages 487–498, 2000. [17] S. Acharya, P. B. Gibbons, V. Poosala, and S. Ramaswamy. Join synopses for approximate query answering. SIGMOD Rec., 28(2), 1999. [18] S. Antony, A. E. Abbadi, and D. Agrawal. P2p systems with transactional semantics. In EDBT, 2008. [19] B. Arai, G. Das, D. Gunopulos, and V. Kalogeraki. Approximating aggregation queries in peer-to-peer networks. In ICDE, 2006. [20] B. Arai, S. Lin, and D. Gunopulos. Efficient data sampling in heterogeneous peer-to-peer networks. In ICDM, 2007. [21] J. Aspnes and G. Shah. Skip graphs. In Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 384–393, 2003. [22] R. Avnur and J. M. Hellerstein. Eddies: Continuously adaptive query processing. In SIGMOD Conference, pages 261–272, 2000. [23] A. Awan, R. A. Ferreira, S. Jagannathan, and A. Grama. Distributed uniform sampling in unstructured peer-to-peer networks. In HICSS, 2006. [24] A. Awan, R. A. Ferreira, S. Jagannathan, and A. Grama. Distributed uniform sampling in unstructured peer-to-peer networks. In HICSS, 2006. [25] O. Axelsson. Iterative solution methods. Cambridge University Press, 1994. [26] B. Babcock, S. Chaudhuri, and G. Das. Dynamic sample selection for approximate query processing. In SIGMOD, pages 539–550, 2003. 160 BIBLIOGRAPHY [27] M. Bawa, A. Gionis, H. Garcia-Molina, and R. Motwani. The price of validity in dynamic networks. In SIGMOD Conference, pages 515–526, 2004. [28] A. R. Bharambe, M. Agrawal, and S. Seshan. Mercury: supporting scalable multi-attribute range queries. SIGCOMM Comput. Commun. Rev., 34(4):353– 366, 2004. [29] K. Bharath-Kumar and J. M. Jaffe. Routing to multiple destinations in computer networks. In IEEE Transactions on Communications, pages 343–351, 1983. [30] M. W. Blasgen, M. M. Astrahan, D. D. Chamberlin, J. Gray, W. F. K. III, B. G. Lindsay, R. A. Lorie, J. W. Mehl, T. G. Price, G. R. Putzolu, M. Schkolnick, P. G. Selinger, D. R. Slutz, H. R. Strong, I. L. Traiger, B. W. Wade, and R. A. Yost. System r: An architectural overview. IBM Systems Journal, 20(1):41– 62, 1981. [31] S. B¨orzs¨onyi, D. Kossmann, and K. Stocker. The skyline operator. In ICDE, 2001. [32] P. Bratley and B. L. Fox. Algorithm 659. implementing sobol’s quasirandom sequence generator. ACM Transactions on Mathematical Software, 14(1):88– 100, March 1988. [33] H. Cai and J. Wang. Foreseer: a novel, locality-aware peer-to-peer system architecture for keyword searches. In Middleware, pages 38–58, 2004. [34] S. Chaudhuri, G. Das, M. Datar, R. Motwani, and V. R. Narasayya. Overcoming limitations of sampling for aggregation queries. In ICDE, 2001. 161 BIBLIOGRAPHY [35] A. Crainiceanu, P. Linga, A. Machanavajjhala, J. Gehrke, and J. Shanmugasundaram. P-ring: an efficient and robust p2p range index structure. In SIGMOD, pages 223–234, 2007. [36] A. Crespo and H. Garcia-Molina. Routing indices for peer-to-peer systems. In ICDCS, page 23, 2002. [37] F. M. Cuenca-Acuna, C. Peery, R. P. Martin, and T. D. Nguyen. Planetp: Using gossiping to build content addressable peer-to-peer information sharing communities. In HPDC, page 236, 2003. [38] S. Datta and H. Kargupta. Uniform data sampling from a peer-to-peer network. In ICDCS, 2007. [39] M. de Berg, O. Schwarzkopf, M. van Kreveld, and M. Overmars. Computational geometry : algorithms and applications. Springer-Verlag, 2000. [40] A. Deshpande, Z. G. Ives, and V. Raman. Adaptive query processing. Foundations and Trends in Databases, 1(1):1–140, 2007. [41] P. Ganesan, B. Yang, and H. Garcia-Molina. One torus to rule them all: multidimensional queries in p2p systems. In WebDB, pages 19–24, 2004. [42] P. Ganesan, B. Yang, and H. G. Molina. One Torus to Rule Them All: Multidimensional Queries in P2P Systems. In WebDB, 2004. [43] C. Gkantsidis, C. Gkantsidis, M. Mihail, and A. Saberi. Random walks in peer-to-peer networks: algorithms and evaluation. Perform. Eval., 63(3), 2006. 162 BIBLIOGRAPHY [44] S. D. Gribble, A. Y. Halevy, Z. G. Ives, M. Rodrig, and D. Suciu. What can database for peer-to-peer? In WebDB, pages 31–36, 2001. [45] P. K. Gummadi, S. Saroiu, and S. D. Gribble. A measurement study of napster and gnutella as examples of peer-to-peer file sharing systems. Computer Communication Review, 32(1):82, 2002. [46] L. M. Haas, D. Kossmann, E. L. Wimmers, and J. Yang. Optimizing queries across diverse data sources. In VLDB, pages 276–285, 1997. [47] P. J. Haas. Large-sample and deterministic confidence intervals for online aggregation. In SSDBM, 1997. [48] P. J. Haas and J. M. Hellerstein. Ripple joins for online aggregation. In SIGMOD, 1999. [49] M. Harren, J. M. Hellerstein, R. Huebsch, B. T. Loo, S. Shenker, and I. Stoica. Complex queries in dht-based peer-to-peer networks. In IPTPS, pages 242– 259, 2001. [50] N. J. A. Harvey, M. B. Jones, S. Saroiu, M. Theimer, and A. Wolman. Skipnet: a scalable overlay network with practical locality properties. In USITS, pages 9–9, 2003. [51] G. Hasslinger and S. Kempken. Efficiency of random walks for search in different network structures. In ValueTools, pages 1–8, 2007. [52] J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online aggregation. In SIGMOD, pages 171–182, 1997. 163 BIBLIOGRAPHY [53] J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online aggregation. In SIGMOD, 1997. [54] R. Huebsch, J. M. Hellerstein, N. Lanham, B. T. Loo, S. Shenker, and I. Stoica. Querying the internet with pier. In VLDB, pages 321–332, 2003. [55] R. Huebsch and S. Jeffrey. Freddies: Dht-based adaptive query processing via federated eddies, 2004. [56] H. V. Jagadish, B. C. Ooi, K.-L. Tan, Q. H. Vu, and R. Zhang. Speeding up search in peer-to-peer networks with a multi-way tree structure. In SIGMOD, pages 1–12, 2006. [57] H. V. Jagadish, B. C. Ooi, and Q. H. Vu. Baton: a balanced tree structure for peer-to-peer networks. In VLDB, pages 661–672, 2005. [58] H. V. Jagadish, B. C. Ooi, Q. H. Vu, R. Zhang, and A. Zhou. Vbi-tree: A peer-to-peer framework for supporting multi-dimensional indexing schemes. In ICDE, page 34, 2006. [59] M. Jelasity, R. Guerraoui, A.-M. Kermarrec, and M. van Steen. The peer sampling service: experimental evaluation of unstructured gossip-based implementations. In Middleware, 2004. [60] M. Jelasity, A. Montresor, and O. Babaoglu. Gossip-based aggregation in large dynamic networks. ACM Trans. Comput. Syst., 23(3), 2005. [61] C. Jermaine, A. Dobra, S. Arumugam, S. Joshi, and A. Pol. A disk-based join with probabilistic guarantees. In SIGMOD, 2005. 164 BIBLIOGRAPHY [62] C. Jermaine, A. Pol, and S. Arumugam. Online maintenance of very large random samples. In SIGMOD Conference, pages 299–310, 2004. [63] N. Kabra and D. J. DeWitt. Efficient mid-query re-optimization of suboptimal query execution plans. In SIGMOD Conference, pages 106–117, 1998. [64] E. Kazumori. Selling online versus offline: theory and evidences from sotheby’s. In Proceedings of the 4th ACM conference on Electronic commerce, 2003. [65] A. Kothari, D. Agrawal, A. Gupta, and S. Suri. Range addressable network: A p2p cache architecture for data ranges. In P2P, 2003. [66] L. T. Kou, G. Markowsky, and L. Berman. A fast algorithm for steiner trees. Acta Inf., 15:141–145, 1981. [67] A. Y. Levy, A. Rajaraman, and J. J. Ordille. Querying heterogeneous information sources using source descriptions. In VLDB, pages 251–262, 1996. [68] W. Litwin, M. anne Neimat, and D. A. Schneider. Lh*—a scalable, distributed data structure. ACM Transactions on Database Systems, 21, 1996. [69] B. T. Loo, J. M. Hellerstein, R. Huebsch, S. Shenker, and I. Stoica. Enhancing p2p file-sharing with an internet-scale query processor. In VLDB, 2004. [70] G. Luo, C. J. Ellmann, P. J. Haas, and J. F. Naughton. A scalable hash ripple join algorithm. In SIGMOD Conference, pages 252–262, 2002. [71] M. Lupu, J. Li, B. C. Ooi, and S. Shi. Clustering wavelets to speed-up data dissemination in structured manets. In ICDE, 2007. 165 BIBLIOGRAPHY [72] M. Lupu, B. C. Ooi, and Y. C. Tay. Paths to stardom: calibrating the potential of a peer-based data management system. In SIGMOD Conference, pages 265–278, 2008. [73] Q. Lv, P. Cao, E. Cohen, K. Li, and S. Shenker. Search and replication in unstructured peer-to-peer networks. In ICS, pages 84–95, 2002. [74] G. S. Manku. Balanced binary trees for id management and load balance in distributed hash tables. In PODC, 2004. [75] G. S. Manku, M. Naor, and U. Wieder. Know thy neighbor’s neighbor: the power of lookahead in randomized p2p networks. In STOC, 2004. [76] V. Markl, V. Raman, D. E. Simmen, G. M. Lohman, and H. Pirahesh. Robust query processing through progressive optimization. In SIGMOD Conference, pages 659–670, 2004. [77] S. Merugu, S. Srinivasan, and E. Zegura. Adding structure to unstructured peer-to-peer networks: the use of small-world graphs. J. Parallel Distrib. Comput., 65(2):142–153, 2005. [78] S. Michel, P. Triantafillou, and G. Weikum. Klee: a framework for distributed top-k query algorithms. In VLDB, pages 637–648, 2005. [79] F. Olken. Random sampling from databases. 1993. [80] F. Olken and D. Rotem. Maintenance of materialized views of sampling queries. In ICDE, 1992. [81] K. Ono and G. M. Lohman. Measuring the complexity of join enumeration in query optimization. In VLDB, pages 314–325, 1990. 166 BIBLIOGRAPHY [82] B. C. Ooi, K.-L. Tan, A. Zhou, C. H. Goh, Y. Li, C. Y. Liau, B. Ling, W. S. Ng, Y. Shu, X. Wang, and M. Zhang. Peerdb: peering into personal databases. In SIGMOD, pages 659–659, 2003. [83] T. Pitoura, N. Ntarmos, and P. Triantafillou. Replication, load balancing and efficient range query processing in dhts. In EDBT, pages 131–148, 2006. [84] S. Podlipnig and L. B¨osz¨ormenyi. A survey of web cache replacement strategies. ACM Comput. Surv., 35(4), 2003. [85] V. Poosala and Y. E. Ioannidis. Selectivity estimation without the attribute value independence assumption. In VLDB, pages 486–495, 1997. [86] R. Ramakrishnan and J. Gehrke. Database Management Systems (3rd Edition). 2002. [87] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Schenker. A scalable content-addressable network. In SIGCOMM, pages 161–172, 2001. [88] R.Nelson and A. Tantawi. Approximate analysis of fork/join synchronization in parallel queues. 37, 1988. [89] G. Robins and A. Zelikovsky. Improved steiner tree approximation in graphs. In SODA, pages 770–779, 2000. [90] P. Rodr´ıguez-Gianolli, A. Kementsietsidis, M. Garzetti, I. Kiringa, L. Jiang, M. Masud, R. J. Miller, and J. Mylopoulos. Data sharing in the hyperion peer database system. In VLDB, pages 1291–1294, 2005. [91] P. Rosch, K.-U. Sattler, C. von der Weth, and E. Buchmann. Best effort query processing in dht-based p2p systems. In NetDB, page 1186, 2005. 167 BIBLIOGRAPHY [92] A. Rowstron and P. Druschel. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In Middleware, pages 329–350, 2001. [93] J. M. H. B. T. L. P. M. T. R. S. S. I. S. Ryan Huebsch, Brent Chun and A. R. Yumerefendi. The architecture of pier: an internet-scale query processor. In CIDR, 2005. [94] O. D. Sahin, A. Gupta, D. Agrawal, and A. E. Abbadi. A peer-to-peer framework for caching range queries. In ICDE, 2004. [95] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. In SIGCOMM, pages 149–160, 2001. [96] M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. O’Neil, P. O’Neil, A. Rasin, N. Tran, and S. Zdonik. C-store: a column-oriented dbms. In VLDB, pages 553–564, 2005. [97] D. Stutzbach, R. Rejaie, N. Duffield, S. Sen, and W. Willinger. On unbiased sampling for unstructured peer-to-peer networks. In IMC, 2006. [98] A. Swami. Optimization of large join queries: combining heuristics and combinatorial techniques. In SIGMOD, pages 367–376, 1989. [99] K.-L. Tan, C. H. Goh, and B. C. Ooi. Online feedback for nested aggregate queries with multi-threading. In VLDB, 1999. 168 BIBLIOGRAPHY [100] I. Tatarinov, Z. Ives, J. Madhavan, A. Halevy, D. Suciu, N. Dalvi, X. L. Dong, Y. Kadiyska, G. Miklau, and P. Mork. The piazza peer data management project. SIGMOD Rec., 32(3):47–52, 2003. [101] P. Triantafillou and T. Pitoura. Towards a unifying framework for complex query processing over structured peer-to-peer data networks. In DBISP2P, pages 169–183, 2003. [102] B. Vance and D. Maier. Rapid bushy join-order optimization with cartesian products. SIGMOD Rec., 25(2):35–46, 1996. [103] V. V. Vazirani. Approximation Algorithms. 2001. [104] S. Wang, Q. H. Vu, B. C. Ooi, A. K. Tung, and L. Xu. Skyframe: a framework for skyline query processing in peer-to-peer systems. The VLDB Journal, 18(1):345–362, 2009. [105] B. Yang and H. Garcia-Molina. Designing a super-peer network. In ICDE, pages 49–60, 2003. [106] B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, and J. Kubiatowicz. Tapestry: A resilient global-scale overlay for service deployment. IEEE Journal on Selected Areas in Communications, 22(1), 2004. 169 [...]... in the PDMS, SQL-like queries must be 3 1.2 QUERY PROCESSING IN PDMS supported to enable the database clients to use the system without further training From the view of users, PDMS should provide a similar interface as the distributed database systems 1.2 Query Processing in PDMS To efficiently process queries in PDMS, we can adopt the techniques in conventional databases But compared to other P2P systems... for multi-join queries But in the PDMS, we are deprived of global information, which is required to generate an optimized query plan Due to node autonomy, we need to collect the statistics of data and query distribution on the fly 4 1.2 QUERY PROCESSING IN PDMS • In P2P systems, network cost dominates the total processing cost, while in conventional database systems, local disk I/O is the main concern... an optimal join sequence When processing multi-way join queries in PDMS, we can reuse their techniques Compared to conventional database systems, optimizing multi-way join in PDMS is even more challenging In PDMS, each node maintains a local database If one node tries to get a global view of the system, it needs to issue queries, which join data from multiple nodes To perform a join, data are shuffled... overlays and structured overlays In unstructured overlays, the node maintains an index for the data in 19 2.2 PDMS its neighborhood [37], whereas in structured overlays, the node publishes its index based on the routing protocols [54] In either case, the nodes need to share local storage to maintain the indexes for the remote data Compared to the unstructured overlays, the index in the structured overlays... conventional database systems, query processing in PDMS is extremely difficult as: • To process a query, we must translate it into a physical plan and send the plan to corresponding nodes, which will collaborate with each other to process the query The physical plan must define the access methods and the ways in which the data are transferred among the nodes • In conventional database systems, index can be used to. .. 21 2.3 ADAPTIVE QUERY PROCESSING R4 R3 R1 R2 R1 R2 R3 R4 Figure 2.3: Adaptive Query Processing 2.3 Adaptive Query Processing In the traditional database systems, statistics of data distribution are collected via histograms and the query processor applies the statistics to generate optimized query plan The same process can be employed in PDMS to improve the query performance However, in PDMS, we cannot... SYSTEM) provide a high-scalable data management systems for P2P networks [44] However, the PDMS (Peer- based Data Management System) is significantly different from conventional database systems in several ways 1 There is no master server in PDMSs 2 Data are maintained by each node individually and queries are processed in a full distributed manner 3 Compared to distributed database system, which typically has... cost dominates the processing cost In this thesis, we address the multi-way join problem by proposing a new optimization model Based on some approximate histograms, an initial query plan is constructed to reduce the total processing cost Due to lack of global information in PDMS, the optimizer dynamically adjusts the query plan in run-time, if it finds a better plan than current one 1.2.3 Aggregate Query. .. Aggregate Query Aggregation query plays an important role in decision making systems In PDMS, if a company wants to know the statistics of its partners, it can issue an aggregate query to the whole network In TPC-H, Q3 is submitted to the system to compute 7 1.2 QUERY PROCESSING IN PDMS the average prices of orders within specific date Q3 : SELECT average(o.totalprice), l.linestatus FROM lineitem l, orders o... autonomous system with few or no administration And they lack of efficient query processing and schema support for data intensive applications Therefore, those systems cannot be used to support enterprise applications, which are widely deployed on conventional database systems Recently, database community attempts to exploit the database technology to 1 1.1 PDMS (PEER- BASED DATA MANAGEMENT SYSTEM) provide . Query Processing in Peer-to-Peer Based Data Management System Sai Wu A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE 2011 2 Query. SINGAPORE 2011 2 Query Processing in Peer-to-Peer Based Data Management System Sai Wu School of Computing National University of Singapore Summary In last ten years, we have witnessed the success of P2P (Peer-to-Peer) . distributed index to pro- cess it. To reduce the maintenance cost of indexes, we propose a just -in- time indexing approach. Instead of indexing the whole dataset, we selectively pub- lish the data based