2009 International Conference on Knowledge and Systems Engineering Building a low-latency, proximity-aware DHT-based P2P network Ngoc Ben DANG, Son Tung VU, Hoai Son NGUYEN Department of Computer network College of Technology, Vietnam National University, Hanoi 144 Xuan Thuy, Cau Giay, Hanoi, Vietnam This paper focuses on the problem of improving the latency of message routing in Chord, a typical DHT-based P2P network protocol There are two main approaches to solve this problem: proximity neighbor selection (PNS) [5], [7], [9] and proximity identifier selection (PIS) [4], [8] In the PNS approach, a node will choose its neighbor nodes based on their proximity In the PIS approach, each node chooses its node identifier based on its geographic location To realize the PIS approach, conventional researches [4], [8] map participating nodes into a proximity-based twodimensional space and assigns node identifier in Chord based on the node position in the two-dimensional space such that nodes that are close in identifier are close in the two-dimensional space This solution has a drawback of requiring a set of landmarks and a pre-processing phase for nodes to compute their own coordinates in the proximitybased two-dimensional space This paper proposes a simple solution to solve the topology mismatch problem for Chord protocol The main idea of the solution consists of two modifications of Chord protocol First, in the node joining phase, a new node can select a position in the Chord ring from a set of randomly selfgenerated positions such that the latency to its successor node and predecessor node is the smallest Second, in the stabilization phase, each node selects a neighbor node for each finger table entry from a set of candidate nodes such that the latency to each neighbor node is small Our method is easy to implement and has small overhead comparing with conventional methods With minor modifications of Chord protocol, however, our method can reduce the latency of message routing compared to the conventional Chord protocol In order to evaluate the effectiveness of our method, we have built a simulator to simulate the execution of our algorithm and measure the latency of message routing between participating nodes in the network The simulation results show that our method can reduce the average latency of message routing by approximately 30%, compared to the conventional Chord protocol Our method also does not affect much on the random mapping of identifier in the Chord ring The rest of this paper is structured as follows Section explains Chord protocol and related works with their pros and cons Section is dedicated to describe our proposal Abstract—DHT-based P2P networks have a problem of topology mismatch which causes high latency for message routing This paper focuses on improving the latency of routing process for Chord, which is a typical DHT-based P2P network protocol We propose a simple method to build a Chord network based on the proximity of nodes in the underlying physical network The idea of our method is the combination of two techniques: identifier selection in node joining phase and neighbor selection in network stabilization phase, both are performed based on physical network latency We have evaluated our proposed solution by simulations In a simulation network with 4096 nodes, the average latency of routing process in our method can reduces 30% comparing with the conventional Chord routing method Keywords-DHT;Chord;low latency;proximity-aware I I NTRODUCTION Peer-to-peer network applications are becoming more and more popular in the Internet today Comparing to traditional server-client network model, peer-to-peer networks have many advantages like its scalability, self-organization and fault tolerance Recently, Distributed Hashing Table (DHT) algorithms such as Chord [1], CAN [2], Pastry [3] attract lots of attention since they can offer a promising solution for scalable and efficient message routing on P2P networks The algorithm constructs a structure of a virtual key space on a P2P network where each node is responsible for a portion of key space and keys are used as destinations to route messages Each node also maintains information about a small set of neighbor nodes, which are selected based on the positions of nodes in the virtual key space The neighbor sets are used to route messages to destination nodes efficiently However, since the construction of a DHT-based P2P network is performed regardless of physical network topology, there are differences between overlay network topology and physical network topology (called ”topology mismatch”) For example, two nodes which are near in the underlying physical network (i.e the latency of message routing between them is small), may be far away (i.e several hops away) from each other in the DHT-based network In this case, the latency of message routing between these nodes in the DHT-based network is significantly higher compared with the latency between them in the underlying physical network 978-0-7695-3846-4/09 $26.00 © 2009 IEEE DOI 10.1109/KSE.2009.49 195 method Section describes our simulator and simulation results Conclusion is written in section II BACKGROUND A Chord protocol Chord is a typical structured peer-to-peer network protocol using an m-bit address space which includes 2m identifiers arranged in a circle topology called a Chord ring (Fig 1) Using a consistent hash function, each participating node randomly chooses an identifier and then joins to a position in the ring by a joining phase A resource is distributed to a Chord network by the use of key-value pairs: the key is the resourcefs identifier and the value is the resource itself Each key-value pair is stored in the node which is nearest the key in Chord ring clockwise (i.e the successor of the key) In this paper, we focus on the construction of Chord ring, which includes a joining phase and a finger table building phase The joining phase of a node to a Chord network is rather simple When a node wants to join a Chord network, it will generate its identifier by making use of a consistent hash function on its own information such as its IP/Port pair The joining node then sends a joining message to a well-known node in the Chord network (so-called ”bootstrap node”), in order to query for its identifier’s successor Once the successor information is returned, the joining node will contact to the successor to join the position next to the successor counter-clockwise in Chord ring After that, the node performs a finger table building phase to build its own finger table In an m-bit address space, each participating node builds a finger table which has m entries Assuming that the identifier of the current node is k, the ith entry in the table holds the information about the successor node of identifier (k+2i−1 ) Finger tables are used to route messages between nodes in the network When a node receives a message destined to an identifier in the Chord ring, it will look up its finger table to find the nearest node to the identifier counter-clockwise and forward the message to that node The forwarded node will recursively repeats the same operation The process continues until the message reaches the successor of the destination identifier In Chord protocol, the construction of a finger table is performed regardless of physical network topology (i.e the latency between nodes) Therefore, the latency of message routing between two nodes in Chord network may be significantly higher compared with the latency between these nodes in the underlying physical network Chord also defines some other operations such as node leaving, network recovery from a node failure The interested reader is referred to [1] for these details 1) Related works: There are several conventional solutions for Chord protocol to improve the latency of message Figure An example of a Chord ring routing due to the differences between overlay network topology and underlying physical network topology In the approach of Proximity Neighbor Selection (PNS), each finger table entry holds the information of a node that is selected from a number of nodes, which belong to a predetermined interval of identifiers, based on their proximity The effectiveness of this approach is showed in [5], [7], [9] However, actual realization of this approach is different between conventional researches In [7], the authors propose a method that makes use of latency information collected from lookup process on the overlay network After a node joins a Chord network, it will gradually obtain lowlatency finger table by updating its finger table entries based on collected latency information In [9], the authors propose the use of Vivaldi algorithm [11] to calculate the position of a node on a virtual two-dimensional space so that distances in the space approximate distances in the underlying physical network Each node then selects neighbor nodes that are physically nearby by looking up for nodes that are close to it on the virtual 2-dimension space In the approach of Proximity Identifier Selection (PIS), a node will choose its node identifier based on its geographic location Like [9], the work [4] and Quasi-Chord [8] uses a set of landmarks to map nodes into points in a twodimensional space The identifier of nodes are selected such that nodes are close in a two-dimensional space are close in the identifier space In common, both of approaches try to decrease the latency of message routing on a DHT network However, they require a set of landmarks and a preprocessing phase for nodes to compute their own coordinates in the proximity-based two-dimensional space 196 III OUR APPROACH A Overview In this chapter, we propose a new solution for the topology mismatch problem of Chord protocol The solution achieves the same goals as related works, but with another approach which requires less cost and can be deployed in a large-scale network easily The idea of our approach is that a node can select a position that it can join in a Chord ring such that nodes that are close to it in the Chord ring are physically nearby to it As the result, nodes are close in a Chord ring will be close regarding to physical network positions The idea comes from a consideration that the proximity neighbor selection approach is not effective in upper entries of finger table It is because the number of nodes that can be selected as an upper entry of a finger table (i.e belong to a predetermined interval of identifiers) is often small Since the very last hops of message routing in Chord usually use the upper entries of finger tables, we need to improve the latency due to the last hops by keeping physically nearby nodes as close together in a Chord ring as possible Unlike conventional researches which force a joining node to calculate its physical position, in our method, we allow a joining node to select its identifier in a set of randomly generated identifiers, each of which corresponds to a position in the Chord ring The node selects an identifier as its joining position such that the latency to the successor node and the predecessor node of the selected identifier is the smallest in the set of candidate identifiers Our method also includes a proximity neighbor selection phase The ith entry of the finger table of a node k will be selected from a set of candidate nodes which are next to the successor node of the identifier (k+2i−1 ) The selected node must satisfy the condition that the latency from the selected node to node k is the smallest in the set of candidate nodes The details of our method are described as follows Figure Pseudo-code of proximity-based identifier selection phase CHOICE: the size of set C of identifiers B Our method Our method includes two phases • Proximity-based identifier selection phase When a new node joins in a Chord network, instead of having the node to generate only one identifier to join in the network, we let the node to generate a set C of identifiers The set C is generated randomly by making use of a hash function on the combination of the nodefs own information and a random value The node then contacts to a bootstrap node in the Chord network to query the successor node and the predecessor node of each identifier in the set C We call CHOICE the size of set C of identifiers The node then measures the network latency from it to the successor node and the predecessor node of each identifier The latency to a node can be simply measured by a ping protocol It then selects the identifier, Figure Pseudo-code of proximity-based neighbor selection phase which successor node or predecessor node has the smallest latency to it, as its identifier to join in the Chord network The pseudo-code for the algorithm is shown in Fig • 197 Proximity-based neighbor selection phase To update an entry in the finger table, first the node, which identifier is k for instance, queries the successor node s of the identifier (k + 2i−1 ) based on the Chord protocol Then, node k queries the successor node s and nodes next to the successor node to get a set of candidate nodes which are next to the successor clockwise and counter-clockwise in the Chord ring The set of candidate nodes has * EXPANSION + nodes including the successor node s, EXPANSION nodes next to node s counter-clockwise and EXPANSION nodes next to s clockwise Here, we call EXPANSION the spread degree To avoid duplication of nodes in multiple entries of the finger table, nodes in the set which are not between k + 2i−1 and k + 2i+1 will be not considered The pseudo-code for the algorithm is shown in Fig By probing for the latency from node k to each node in the set of candidate nodes, we can select the desired node for the entry of the finger table which latency from node k to the selected node is the smallest in the set The update of finger table entries is performed whenever the network stabilization phase of Chord protocol is run Hence, it can adapt to the join/leave of nodes With high probability, the selection of node identifier based on network latency can keep the latency among nearby nodes in a Chord network relative small Hence, it takes a short time to route messages of which destinations are nearby in a Chord ring Furthermore, the update of finger tables based on network latency can reduce routing time to forward query messages among nodes Therefore, our solution can achieve the goal of reducing latency of message routing among participating nodes in a Chord network The solution has a drawback of delaying the joining process of new nodes Comparing to the original Chord protocol, the solution requires a time interval to probe for the latency from a joining node to a number of randomly generated set of node IDs Updating routing table process also needs some time interval to select one node from a set of candidate nodes The more nodes or identifiers there are in the sets, the more optimal degree of latency we get and the longer time it takes for nodes to join in the network or update routing tables It is a trade-off and we need to run simulations to get reasonable parameter values Figure Average query latency comparison information and the returned results are collected to build statistics The simulation network topology is divided into separated domains Each domain has a number of participating nodes Nodes in the same domain are connected to each other to create a star topology with a center switch All the domains’ switches are connected directly to each other Each of connections between nodes and their domainfs switch, and between switches has its own latency The latency between node A and node B is calculated by summing up the latency between node A and its domain’s switch and the latency between the switch to node B if A and B are in the same domain Otherwise, the latency is calculated as the sum of the latency between node A and its domain’s switch, the latency between the switch of A’s domain and the switch of B’s domain and the latency between the switch of B’s domain and node B With the simulator, we can specify following parameters: number of participating nodes, number of domains, and number of nodes in a domain and latency of connections We divide experiment time into intervals of time called ”round” To simulate the join/leave of nodes, the simulator makes nodes join in and leave the network randomly after each round The join/leave timing of nodes is generated based on the Pareto distribution Then, all the nodes’ routing tables are updated and the simulator generates queries The average latency in each round after updating routing tables and the overall average latency of the experiment are stored to build the statistics for reviewing later IV EVALUATION To evaluate the efficiency of the proposed solution, we have performed experiments with a simple network simulator Our simulation focuses on measuring the difference of message routing latency between the conventional Chord and our proposed method B Simulation results The first experiment aims to compare our solution’s efficiency to conventional Chord’s The simulator creates 4096 nodes which are divided into 32 domains, with EXPANSION value of and CHOICE value of and 16 respectively The latency of connections between nodes in a domain is assigned randomly from to 30 ms, and the latency between A Simulator We build a simulator which generates a physical network topology of participating nodes and maps it to conventional and improved Chord overlay network Generated queries are routed among participating nodes based on routing 198 Figure Figure Average query latency comparison Figure Average latency with CHOICE adjustment Average latency with EXPANSION adjustment Figure The distribution of ID intervals between adjoining nodes in a Chord ring two switches is assigned randomly from 50 to 250ms The experiment is carried out in 360 rounds and there are about 1000 queries generated in each round The average latency of the network in the first 100 rounds is presented in Fig The result indicates that applying our proposed method can make average query latency in each round be steady and reduce significantly comparing to the conventional Chord protocol After 360 rounds, the average latency in our method with CHOICE=8 is about 759 ms, which equals to 67% of conventional Chordfs result (about 1123 ms) With CHOICE=16, the average latency is improved slightly to 731 ms We also perform the same experiment but increase the number of nodes in a Chord network Figure shows that our solution can reduce the latency of message routing about 33% comparing with conventional Chord protocol even when the number of nodes in a network increases The second experiment focuses on examining the latency optimization degree when we adjust CHOICE value - the number of randomly generated node IDs in the joining phase The simulator parameters are the same as the first experiment: 4096 nodes are divided into 32 domains, EXPANSION value is 3, but there are 3600 rounds and about 1000 queries generated in each round The result is presented in Fig The case in which CHOICE value is can be viewed as conventional Chordfs joining phase and only the selection of routing table entries based on network latency is applied In this case, the average latency is 814ms The more choices of ID for a node in the joining phase are, the better latency optimization degree we can achieve When the CHOICE value is 32, the average latency is 660 ms The third experiment focuses on examining optimization degree of the solution when we adjust EXPANSION value - the number of candidate nodes in the neighbor selection phase As the EXPANSION value increases, each node has more choices to select a node for a routing table entry Hence, we get better latency optimization degree (Fig 7) The random mapping of identifier in Chord rings is one of Chord advantages that we needs to keep Conventional Chord protocol uses a consistent hash algorithm to assign 199 one ID to each newly joining node In our solution, an ID is assigned to a newly joining node based on node proximity Because of the difference between two joining algorithms, we need to examine the affects of our solution on the random mapping of identifier in the Chord ring The simulation of our method is performed with CHOICE=8, EXPANSION =3 The simulator results show the statistics of ID intervals between every pair of adjoining nodes in a Chord ring As shown in Fig 8, with about 2000 available nodes existed in the network, the distribution of ID intervals between adjoining nodes in a Chord ring in the case of our method is similar to the one in the case of conventional Chord protocol It means that our method does not affect much on the random mapping of identifier in the Chord ring [6] F Dabek, J Li, E Sit, J Robertson, M F Kaashoek and R Morris, ”Designing a DHT for low latency and high throughput”, Proceedings of 1st Symposium on Networked Systems Design and Implementation, March 2004 [7] A Goel, R Govindan, H Zhang, ”Improving Lookup Latency in Distributed Hash Table Systems using Random Sampling”, IEEE/ACM Transactions on Networking, Volume 13, February 2005 [8] M Sun, Z Zhang, ”Quasi-Chord: physical topology aware structured P2P network”, Proceedings of the 11th Joint Conference on Information Sciences 2008 December 2008 [9] H Duan, X Lu, H Tang, X Zhou, Z Zhao, ”Proximity Neighbor Selection in Structured P2P Network”, Proceedings of The Sixth IEEE International Conference on Computer and Information Technology, September 2006 V CONCLUSION AND FUTURE WORKS [10] T S Eugene Ng and H Zhang, ”Predicting Internet Network Distance with Coordinates-Based Approaches”, Proceedings of INFOCOM 2002 In the paper, we have proposed a method to optimize Chord topology based on underlying network latency The solution focuses on two modifications of the Chord protocol: selecting the ID in a set of randomly generated IDs from which latency to its next and previous node is smallest and updating routing table by selecting neighbor nodes based on network latency By doing experiments with simulator, we have proved that our method can reduce average query latency significantly but does not affect much on the random mapping of identifier in the Chord ring Moreover, it is rather simple to deploy the solution in Chord-based applications In the near future, we are going to continue to evaluate the efficiency of our method in a real network topology and deployed our method in Chord-based applications [11] R Cox, F Dabek, F Kaashoek, J Li, and R Morris, ”Practical, distributed network coordinates”, Proceedings of the Second workshop on Hot Topics in Networks (HotNetsII) (Nov 2003) ACKNOWLEDGMENT This work is partly supported by the research project No QC.09.12 granted by Vietnam National University, Hanoi R EFERENCES [1] I Stoica, R Morris, D Karger, M.F Kaashoek, H Balakrisnan, ”Chord: A Scalable peer-to-peer lookup service for Internet applications”, In Proceedings of ACM SIGCOMMf01, August 2001 [2] S Ratnasamy, P Francis, M Handley and R Karp, ”A Scalable Content-Addressable Network”, In Proceedings of ACM SIGCOMMf01, Aug 2001 [3] A Rowstron and P Druschel, ”Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems”, In Proceedings of IFIP/ACM International Conference on Distributed Systems Platforms, Nov 2001 [4] S Ratnasamy, M Handley, R Karp and S Shenker, ”Topologically-Aware Overlay Construction and Server Selection”, Proceedings of the INFOCOM 2002 [5] K Gummadi, R Gummadi, S Gribble, S Ratnasamy and S Shenker, I Stoica, ”The Impact of DHT Routing Geometry on Resilience and Proximity”, Proceedings of SIGCOMM 2003 200 ... achieve the goal of reducing latency of message routing among participating nodes in a Chord network The solution has a drawback of delaying the joining process of new nodes Comparing to the original... pair of adjoining nodes in a Chord ring As shown in Fig 8, with about 2000 available nodes existed in the network, the distribution of ID intervals between adjoining nodes in a Chord ring in the. .. its domain’s switch and the latency between the switch to node B if A and B are in the same domain Otherwise, the latency is calculated as the sum of the latency between node A and its domain’s