A network centric approach to enhancing the interactivity of large scale distributed virtual environments

This article was originally published in a journal published by Elsevier, and the attached copy is provided by Elsevier for the author’s benefit and for the benefit of the author’s institution, for non-commercial research and educational use including without limitation use in instruction at your institution, sending it to specific colleagues that you know, and providing a copy to your institution’s administrator All other uses, reproduction and distribution, including without limitation commercial reprints, selling or licensing copies or access, or posting on open internet sites, your personal or institution’s website or repository, are prohibited For exceptions, permission may be sought for such use through Elsevier’s permissions site at: http://www.elsevier.com/locate/permissionusematerial Computer Communications 29 (2006) 3553–3566 www.elsevier.com/locate/comcom co p y A network-centric approach to enhancing the interactivity of large-scale distributed virtual environments Duong Nguyen Binh Ta *, Suiping Zhou School of Computer Engineering, Nanyang Technological University, Singapore 639798, Singapore al Received 25 December 2005; received in revised form 31 May 2006; accepted 31 May 2006 Available online 27 June 2006 Abstract pe rs on In Distributed Virtual Environments (DVEs), where many simultaneous human users interact with each other in a shared, 3D virtual world, enhancing interactivity is of crucial importance, since the success of a DVE greatly depends on users’ perceptions In general, enhancing interactivity for large-scale DVEs with thousands of geographically distributed users faces various challenges such as large Internet latencies, high resource demands and high potential of security threats In this paper, we propose a network-centric approach to enhance the interactivity of DVEs by directly minimizing the network latencies in client–server communications We consider two key problems: the server placement problem and the client assignment problem We have formulated these two problems and proved that they are both NP-hard We then adapt and develop several degree-based server placement approaches and greedy client assignment algorithms Extensive simulation studies using realistic models have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs Ó 2006 Elsevier B.V All rights reserved Introduction or 's Keywords: Distributed Virtual Environments; Geographically distributed server architecture; Server placement; Client assignment; Interactivity enhancement Au th In recent years, advances in networking technologies, computer graphic and CPU power have popularized the deployments of Distributed Virtual Environments (DVEs) Large-scale DVEs refer to a class of applications that enable thousands of geographically distributed users1 to simultaneously interact with each other in a shared, computer-generated 3D virtual world, where each client is represented by an avatar [30] A client controls the behavior of his/her avatar by various inputs, and the updates of an avatar’s state need to be sent to other clients in the same zone of the virtual world to support the interactions among * Corresponding author Tel.: +6591186750; fax: +6567926559 E-mail addresses: binhduong@pmail.ntu.edu.sg (D N B Ta), asspzhou@ntu.edu.sg (S Zhou) In this paper, the terms ‘‘user’’, ‘‘participant’’ and ‘‘client’’ all refer to DVE users and are used interchangeably 0140-3664/$ - see front matter Ó 2006 Elsevier B.V All rights reserved doi:10.1016/j.comcom.2006.05.015 clients Prominent applications of DVEs include online games, military simulations, collaborative designs, virtual shopping mall, etc The latest study by the market research company IDC (http://www.idc.com) on the online gaming market of Asia/Pacific (excluding Japan) revealed that the subscription revenue is about US$1.09 billion in 2004, an increase at about 30% compared to 2003 Also according to this study, this market will be more than doubled in 2009 Enhancing the interactivity of DVEs is of crucial importance, since the users may not feel that they are interacting with other elements in the virtual world if the responses from the DVE are much slower than what the users experience in real-life However, in general, seeking a good balance between enhancing interactivity and other important issues in DVEs such as maintaining consistency, managing resource bottlenecks and securing the virtual world system under various technical problems such as large Internet latency, high resource demands and high 3554 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 co p y and the servers This problem has been studied extensively in the context of Web replica placement [10,27,24] However, to our knowledge there is no existing work that assesses the suitability of Web replica placement approaches to the server placement problem in DVEs In this paper, we look at the state-of-the-art results in the Web replica placement problem We then discuss the difference between the server placement problem in DVEs and the Web replica placement problem Finally, we suggest some possible approaches adapted from Web replica placement strategies that we believe appropriate to the server placement problem in DVEs 2.1 Web replica placement problem on al The explosive growth of the World Wide Web in the late 1990’s had demanded efficient content delivery architectures to offer better service to web users at lower costs A Content Distribution Network (CDN) is a system of many (hundreds or thousands) well-connected servers deployed over the Internet to transparently deliver web contents to end users by allocating the replicas of contents at the edges of the Internet, i.e., close to the users Hence, in large CDNs like Akamai [1], the Web replica placement is crucial to the system performance Informally stated, the Web replica placement problem concerns how to choose K web replicas among N potential sites (i.e., network locations) in the network, where K < N, to minimize a given objective function, usually the clients’ access latencies, or the total bandwidth consumption After the replicas are in place, each web client connects to its closest replica to retrieve the contents needed Here, it is assumed that each client needs to access only one replica to get all of its interested contents The Web replica placement is usually formulated as a minimum K-median problem, which is a well-known NP-hard problem In [27], the authors proposed a greedy placement algorithm that is shown to be able to perform very well compared to a ‘‘super-optimal’’ algorithm3 that uses Lagrangian relaxation with subgradient optimization The authors showed in [27] that the median performance in terms of client–server communication delays of the greedy algorithm is within a factor of 1.1–1.5 of the optimal Au th or 's pe rs potential of security threats is a very challenging task In this paper, we have proposed a network-centric approach to enhancing the interactivity of large-scale DVEs Our methodology is different from the traditional applicationcentric approaches, e.g., dead reckoning, which aims to ‘‘hide’’ the effect of network latencies by predicting the state of remote users at the application level Since the activities of human users in DVEs are highly unpredictable, such approach may result in visual perception errors, or in more serious cases, inconsistent DVE states [34] Our network-centric approach is based on a geographically distributed server architecture (GDSA) [21,9], in which multiple geographically distributed servers are connected to each other Each client is connected to one of these servers, and clients interact with each other through these servers This server-based architecture essentially provides good ways to implement consistency control, resource management and security mechanisms In addition, in order to deal with large-scale DVEs with hundreds, or even thousands of clients interacting simultaneously, usually the virtual world is spatially partitioned into many distinct zones, with each zone managed by only one server, as in [3] A client only interacts with other clients in the same zone,2 and may move to other zones As a server only needs to handle one or more zones instead of the entire virtual world, the system is more scalable In this paper, we refer to such a partitioning approach as the zone-based approach In this paper, we consider two key problems with the GDSA and the zone-based partitioning approach: the server placement problem and the client assignment problem Both problems aim to reduce the client–server communication delays, but the former achieves this goal by finding good locations to place the servers, while the latter does so by assigning clients to appropriate servers We have formulated the two problems and proved that they are both NP-hard To address these two problems, we have developed and adapted several server placement approaches and some client assignment algorithms Extensive simulation studies have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs The rest of the paper is organized as follows Section introduces and addresses the server placement problem in DVEs The client assignment problem and algorithms are presented in Section Section discusses the relations between server placement and client assignment Simulation study is described in Section 5, and Section concludes the paper 2.2 Server placement problem in DVEs A key question in the GDSA is where to place the servers to reduce the communication delay between the clients Although the Web replica placement problem has been well studied in the context of CDNs with promising results, the appropriateness of the proposed approaches to the server placement problem in DVEs has not yet been studied The main difference between the Web replica placement problem and the server placement problem in DVEs lies in the ‘‘connection’’ of clients to servers In the For simplicity, we say that a client is in a zone if its avatar is currently residing in that zone The ‘‘super-optimal’’ solution may be better than the optimal, but may not be feasible Server placement problem 3555 Fig Max-router placement approach al placement In this approach, servers are randomly placed in the network on Remark 2.1 The computational cost of the random placement approach is O (Mm), where M is the total number of routers, and m is the number of servers 2.3.2 Degree-based placement The degree-based server placement approaches aim to place the servers at some key locations in the network These key locations are usually the network nodes (routers or ASes) with high node degrees The degree-based approaches have been shown to perform comparably to the greedy server placement algorithm in [27] in the context of the Web replica placement problem [24] However, whether they are appropriate for DVEs needs to be further investigated or 's pe rs first problem, it is relatively straightforward to let web clients connect to their closest replicas, hence the greedy placement approach is feasible However, in DVEs, the large virtual world is partitioned into multiple distinct zones, and each zone is hosted by a separate server A client cannot simply connect to its closest server since that server may not manage its zone Hence, the greedy placement approach is not appropriate for the server placement problem in DVEs, as it assumes that clients always connect to their closest servers So, for the server placement problem in DVEs, we need an approach that can place the servers in appropriate network locations to reduce clients’ communication latencies without knowing in advance which server the clients will connect to In [10,24], the authors have shown that by placing Web replicas at some key network locations, i.e., at the network nodes (routers or Autonomous Systems (ASes)) with high node degrees, the clients’ communication latencies can be significantly reduced, regardless of the clients’s locations in the network We feel that such approaches are applicable to the server placement problem in DVEs However, the mechanisms in [24,10] were only evaluated under the assumption that clients would eventually connect to their closest servers, although this assumption was not used in deciding server placements Hence, in this paper, we adapt the placement approaches in [24,10] to address the server placement problem in DVEs, and evaluate the appropriateness of these approaches in DVEs To our knowledge, our work is the first to consider server placements to enhance the interactivity of DVEs In the following section, we suggest several possible server placement approaches co p y D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 2.3.1 Random placement The random placement serves as a basic approach for comparison in the sense that any good server placement approach should perform better than the random Indeed, network hop counts are good indications of round-trip network delays [23] Au th 2.3 Server placement approaches for DVEs 2.3.3 Max-router In the Max-router placement approach, we sort all the routers in the network in descending order of their node degrees using Quicksort, and place servers at the routers with the highest node degrees Intuitively, a node with larger node degree is likely to be closer to other nodes, hence it is a good choice for server placement For example in Fig 1, router B has the largest node degree (which is 5) If a server is placed there, then the network distances (in number of hops4) from client c1 and c2 to the server are and 1, respectively However, if the server is placed at router A, which has a node degree of 1, then the above client– server distances become and 2, respectively Thus, placing servers at nodes with large node degree may effectively reduce the client–server communication delay, which implies good interactivity 3556 The computational cost of the Max-router approach is O (M logM) + O (m), or where M is the total number of routers in topology, and m is the number of servers, 3.1 Definitions al Before formulating the CAP, we introduce the following notations and concepts: pe Remark 2.3 The computational cost of the Max-AS/Maxrouter placement approach is O (M logM) + O (M), or O (M logM), where M is the total number of routers Au th or 's 2.3.5 Other placement approaches As comparative references, we also consider some alternative server placement approaches, namely Min-router, Max-AS/Min-router and Min-AS/Max-router The Minrouter approach, which place the servers at the nodes with the minimum node degrees, may serve as an ‘‘upper bound’’ on how bad a server placement approach can be The Max-AS/Min-router5 and Min-AS/Max-router6 may help to identify which (AS or router) is the major factor in the hierarchical server placement approach However, note that in practice, sometimes we have to adopt these methods for server placement since typically the routers with high node degrees may be very busy due to the high volumes of network traffic, thus it may be impossible to add any more service to them [24] Moreover, some economic and administrative constraints may make it harder to place servers at the selected network locations This approach is similar to Max-AS/Max-router, but in each AS, the router with minimum node degree is selected to place the server for that AS This approach is similar to Max-AS/Max-router, but we select the ASes with minimum node degrees, and in each selected AS we select the router with maximum node degree to place the server co p y In CDNs, after the web replica placement is done, assigning clients to servers is relatively straightforward, i.e., each client is connected to its closest server However, in the context of DVEs with the zone-based partitioning approach, this simple client assignment approach is not appropriate, since a client may interact in a zone which is not hosted by the client’s closest server Hence, in this section, independent of any specific server placement, we formulate a new problem, termed the client assignment problem (CAP) This problem essentially concerns how to assign clients to distributed servers to reduce the client– server communication delays • ci – A client in the DVE • C = {c1, , ck} – The set that consists of all clients in the DVE • zi – A zone in the DVE This is also used to denote a set that consists of all clients in a zone • Z = {z1, , zn} – The set that consists of all zones in the DVE • si – A server in the DVE • S = {s1, , sm} – The set that consists of all servers in the DVE • Rsi – The resource consumption on a server si This can be measured by CPU usage, network bandwidth usage, etc Since the network bandwidth often represents the major operating cost in current server-based online games [18], in this paper, we assume that the server CPU is not a bottleneck, and measure the resource consumption by the network bandwidth usage • Rzi – The total amount of resource, i.e., network bandwidth, used by all the clients that are interacting in zone zi on the server which is hosting zi • C si – The resource capacity of a server si • d ci sj – The round-trip network delay between a client ci and a server sj • D – The delay bound of a DVE The delay bound indicates the required upper bound of the round-trip communication delay between a client and its server to guarantee the interactivity of the DVE For different types of DVEs, there are different delay bound requirements For example, First Person Shooter (FPS) games typically require a delay bound of 250 ms [16], while carracing games have much more stringent latency requirement, at about 100 ms [25] It should be noted that the communication delay between a client and its server is different from the network delay between the client and its server The communication delay is the sum of the network delay and the processing delay at the server However, in this paper we assume that the server CPU is rs 2.3.4 Max-AS/Max-router The Max-AS/Max-router placement approach is similar to the Max-router approach in the sense that they both aim to place servers at nodes with large degree in the network topology The main difference between the two approaches is that Max-AS/Max-router uses a hierarchical placement approach which is based on the structural nature of real network topologies First, Max-AS/Max-router approach selects the ASes with the largest AS-level node degrees In each AS, we place one server at the router with the largest node degree By spreading servers over multiple ASes, this approach may further reduce network latencies for geographically distributed clients compared to the Max-router approach To implement the Max-AS/Max-router approach, we sort all the routers according to the node degrees of their corresponding ASes (each router belongs to an AS) If there are two routers with the same AS-level node degree, we further consider their router-level node degrees We then iterate over the sorted list of all routers to select one best router in each AS to place the server Client assignment problem on Remark 2.2 placement O (M logM), the network m > M D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 3557 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 number of clients with QoS in the DVE would be maximized The client assignment problem is formulated as follows CX ị ẳ Fig Geographically distributed server architecture n m X X i¼1 subject to n X Rzj xij C si ; not a bottleneck, thus the client–server communication delay is determined by the client–server network delay In this paper, the terms ‘‘network delay’’ and ‘‘communication delay’’ are used interchangeably C ij xij 8i I; 2ị 3ị m X al jẳ1 xij ẳ 1; iẳ1 on xij f0; 1g; 8j J ; ð4Þ 8i I; 8j J : ð5Þ Remark 3.1 In Definition 3.1, constraint (3) ensures that the capacity of each server will not be exceeded Constraint (4) means that each zone is assigned to only one server or 's pe rs For interactive applications like DVEs, the client–server communication delay is the most important Quality of Service (QoS) parameter that the system provides to clients [16] In this paper, we say that a client is with QoS or without QoS if the communication delay between the client and its server is smaller or larger than the delay bound, respectively To illustrate the client assignment problem, we consider the example in Fig Assuming that d c1 s1 ¼ 50 ms; d c1 s4 ¼ 80 ms; d c2 s1 ¼ 150 ms; d c2 s4 ¼ 50 ms; d c4 s1 ¼ 200 ms; d c4 s4 ¼ 50 ms and the delay bound of this DVE is 100 ms, then it would be better for the two clients c2 and c4 which are interacting in the zone z1 of the virtual world if z1 is hosted by server s4 instead of s1 For client c1, her experience would not be damaged when zone z1 is hosted by s4 since d c1 s4 < 100 ms From this example, it is obvious that appropriate assignment of clients to servers7 is crucial to enhance the interactivity of DVEs j¼1 co p y Definition 3.1 Client assignment problem (CAP) Let I = {1, , m} and J = {1, , n} be the set of indexes of servers and zones in the system, respectively For each i I and j J, given the cost Cij of assigning zone zj to server si as defined in Eq (1), find an assignment matrix X = (xij), with xij = if zone zj is assigned to server si or xij = otherwise, which minimizes the total cost 3.2 Formulation Au th With the geographically distributed server architecture and the zone-based partitioning approach, the CAP concerns how to assign each zone (with all of its clients) of the virtual world to an appropriate server so that the total number of clients with QoS in the system is maximized To formulate this problem, we first propose the following metric to measure the ‘‘cost’’ of assigning a zone zj to a server si C ij ¼ jfck zj : d ck si > Dgj ð1Þ where j Ỉ j denotes the cardinality of a set Cij measures the number of clients in a zone zj that not satisfy the delay bound D, i.e., without QoS Therefore, by minimizing the total cost of the assignment, the total Recall that the assignment of clients to servers is determined by the assignment of zones to servers Theorem 3.1 The client assignment problem (CAP) as described in Definition 3.1 is NP-hard Proof We consider in our model the special case where the resource capacity C si of each server is the same We further assume that the cost Cij when assigning a zone zj,j J = {1, , n} to server si,i I = {1, , m} has the special form Di À 1, where D>1 is a constant, i.e assigning any zone to server s1 costs C1j = 1, to server s2 costs C2j = D, to server s3 costs C3j = D2, etc Then the optimization goal of CAP in this specific case would be to assign all the zones to the smallest possible number M of servers, with server indexes range from to M This is exactly the well-known Bin Packing problem [17], in which a bin corresponds to a server and an item corresponds to a zone Since the CAP generalizes the Bin Packing problem, which is NP-hard, CAP is also NP-hard h 3.3 Related work To the best of our knowledge, there is no existing work that directly addresses the client assignment problem in DVEs as we have described Research on how to assign clients to servers in DVEs is usually formulated as a load distribution problem in a locally distributed server architecture, i.e., all the servers are placed in the same machine room [31,20] Such approaches may damage the interactivity of the DVE, since clients may be far away (in terms of network delays) from the servers 3558 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 co p al pe rs on 3.4 Assignment algorithms Since the CAP is NP-hard, we seek some heuristic solutions instead of the optimal one We propose three algorithms for the CAP The first one randomly assigns the zones, while the second one is a greedy heuristics to minimize the number of clients without QoS in the system The third algorithm is also a greedy heuristics similar to the second one, except that it uses a different cost metric with the highest value of lij and with sufficient resource capacity This procedure is adapted from the well-known approach used to solve the Generalized Assignment Problem [28] Let us illustrate the effectiveness of the Greedy-1 algorithm using an example Assuming that the DVE has two zones z1 and z2, and two servers s1 and s2 Each server can only take one zone, due to its capacity constraint Let us further assume that assigning z1 to s1 or s2 costs C11 = 10 or C21 = 20, respectively Similarly, assigning z2 to s1 or s2 costs C12 = or C22 = 10, respectively The desirability difference q1 of zone z1 is equal to l11 À l21 = ÀC11 À (ÀC21) = À 10 À (À20) = 10, while q2 of z2 is equal to l12 À l22 = ÀC12 À (ÀC22) = À5 À (À10) = Hence, the Greedy-1 algorithm will select z1 to assign first As a result, z1 is assigned to server s1, and z2 is assigned to server s2, since each server can only take one zone The total cost of this assignment is 10 + 10 = 20, i.e., in total there are 20 clients that are without QoS in the system However, if we not follow the procedure of Greedy1, then z2 may be chosen to assign first In this case, z2 will be assigned to s1, thus z1 has to be assigned to s2 The total cost in this case would be 20 + = 25, which is larger than the cost obtained using Greedy-1 Thus, by sorting the zones according to their desirability differences q, Greedy-1 can effectively reduce the number of clients without QoS in the system We analyze the computational cost of the Greedy-1 algorithm Let m, n and k denote the number of servers, the number of zones and the total number of clients, respectively The Greedy-1 algorithm consists of four parts The first part (from line to line 8) is to find all the values of lij The cost of this part is O (mk) The second part (from line to line 12) is to find qj for all zj Z In the line 10, Quicksort is used Hence, the cost of this part is O (n)[O (m logm) + O (1)], or O (mn logm) The third part (line 13) is to sort the list of n values of qj Using Quicksort, the cost is O (n logn) The final part (from line 14 to line 27) is to assign zones to servers There are m servers and n zones, hence the cost of this part is O (nm) Hence, the overall computational cost of the Greedy-1 algorithm is O (mk) + O (mn logm) + O (n logn) + O (nm), or O (max{O (mk), O (mn logm), O (n logn)}) y Recently, the interest in investigating server and network architectures to reduce the effect of network latency for DVEs has been increased In [22,29], the authors introduced the concept of latency driven distribution (LDD) for the distribution of a DVE over the networking architecture, as opposed to the traditional resource driven distribution (RDD), i.e., load distribution Our client assignment problem shares the idea of the LDD concept However, the authors of [22,29] only investigate the provision of immersive audio communication in DVEs within the context of the LDD concept In their work, two clients that belong to two adjacent DVE zones are able to hear each other However, several important kinds of interactions that may happen in a DVE such as shooting enemies, manipulating objects, changing object ownership, etc., were not studied These interactions may raise complicated consistency issues, if the clients in adjacent zones are managed by different physical server machines or 's 3.4.1 Random assignment (Random) In the random assignment algorithm, zones are assigned to randomly selected servers with the only concern of not to overload the servers In this algorithm, the following procedure is repeated until all zones have been assigned: first we select a random zone zj, and then a random server si with sufficient capacity is selected to take zj, i.e., the target server of all clients in zj is set to si th Remark 3.2 The computational cost of the Random algorithm is O (mn), where n is the number of zones and m is the number of servers Au 3.4.2 Greedy assignment – algorithm (Greedy-1) Since the random assignment algorithm is oblivious to client–server network delays when assigning zones to servers, the obtained performance in terms of the number of clients with QoS may not be good, i.e., the cost of the assignment as defined in Eq (2) may be high Hence, in the Greedy-1 algorithm, we use a greedy heuristics to minimize the total number of clients without QoS in the system The pseudo-code is shown in Fig Let lij = ÀCij be a heuristic measure of the desirability of assigning zone zj to server si Thus, the smaller the cost Cij is, the higher the desirability lij is The algorithm iteratively considers all the unassigned zones and pick a zone zj with the maximum difference qj between the largest desirability lijj and the second largest desirability lsj Then, zj is assigned to a server si Remark 3.3 The complexity of the Greedy-1 algorithm is O (max{O (mk), O (mn logm), O (n logn)}), where n is the number of zones, m is the number of servers and k is the number of clients in the system In practice, usually k ) m and k ) n, hence the overall complexity of Greedy-1 for most cases can be written as O (mk) 3.4.3 Greedy assignment – algorithm (Greedy-2) The only difference between Greedy-1 and Greedy-2 lies in the cost metric they use With the cost metric in Eq (1), Greedy-1 aims to minimize the total number of clients without QoS in the system directly For the Greedy-2 3559 or 's pe rs on al co p y D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 Fig Greedy assignment, algorithm ck 2zj jzj j Au C0ij ¼ th algorithm, we propose to use the following metric to measure the cost when assigning a zone zj to a server si P d ck si ð6Þ C ij measures the average delay from all client ck zj to server si if they are all assigned to si By minimizing this cost function, it is expected that the selected server si for zj will be near (in terms of network delay) to the center of mass of the client population of zi, thus the number of clients with QoS may become large 3.4.4 Reference algorithm: lp_solve For comparison, based on the Integer Programming formulation of the CAP, we use the so-called ‘‘branch-and-bound’’ algorithm implemented in the free Mixed Integer Linear Programming (MILP) solver lp_solve [4] to obtain the optimal solutions for the CAP Note that this approach is only applicable when the system size is small, otherwise the running time of lp_solve will become very long (on the order of several hours), which is clearly impractical for highly interactive applications like DVEs 3.5 Implementation considerations In this section we discuss some practical considerations on the implementation of the proposed assignment algorithms The first consideration is the dynamic property of DVEs During the course of interactions in the virtual world, clients may move from one zone to another, new clients may join, existing clients may also leave the virtual 3560 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 co p y does not require any special infrastructure to be deployed As we have already mentioned, placing servers at good network locations is not always achievable, due to several technical and economic constraints In such scenarios, efficient client assignments are highly desirable to enhance the interactivity of DVEs Hence, to some extent, we may say that client assignments are of greater importance than server placements, although it would be highly advantageous to combine the effectiveness of both methods Performance evaluation on al In this section, we compare various server placement approaches and client assignment algorithms We first study the impacts of server placement and client assignment separately Then, their combined impacts are investigated More specifically, while studying the impacts of server placement approaches, we use the basic client assignment mechanism, i.e., the Random assignment algorithm On the other hand, while studying the impacts of client assignment algorithms, the Random server placement approach is used 5.1 Simulation models and parameters 5.1.1 Network models Both synthetic topologies generated by the popular topology generator BRITE [2] and real Internet topologies are used in the simulations Table lists the topologies used in this paper In our simulations, we used both synthetic topologies generated by the popular topology generator BRITE [2] and real Internet topologies Table lists the topologies used in this paper We use BRITE to generate both flat and hierarchical topologies based on the well-known Waxman and Barabasi–Albert model (F-W, F-BA, H-BA/W) The Waxman model [33] considers all pairs of nodes, and then decides whether to add a link between any two nodes with a probability that depends on the distance between these two nodes and the longest distance between any two nodes in the network The Barabasi–Albert model [8] generates network topologies that exhibit power-laws as observed in the seminal paper by Faloutsos et al [11] in 1999 To complement the synthetic topologies generated by BRITE, we use a real, flat Internet topology collected from NLANR [5] For diversity, we also collect a real AS-level or 's pe rs world An obtained client assignment may not be good after some time Thus, the proposed algorithms need to be executed from time to time to ensure good client assignments We should note that dynamic execution of the client assignment algorithms may incur inter-server bandwidth consumption due to the migrations of zones and clients across servers Nevertheless, we believe this cost is much smaller compared to the bandwidth consumption due to the interactions of users in the DVE This suggests that the bandwidth cost of re-executing the assignment algorithms is not a big concern Indeed, a similar assumption is used in the context of the Web replica placement problem [27] Moreover, in a recent study [19] of the very popular MMOG Lineage II developed by NCsoft, Korea, the authors found that the average value of session durations, i.e., the average time a player keeps on playing the game, is about h, which is rather long This may indicate that we not have to execute the assignment algorithms frequently Another issue is how to obtain the input data for the assignment algorithms The input data includes the client–server round-trip network delays and the resource requirement of each client on a server The network delays can be obtained using some scalable network measurement tools such as King [14] or IDMaps [13] King uses existing recursive DNS queries to accurately estimate the roundtrip network delays between arbitrary Internet end hosts, while IDMaps relies on special end hosts (referred to as the tracers) deployed at some strategic locations in the Internet Both approaches are scalable and incur little estimation overhead In this paper, the server resource requirement of each client is measured as the bandwidth requirement It is well known that the bandwidth requirement in client–server architectures increases quadratically with the total number of clients that are interacting with each other [26] Thus, we can estimate in advance the bandwidth requirement of each client in a zone based on the number of clients in that zone, as in [26] th Server placement versus client assignment Au In Sections and 3, we have discussed several methods for interactivity enhancement for two different problems, namely server placement and client assignment In this section, we briefly discuss their relations First, server placement and client assignment can complement to each other in enhancing the interactivity of DVEs For example, if all the servers are already placed at some good network locations, then appropriate client assignment algorithms can be used to further enhance the interactivity of the DVE On the other hand, a good server placement certainly helps to maximize the performance of client assignment algorithms Second, we believe that client assignment are more flexible than server placement in the sense that the former Table Network topologies Topology Nodes Links Flat, Waxman (F-W) Flat, Barabasi–Albert (F-BA) Flat, real (F-R) Hierarchical, Barabasi–Albert/Waxman (H-BA/W) Hierarchical, real/real (H-R/R) 3000 3000 3024 3000 3300 6000 5997 5192 6197 13442 3561 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 topology (generated by processing the Border Gateway Protocol (BGP) routing tables) with 110 nodes from [6], and use the DFN (German Research Network) topology [15] with 30 nodes as the router-level topology to construct a realistic hierarchical topology (H-R/R) While these topologies are not complete, they at least partially reflect the ‘‘true’’ topology of the Internet, which may have great impacts on the simulation results For simplicity, we assume that the round-trip network delay between any two nodes in the network topology is proportional to the number of link-hops between them This assumption is similar to the one used in most of the previous work [27,24] In fact, a recent Internet measurement [23] also showed that round-trip delay is well-correlated with network hop counts Moreover, to obtain more realistic simulations results, we use both shortestpath routing and AS-level hierarchical routing [32] where possible to calculate the network delays The AS-level hierarchical routing is a more realistic routing strategy for our simulations, since to some extent it may reflect the ‘‘true’’ routing practice in the current Internet y co p 5.2 Impacts of server placement on al Figs and show the cumulative distribution function (CDF) of client–server round-trip communication delays using different server placement approaches and network topologies For flat topologies (Fig 4), we observe that the Max-router placement performs very well: in terms of pQoS, the performance improvements are 157% (F-W topology), 168% (F-BA topology) and 26% (F-R topology) compared to the Random placement, and 157%, 275% and 92% compared to the Min-router placement Moreover, from these figures, it is clear that Max-router not only has a high ratio of clients with QoS, but also provide better interactivity for clients that are without QoS It should be noted that the performance improvement in F-R topology is not as high as that in F-W and F-BA topologies However, this is not because the Max-router placement performs poorly in F-R topology As observed in Fig 4(c), the Random placement also performs quite well in F-R topology One possible explanation for this result is that the F-R topology, which is a real-world topology collected from [5], has good connectivity, i.e any node can reach any other node with a small number of hop counts Indeed, the average round-trip delay between any two nodes in F-R is much smaller than those in F-W and F-BA For hierarchical topologies (Fig 5), we obtain similar results to the case of flat topologies, except that the MaxAS/Max-router placement performs the best in this case: its performance improvements in terms of pQoS over the Random placement are 38% (H-BA/W) and 48% (H-R/R) or 's pe rs 5.1.2 Workload models Based on existing studies of real online game systems, e.g., [12], we simulated different client distributions (clustered, uniform, etc.) both in the physical and the virtual world We obtained similar results for all tested network topologies and client distributions For clarity, we only present here the most representative results Different DVE configurations are used for performance evaluation A specific DVE configuration is determined by the number of servers, the number of zones, the number of clients, and the total resource capacity of the system We use the notation number of servers-number of zones-number of clients-capacity to denote a DVE configuration For example, the notation 20s-400z-5000c-2500cp means that the DVE has 20 servers, 400 zones, 5000 clients and 2500 Mpbs server bandwidth in total 5.1.4 Default settings Unless otherwise stated, the following assumptions and default values are used in the simulations The clients are uniformly distributed in the physical world as well as in the virtual world To estimate bandwidth requirement [26], the input sending frequency of each client (frame rate) is set to 25 messages/s, and the size of each input or update is 100 bytes, which are close to real settings [7] The maximum round-trip delay between any two nodes in the network topology is set to 300 ms The interactivity requirement, i.e., the DVE delay bound D, is set to 150 ms The default DVE configuration is 20s-400z5000c-2500cp b Random Max-router Min-router 175 200 225 250 delay (msec) 275 300 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 150 c CDF 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 150 CDF CDF a Au th 5.1.3 Performance measure The main performance measure used in the analysis is the percentage of clients with QoS in the system, denoted as pQoS Results presented here are obtained by averaging the results of 50 simulation runs Random Max-router Min-router 175 200 225 250 delay (msec) 275 300 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 150 Random Max-router Min-router 175 200 225 250 delay (msec) Fig Impacts of server placement, flat topologies (a) F-W topology (b) F-BA topology (c) F-R topology 275 300 3562 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 b 0.9 0.9 0.8 0.8 Random Max-router Max-AS/Max-router Min-router Max-AS/Min-router Min-AS/Max-router 0.7 0.6 0.5 150 175 200 225 250 Random Max-router Max-AS/Max-router Min-router Max-AS/Min-router Min-AS/Max-router 0.7 0.6 275 0.5 150 300 175 200 225 250 delay (msec) 275 300 co p delay (msec) y CDF CDF a Fig Impacts of server placement, hierarchical topologies (a) H-BA/W topology (b) H-R/R topology a c b 0.9 0.95 0.8 0.9 0.9 0.5 175 200 225 250 delay (msec) 275 Random Greedy-1 Greedy-2 lp_solve 0.75 300 0.7 150 0.8 0.75 175 200 225 250 delay (msec) 275 Random Greedy-1 Greedy-2 lp_solve 0.7 0.65 300 on 0.3 0.85 0.8 Random Greedy-1 Greedy-2 lp_solve 0.4 CDF 0.6 0.85 al CDF CDF 0.7 0.2 150 0.95 0.6 150 175 200 225 250 delay (msec) 275 300 Fig Impacts of client assignment, small configuration (a) F-W topology (b) F-R topology (c) H-BA/W topology 5.3 Impacts of client assignment Au th or 's pe rs This can be explained by the fact that Max-AS/Max-router spreads the servers across multiple ASes, while Max-router may place several servers in the same AS In addition, it is observed that Max-AS/Min-router and Min-AS/Maxrouter both perform worse than Max-AS/Max-router, while Min-AS/Max-router is the worst among the three This observation suggests that the AS-level node degree seems to be the major factor that affects the performance However, we should also note that only selecting the ASes with largest AS-level degrees to place the servers is not enough: the router-level node degree within an AS is also an important factor It is interesting to note that our results here are similar to the observations in [24], although [24] addresses the Web replica placement problem, which is very different from the server placement problem in DVEs Recall that for a typical DVE, which is partitioned into multiple distinct zones, we not know in advance which server a client will connect to Thus, we have to place the servers without knowing clients’ locations, then randomly assign zones to servers and let each client connect to the server hosting its zone.8 In contrast, in the context of Web replica placement problem, a client just simply connects to its closest server Hence, to some extent, the Web replica placement problem may be ‘‘easier’’ to solve than the DVE server placement problem Nonetheless, it is clear that the degree-based placement approach is universal It can be applied into both DVE server placement and Web replica placement with good results Here, we are using the Random assignment algorithm To investigate the proposed client assignment algorithms, we use two configurations, 5s-30z-400c-200cp (small) and 20s-400z-5000c-2500cp (large) The small configuration is only used to enable the comparison between our algorithms and the optimal solution produced by the MILP solver lp_solve [4] In all other cases, we use the large configuration Figs and show the CDF of client–server round-trip communication delays for our algorithms in the small and large configurations In all cases, we observe that the Greedy-1 and Greedy-2 significantly outperforms the delay-oblivious Random algorithm in terms of pQoS We also observed that among the three algorithms (Random, Greedy-1 and Greedy-2), Greedy-1 has the best performance in terms of pQoS This can be explained by the fact that Greedy-1 aims to directly minimize the total number of clients without QoS in the system, while Greedy-2 aims to assign a zone to a server that is close (in terms of network delay) to the center of mass of the client population in that zone In all results, the performance improvements of Greedy-1 over Random are 63% (F-W), 58% (F-BA), 16% (F-R), 35% (H-BA/W) and 25% (H-R/R) for small configuration, and 116% (F-W), 110% (F-BA), 26% (F-R), 48% (H-BA/W) and 38% (H-R/R) for large configuration.9 We note that the performance improvements are better in large configuration, since having more distributed servers means that the greedy algorithms may have more chance to assign clients to good server locations 3563 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 c b 0.9 0.95 0.95 0.8 0.9 0.9 0.6 0.5 0.85 Random Greedy-1 Greedy-2 175 200 225 250 delay (msec) 275 0.7 150 300 0.7 Random Greedy-1 Greedy-2 0.75 175 200 225 250 delay (msec) 275 Random Greedy-1 Greedy-2 0.65 0.6 150 300 175 200 225 250 delay (msec) 275 300 co p 0.3 0.8 0.75 0.8 0.4 0.2 150 0.85 CDF CDF CDF 0.7 y a Fig Impacts of client assignment, large configuration (a) F-W topology (b) F-R topology (c) H-BA/W topology b 0.9 0.85 0.8 0.9 0.85 0.8 0.75 0.7 0.7 Random Greedy-1 Greedy-2 0.65 175 200 225 250 275 Random Greedy-1 Greedy-2 0.65 300 0.6 150 0.8 0.75 al 0.75 0.95 CDF 0.9 0.85 0.6 150 c 0.95 CDF CDF 0.95 175 200 225 250 275 300 0.7 delay (msec) Random Greedy-1 Greedy-2 0.65 0.6 150 on a delay (msec) 175 200 225 250 275 300 delay (msec) rs Fig Impacts of imperfect inputs, H-BA/W (a) Error factor is 1.2 (b) Error factor is (c) Error factor is or 's pe We also note that the pQoS values of Greedy-1 are close to the optimal results given by the branch-and-bound algorithm implemented in the lp_solve software Note that lp_solve can only be applied to small size DVEs, i.e., the small configuration The average execution time of lp_solve for this configuration is around s For the large configuration, the results by lp_solve are not shown since the execution time was too long (not finished after more than 10 h), which is clearly impractical for interactive application like DVEs which need timely assignment decisions In both small and large configurations, all of our proposed algorithms took less than 0.5 s to execute Au th 5.3.1 Imperfect input data The simulation results obtained above for the client assignment algorithms are based on the assumption that we have perfect information about the network delays between clients and servers In practice, we usually have rough estimations of network delays rather than perfectly accurate information To simulate the estimation error, we use the method presented in [27] More specifically, we apply an error factor e to the perfect input data, i.e., assuming the exact value of the delay is d, then the delay value input to our algorithms is uniformly distributed in the range [de , de] Note that although the algorithms use inaccurate inputs to make assignment decisions, their resulted performances, i.e., pQoS, are still calculated based on the actual network delays after the assignment is done As in [27], we use three different values of e: 1.2, and 4, representing the inaccuracies of the popular network delay estimation tools King [14] and IDMaps [13] Fig shows a typical simulation result obtained with the inaccurate estimations of network delays Despite the imperfect knowledge in network delay, the greedy algorithms are still much better than the Random algorithm which does not consider network delays In particular, for e = 1.2, Greedy-1 is still the best algorithm, and its pQoS only decreases slightly compared to the cases that use perfect information However, when the estimation error becomes large, i.e., e = or 4, Greedy-2 is the best algorithm Fig shows that Greedy-2 exhibits low sensitivity to input data estimation errors, even when the error is very large, i.e., e = This suggests that Greedy-2 is a more suitable choice for the client assignment problem under Internet environments, where perfectly accurate input data for assignment algorithms is usually impossible or very costly to obtain 5.4 Combined impacts In this section, we investigate the possibility of combining server placement and client assignment to further enhance the interactivity of DVEs We propose to study two cases The first one is the case in which the DVE requires a tighter delay bound, e.g., 100 ms, while the second case considers the scenario in which we are forced to place servers at some bad network locations, i.e., the routers with lowest node degrees The reason for studying the first case is that we would like to know to which extent a combination of good server placement and good client assignment can enhance the interactivity of DVEs For the second case, our aim is to answer the question whether 3564 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 b c 0.9 CDF 0.6 0.5 0.4 Max-router + Random Random + Greedy-2 Max-router + Greedy-2 0.2 0.1 100 150 200 250 delay (msec) 0.7 0.6 Max-router + Random Random + Greedy-2 Max-router + Greedy-2 0.6 0.5 100 300 0.7 150 200 250 delay (msec) 0.5 0.4 100 300 Max-AS/Max-router + Random Random + Greedy-2 Max-AS/Max-router + Greedy-2 150 200 250 delay (msec) 300 co p 0.3 0.8 0.8 CDF 0.7 CDF 0.9 0.9 0.8 y a Fig Combined impacts, tight delay bound (a) F-W topology (b) F-R topology (c) H-BA/W topology a b c 0.9 0.9 0.4 0.3 0.1 150 175 200 225 250 delay (msec) 275 0.7 0.6 Min-router + Random Min-router + Greedy-2 0.2 0.8 CDF 0.5 0.9 300 0.58 150 175 0.8 0.7 al 0.6 CDF CDF 0.7 0.6 Min-router + Random Min-router + Greedy-2 on 0.8 200 225 250 delay (msec) 275 300 0.5 150 Min-router + Random Min-router + Greedy-2 175 200 225 250 delay (msec) 275 300 rs Fig 10 Combined impact, bad server placement (a) F-W topology (b) F-R topology (c) H-BA/W topology Au th or 's pe it is possible for good client assignment algorithms to improve the interactivity level of DVEs when the servers are placed at bad network locations, due to some technical or economic constraints In the case of tight delay bound (Fig 9) we use a combination of Max-AS/Max-router (or Max-router)10 and Greedy-2.11 In all cases the combined approach Max-AS/ Max-router (Max-router) + Greedy-2 outperforms the Max-AS/Max-router (Max-router) or Greedy-2 alone More specifically, the improvements in terms of pQoS of the combined approach over Max-AS/Max-router (Maxrouter) are 144% (F-W), 130% (F-BA), 29% (F-R), 90% (H-BA/W) and 59% (H-R/R) Similarly, the improvements over Greedy-2 are 144% (F-W), 225% (F-BA), 62% (F-R), 34% (H-BA/W) and 33% (H-R/R) This observation suggests that combining good server placement with good client assignment is a very effective approach to enhance the interactivity level of DVEs In the case of bad server placement (Fig 10), it is also observed that, despite the fact that servers are already placed at network nodes with lowest degrees, the Greedy2 can still improve the interactivity of the system significantly More specifically, the improvements of Min-router + Greedy-2 over Min-router + Random in terms of 10 If hierarchical topologies are used, the best server placement approach is Max-AS/Max-router Otherwise, the best one is Max-router 11 We select Greedy-2 since it gives the best performance under Internet environments, where input data to client assignment algorithms are usually inaccurate pQoS are 150% (F-W), 100% (F-BA), 44% (F-R), 42% (H-BA/W) and 28% (H-R/R) 5.5 Summary of results We have evaluated several degree-based server placement approaches and client assignment algorithms by extensive simulations Our main findings are as follows • First, appropriate degree-based server placement approaches perform very well to enhance the interactivity of DVEs These approaches not require to know the clients’ locations in advance, thus they not suffer from the inaccuracies in network delay estimations, and can be applied straightforwardly to DVEs However, these approaches may not be applicable in every scenarios, due to some inherent technical and economic constraints • Second, client assignment algorithms use the knowledge of network delays to determine appropriate client-toserver assignment This approach also yields excellent performance, and is universally applicable, since it does not require any special server positioning However, the performance of client assignment algorithms seems to inevitably depend on accurate information of the client–server network delays Nevertheless, our experiment results have shown that a greedy assignment algorithm, Greedy-2, exhibits low sensitivity to delay estimation errors, and performs well even when the error is very large This property is very useful in Internet 3565 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 Acknowledgements or 's co p pe rs on al Conclusions Due to the huge resource demands and the real-time requirement of large-scale DVEs, a geographically distributed server architecture is usually needed to support such applications In this architecture, multiple servers are geographically distributed over the network, and the large virtual world is partitioned into distinct zones to distribute load among the servers In this paper, we have proposed a network-centric approach to enhancing the interactivity of large-scale DVEs With this approach, we have considered two key problems, namely server placement and client assignment Both of them are proved to be NP-hard Several heuristic placement approaches and assignment algorithms are proposed to address these problems Finally, extensive simulation studies with realistic models have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs For the future work, it would be very interesting to further investigate the performance of the degree-based server placements and client assignments in the real deployment of a large-scale DVE, such as an MMOG In addition, distributed algorithms for client assignment will also be investigated to extend our current work [8] A.L Barabasi, R Albert, Emergence of scaling in random networks, Science (1999) 509–512 [9] D Bauer, S Rooney, P Scotton, Network infrastructure for massively distributed games, Proc NetGames (2002) [10] E Cronin, S Jamin, C Jin, A.R Kurc, D Raz, Y Shavitt, Constrained mirror placement on the internet, IEEE J Selected Areas Commun (2002) [11] M Faloutsos, P Faloutsos, C Faloutsos, On power-law relationships of the internet topology, ACM SIGCOMM (1999) [12] W.C Feng, W.C Feng, On the geographic distribution of online game servers and players, Proc NetGames (2003) [13] P Francis, S Jamin, C Jin, Y Jin, D Raz, Y Shavitt, L Zhang, IDMaps: a global internet host distance estimation service, IEEE/ ACM Trans Networking (5) (2001) [14] K Gummadi, S Saroiu, S Gribble, King: estimating latency between arbitrary internet end hosts, Proc ACM SIGCOMM IMW (2002) [15] O Heckmann, M Piringer, J Schmitt, R Steinmetz, Generating realistic ISP-level network topologies, IEEE Commun Lett (2003) [16] T Henderson, S Bhatti, Networked games: a QoS-sensitive application for QoS-insensitive users, Proc ACM SIGCOMM (2003) [17] D.S Hochbaum, Approximation Algorithms for np-hard Problems, PWS Publishing Company, Boston, MA, 1997 [18] B Patrovsky, J Mulligan, Developing Online Games: An Insider’s Guide, New Riders Games, 2003 [19] J Kim, J Choi, D Chang, T Kwon, Y Choi, E Yuk, Traffic characteristics of a massively multi-player online role playing game, Proc NetGames (2005) [20] J Lui, M Chan, An efficient partitioning algorithm for distributed virtual environment systems, IEEE Trans Parall Distrib Syst 13 (3) (2002) [21] M Mauve, S Fischer, J Widmer, A generic proxy system for networked computer games, Proc NetGames (2002) [22] C.D Nguyen, F Safaei, P Boustead, Comparison of distributed server architectures in providing immersive audio communication to massively multi-player online games, Proc ATNAC (2004) [23] K Obraczka, F Silva, Network latency metrics for server proximity, in: Proceedings of the IEEE Globecom 2000, 2000 [24] R.G.P Radoslavov, D Estrin, Topology-informed internet replica placement, in: Proceedings of Web Caching and Content Distribution Workshop, 2001 [25] L Pantel, L Wolf, On the impact of delay on real-time multiplayer games, Proc NOSSDAV (2002) 23–29 [26] J Pellegrino, C Dovrolis, Bandwidth requirement and state consistency in three multiplayer game architectures, Proc ACM NetGames (2003) [27] L Qiu, V Padmanabhan, G Voelker, On the placement of web server replicas, Proc IEEE INFOCOM (2001) [28] H Romeijn, D.R Morales, A class of greedy algorithms for the generalized assignment problem, Discreet Appl Math 103 (2000) 209–235 [29] F Safaei, P Boustead, C.D Nguyen, J Brun, M Dowlatshahi, Latency driven distribution: infrastructure needs for participatory entertainment applications, IEEE Communication Magazine on Entertainment Everywhere: System and Networking Issues in Emerging Network-Centric Entertainment Systems 43 (5) (2005) [30] S Singhal, M Zyda, Networked Virtual Environments: Design and Implementation, Addison-Wesley, Reading MA, 1999 [31] D.N.B Ta, S Zhou, A dynamic load sharing algorithm for massively multi-player online games, in: Proceedings of the 11th IEEE International Conference on Networks, 2003 [32] H Tangmunarunkit, R Govindan, S Shenker, D Estrin, The impact of routing policy on internet paths, Proc IEEE INFOCOM (2001) [33] B.M Waxman, Routing of multipoint connections, IEEE J Select Areas Commun (1988) 1617–1622 [34] S Zhou, W Cai, B.S Lee, S.J Turner, Time-space consistency in large-scale distributed virtual environments, ACM Trans Model Comput Simulation 14 (1) (2004) 31–47 y environments, where perfectly accurate input data for assignment algorithms is usually impossible or very costly to obtain • Third, a combination of good server placement with good client assignment is able to significantly improve the interactivity of the DVE, compared to the cases in which only server placement or client assignment is used Moreover, a good client assignment can still improve the interactivity of DVEs even if servers are placed at some bad network locations [1] [2] [3] [4] Au References th The authors thank Dr Gunther Raidl for helping with the NP-hardness of the Generalized Assignment Problem We also thank the reviewers for their constructive comments and valuable suggestions which help to improve the quality and presentation of this paper Akamai Available from: BRITE Available from: Everquest Available from: Mixed Integer Linear Programming solver lp_solve Available from: [5] NLANR Available from: [6] SSF Available from: [7] A Abdelkhalek, A Bilas, A Moshovos, Behavior and Performance of Interactive Multi-player Game Servers, Special Issue of Cluster Computing: the Journal of Networks, Software Tools and Applications, 2002 3566 D N B Ta, S Zhou / Computer Communications 29 (2006) 3553–3566 y Suiping Zhou is currently an Assistant Professor in the School of Computer Engineering at Nanyang Technological University, Singapore Previously, he was a Post-doctoral fellow at Weizmann Institute of Science, Israel He received his B.Eng., M.Eng and Ph.D in Electrical Engineering from Beijing University of Aeronautics and Astronautics, P.R China His current research interests include distributed interactive applications, hybrid control systems and human behavior representation for virtual training systems Au th or 's pe rs on al co p Duong Nguyen Binh Ta is currently a Ph.D student in Computer Science, School of Computer Engineering, Nanyang Technological University, Singapore He received his B.Eng in Computer Science from the Department of Information Technology, University of Technology, Ho Chi Minh city, Vietnam His current research interests include Distributed Virtual Environments, Internetworking, Grid computing and optimization View publication stats ... 2.3.4 Max-AS/Max-router The Max-AS/Max-router placement approach is similar to the Max-router approach in the sense that they both aim to place servers at nodes with large degree in the network topology... approach may further reduce network latencies for geographically distributed clients compared to the Max-router approach To implement the Max-AS/Max-router approach, we sort all the routers according... server architecture and the zone-based partitioning approach, the CAP concerns how to assign each zone (with all of its clients) of the virtual world to an appropriate server so that the total number