Scheduling of virtual screening application on multi-user pilot-agent platform on grid/cloud to optimize the stretch Bui The Quang & Nguyen Hong Quang Emmanuel Medernach Vincent Breton Laboratoire de Physique Corpusculaire 24, avenue des Landais 63171 Aubière, France +33 (0) 73 40 72 72 Laboratoire de Physique Corpusculaire 24, avenue des Landais 63171 Aubière, France +33 (0) 73 40 72 72 IFI, Equipe MSI; IRD, UMI 209 UMMISCO 42 Ta Quang Buu, Vietnam National University, Hanoi, Vietnam medernach@clermont.in2p3.fr + 84 (0) 38696017 breton@clermont.in2p3.fr buithequang@gmail.com ABSTRACT In this paper, we present our research on the scheduling of virtual screening platform on grid/cloud which is shared by many users We find the scheduling policy to ensure the fairness between users We evaluate two policies in existing platform (FIFO and Round Robin) and two candidate policies from literature (SPT and LPT) by our simulator Simulation result showed that SPT improve performance of scheduling policies in existing platform Categories and Subject Descriptors D.4.1 [Process Management]: Scheduling, Simulation, Cloud computing, Grid computing, Large scale biological query, Cloud Computing for bioinformatics, Grid Computing for bioinformatics General Terms Algorithms, Performance, Experimentation Keywords Virtual screening, grid computing, cloud computing, scheduling, fairness, stretch, online-algorithm, SimGrid INTRODUCTION Virtual screening has proven very effective on grid infrastructures where large scale deployments have led to the identification of active inhibitors for biological targets of interest against malaria, SARS or diabetes Operating a dedicated virtual screening platform shared by many users on grid resources requires optimizing the scheduling policy to ensure a certain fairness between users The scheduling of a virtual screening platform can be done at levels; at site level and at platform level Site scheduling is done at each site independently; each site is autonomous in its choice of job scheduling Platform scheduling is done at group level: inside a time slot, jobs from many users are allocated Pilot agents are sent to sites and act as a container of actual users jobs They pick up users jobs from a central queue where the second stage scheduling is done We built a simulator for studying the scheduling of users jobs at platform level and compared scheduling policies in existing virtual screening platforms (FIFO, Round Robin) to two candidate policies (SPT, LPT) from literature [3] Optimal criterion used in our research is the stretch, a measure for the user experience on the platform For job j with size Wj and flow time Fj (Flow time is the time an individual job spends in the platform), the stretch Sj is defined as Fj/Wj The stretch represents the user experience on the platform: the larger the stretch, the lower the satisfaction We use two indicators, sumstretch (Ssum) and max-stretch (Smax), to evaluate the performance of scheduling policies Simulation tests were compared to experimental results collected using a DIRAC platform [2] on the European Grid Initiative Biomed virtual organization SIMULATION OF VIRTUAL SCREENING PLATFORM ON GRID/CLOUD We programmed a simulator with the two-level scheduling of virtual screening platform, using SimGrid, a toolkit that provides core functionalities for the simulation of distributed applications in heterogeneous distributed environments Figure shows the structure of simulator with two components: Grid Environment component and Virtual Screening Platform component a Grid environment component To simulate the grid environment, we built the following modules in our simulator: Grid Workload, Computing Element (CE) and Worker Node (WN) Grid Workload module simulates grid user’s request to CE This module uses Grid Workload Archive (GWA) of a real grid as input data to generate user’s request to CE The CE node receives and manages grid job in queues WN sends job request to CE when it is ready to run jobs b Platform component The platform component has three modules: Agent Manager, Task Manager and Docking User Workload Copyright is held by author/owner(s) BCB ’13, September 22 - 25, 2013, Washington, DC, USA ACM 978-1-4503-2434-2/13/09 Agent Manager module sends automatically pilot agents to CE CE finds available WN and sends pilot jobs to this WN Pilot jobs in WN pull docking tasks from Task Manager to execute http://dx.doi.org/10.1145/2506583.2512369 ACM-BCB 2013 692 When the computing time of a pilot agent reaches WN maximum computing time, it is automatically terminated 3.3 Comparative test of scheduling policies Docking User Workload module sends user task to Task Manager Task Manager receives and manages user task in queues Scheduling policy is applied to calculate priority of user task in queue When Task Manager receives task request from pilot agent, user task with highest priority is sent to pilot agent In our simulation, we have implemented classical scheduling policies for the purpose of comparing them: FIFO, SPT, LPT and Round Robin Virtual screening user workloads were generated using a Poisson random distribution for arrival time of user project and a geometry distribution for the number of docking tasks in each project We generated a dataset with 100 examples of virtual screening user workloads with 100 to 150 users Docking user Workload 3.3.1 Virtual screening user workload 3.3.2 Simulation result Grid Environment component Virtual Screening platform component Docking tasks Task Manager Agent Manager Grid Workload Regular grid job Docking tasks CE Regular Worker Node grid job Worker Node CE Regular Worker Node grid job Worker Node Figure 1: Simulator structure EXPERIMENTATION 3.1 Grid environment configuration We used archive data from AuverGrid, a regional multidisciplinary grid infrastructure in Auvergne (France), as input data to simulate the configuration for the grid infrastructure and the grid user workload 3.2 Validation of simulation Simulation results were compared to experimental data produced on the European Grid Infrastructure using a DIRAC platform [2] A virtual screening project with 1200 docking tasks was submitted from a DIRAC platform installed at the Institut de la Francophonie pour l’Informatique in Hanoï Figure shows the stretch measured experimentally (diamonds) and simulated (squares) as a function of x, the number of docking tasks in a packet Simulator was run using experimentally measured values for the latency between pilot agent and grid Storage Element Figure Sum-stretch in 100 user workload examples The results in figure show the sum-stretch for scheduling policies in 100 examples of virtual screening user workloads SPT scheduling policy has the best results, followed by the Round Robin policy The worst policies are alternatively FIFO and LPT CONCLUSION AND PERSPECTIVE In this research, we successfully built a simulator for such a platform using SimGrid [1] It is used to evaluate scheduling policies used on existing virtual screening platforms (FIFO and Round Robin) and two candidate policies (SPT and LPT) from literature Simulation result showed that in most cases SPT is better than FIFO and Round Robin We implemented a SPT-scheduler on a DIRAC server in HanoiVietnam Experimental results confirm that SPT policy reduces the sum-stretch in this case Moreover, the technique of grouping some docking tasks in packet was compared on experimental and simulated data Because Infrastructure as a Service cloud users buy access to computing resources for a limited time This is similar with limited availability of pilot agent on grid Therefore, we also propose to implement SPT in deployment of virtual screening application on cloud environments REFERENCE [1] Casanova, H., Legrand, A., and Quinson, M 2008 SimGrid: a generic framework for large-scale distributed experiments In Proceeding 10th International Conference Computer Modeling and Simulation (Mar 2008) 126-131 Figure Simulation result and DIRAC experience Both curves display the same behavior: at small x, stretch increases because of the latency between pilot agents and grid Storage Elements At large x, stretch increases because packets grow linearly with x ACM-BCB 2013 [2] van Herwijnen, E and Gandelman, M 2003 DIRACdistributed infrastructure with remote agent control In Conference for Computing in High-Energy and Nuclear Physics (CHEP 03) [3] Legrand, A., Su, A., and Vivien, F 2006 Minimizing the stretch when scheduling flows of biological requests In Proceedings of the eighteenth annual ACM symposium on Parallelism in algorithms and architectures 103-112 693 ... simulate the configuration for the grid infrastructure and the grid user workload 3.2 Validation of simulation Simulation results were compared to experimental data produced on the European Grid. .. Simulator structure EXPERIMENTATION 3.1 Grid environment configuration We used archive data from AuverGrid, a regional multidisciplinary grid infrastructure in Auvergne (France), as input data to. .. examples of virtual screening user workloads with 100 to 150 users Docking user Workload 3.3.1 Virtual screening user workload 3.3.2 Simulation result Grid Environment component Virtual Screening platform