Tài liệu Grid Computing P29 pptx

29 Unicore and the Open Grid Services Architecture David Snelling Fujitsu Laboratories of Europe, Hayes, Middlesex, United Kingdom 29.1 INTRODUCTION This chapter describes a GridService demonstrator built around the Unicore Grid envi- ronment, its architectural design and implementation [1]. It then examines some lessons learned from the process of developing an implementation of a family of GridServices that conforms to the Open Grid Services Architecture (OGSA) [2] and the Grid Service Specification [3]. The goals of this project were two fold. Primarily, it is only through implementation that complexities such as those that arise in OGSA can be fully understood and analyzed. Secondly, Unicore (www.unicore.org) is a complete production Grid environment that seemed to conform extremely well to the OGSA. The author wanted to test this hypothesis and see how difficult it would be to construct an OGSA-hosting environment on top of the Unicore infrastructure. By way of a brief summary, both goals were met easily and with satisfaction. The OGSA model is remarkably clean and the abstractions well matched to what is needed at the implementation level. The lessons learned reveal where cracks exist and, in many Grid Computing – Making the Global Infrastructure a Reality. Edited by F. Berman, A. Hey and G. Fox  2003 John Wiley & Sons, Ltd ISBN: 0-470-85319-0 702 DAVID SNELLING cases, support the changes currently under discussion in the Open Grid Services Infras- tructure (OGSI) working group of the Global Grid Forum (GGF). The reader should note that this chapter and the implementation on which it is based are targeted at the first draft (2/15/2002) of the Grid service specification [3]. By the time of going to the press, many of the issues discussed here and elsewhere will have become part of the next version. With respect to the second goal, the implementation of a complete OGSA GridService environment that could be used to deploy task specific GridServices was significantly easier than expected initially. The combination of the Unicore Grid infrastructure with a flexible Web Services tooling environment (GLUE from www.themindelectric.com) proved to be effective and easy to use. 29.1.1 Caveats The implementation discussed here conforms to the OGSA model, but not to the speci- fications as such. The specification is still under development and requires extensions to the Web Services Definition Language (WSDL) [4]. It will be some time before tooling platforms are available that support these extensions. Therefore, our approach has been to develop the architecture described by OGSA, but to work within the constraints of existing Web Services. The OGSA abstractions (serviceTypes in particular) are exposed in Java implementation, and then merged into a ‘flat’ WSDL representation, as the var- ious portTypes are aggregated into a single Web service. This restriction has very little impact on our ability to meet the goals of this project. In fact, the extent to which this project was possible within the Web Services framework raises questions about the need for extensions, see Section 29.3.1 below. 29.2 IMPLEMENTATION This section outlines the overall architecture and implementation of this GridService demonstrator. On top of a ‘hosting environment’, the demonstrator supports a generalized, distributed, application-steering GridService. This section also includes a summary of this application-steering GridService. 29.2.1 Infrastructure and architecture In ‘The Physiology of the Grid’ [2], Foster et al., describe a hosting environment as follows: In practice, Grid services are instantiated within a specific execution environment or hosting environment. A particular hosting environment defines not only implementa- tion programming model, programming language, development tools, and debugging tools, but also how an implementation of a Grid service meets its obligation with respect to Grid service semantics. The platform component of this demonstrator fits easily into this definition; however, the overall environment is more complex than this definition indicates, see Figure 29.1. UNICORE AND THE OPEN GRID SERVICES ARCHITECTURE 703 Hosting environment GridService platform Client GLUE server Unicore servlet Unicore server B Computing resource B Unicore server A Computing resource A http(s)+SOAP UPL SSL Figure 29.1 Architecture of Unicore GridService hosting environment. Implicit in the above definition, a hosting environment is based on a single programming model/language; it implies that the GridService client interacts with a single ‘program’. Although true of this demonstrator’s platform component (i.e. the client interacts with a Java program constructed as described below), this simplification belies the complexity of the Unicore infrastructure that interacts with the platform and the distributed computing resources. The Unicore hosting environment supports applications written in a variety of programming languages and run on many different computer architectures. Nonetheless, the notion of Unicore deployed in this way is well described as a hosting environment. This hosting environment is based on two primary building blocks, the Web Services platform GLUE from The Mind Electric, (www.themindelectric.com) and the Unicore server implementation developed by Fujitsu Laboratories of Europe for Unicore-related projects (www.unicore.org). Both environments are freely available for downloading along with the source code of this demonstrator. 1 All components are written in Java. Only a small part of the Unicore environment requires Perl. 29.2.1.1 Client The user accesses several GridServices from a single client application, linked with the client side classes from GLUE. The client included in the demonstration download is a crude, command line client supporting all the GridServices provided, for example, Factory, Registry, HandleMap, a NotificationService, and various JobGridServices. More realistic and user friendly clients could be written using any Web Service environment, for example, .NET. The client connects to the hosting environment’s platform component, via http or https, with Simple Object Access Protocol (SOAP) bindings, as specified in the WSDL 1 Note the demonstrator download does not include the Unicore or GLUE infrastructure components. To experiment with this demonstrator, the GLUE Web Services environment is required and must be obtained from The Mind Electric. A ‘DummyGrid- Service’ is included in the demonstrator that does not require a complete Unicore Grid. This demonstrator will interact with the demonstration Unicore Grid at (www.fz-juelich.de/unicore-test), which supports some of the demonstration applications. 704 DAVID SNELLING sent from the platform. For https, a pair of certificates and the associated key stores are required for the client and server. The use of an https binding automatically activates a password-based authorization step. In the http mode, no user authentication is needed. With the simple http binding, the ‘console’ supported by GLUE can be started (see GLUE documentation for details). The console introspects the WSDL and generates dynamic Web pages for most of the GridService operations. The exceptions are those that return structured data types, which the simple console interface cannot introspect; however, no fundamental barriers exist to automatic tooling within this GridService environment. 29.2.1.2 Platform On the server side, the GLUE Web Services server is integrated with a servlet pack- age that supports the Unicore protocols. From the servlet, Unicore Abstract Job Objects (AJOs) can be consigned to Unicore sites, monitored, controlled, and the results retrieved. Within a Unicore Grid, this servlet can be used to construct lightweight clients. In this implementation, the GridService package implements the basic OGSA portTypes and a collection of JobGridServices. The servlet package manages the interaction between the GridService package and the Unicore Grid. This servlet also manages Unicore security, see section 29.2.1.4. 29.2.1.3 Unicore server From the servlet, Unicore sites are contacted via the Unicore Protocols Layer (UPL). The Unicore Server receives AJOs from Unicore clients, the GridService platform, and other Unicore Servers. As the various workflow tasks of the AJO are processed, they are converted from their abstract representation in the AJO to platform-specific forms by Unicore mechanisms. All aspects of the Unicore Grid, including the AJO, are of course hidden from the OGSA user. Further details of the Unicore side are not central to this chapter. See Unicore download documentation and Reference [5] for more information. 29.2.1.4 Security framework At the time of writing, the Grid Service Specification [3] only indicated that security was a binding time property. As a result, there are two separate levels to the security framework, one at the https binding between the client and the GridService platform and another between the platform and the rest of the Unicore hosting environment. Https is used to connect the client to the platform component of the GridService. The use of https triggers the need for simple authentication using user names and passwords. 2 Once the https connection is established, a Unicore certificate, loaded by the servlet when it starts up, is used to authenticate and authorize work by all users of the GridService. In 2 The user name/password step is redundant, given that the authentication can be taken from the https connection. However, we chose to leave it in place, since the normal practice for an https connection assumes the use of a generic client certificate (e.g. as supplied with most browsers) and a separate login to sites requiring authentication. UNICORE AND THE OPEN GRID SERVICES ARCHITECTURE 705 this case, the GridService is acting like an Application Service Provider (ASP). The user authenticates the ASP, using user name and password over https, and the ASP authenti- cates the supercomputer centres, using Unicore authentication over UPL. In this demon- strator, the computer centres do not know the identity of the end user, only that of the ASP. Other security and trust models are clearly possible as detailed below: 1. Globus style proxy credentials could be generated by the Web Services client and sent to the platform. This proxy certificate could then be used by Unicore to identify the user to the computer centres. 2. The servlet can be run on the client side. In this case, Unicore authentication and authorization are used throughout the process. The Web Services authentication can be omitted since this connection is local. This approach has little to offer byway of practical GridServices beyond providing a GridService client to existing Unicore users. 3. The Unicore sites could accept authentication according to the ASP model as described above, but the Unicore sites can be configured to authorize on the basis of the login used at the Web Services point of entry. 4. As OGSA security plans evolve, other approaches will be considered. 29.2.2 Supported Grid Services The basic set of portTypes, as outlined in Reference [3], has been implemented. By aggre- gating and extending these, a collection of practical GridServices has been implemented. 29.2.2.1 Basic GridService portTypes All the basic portTypes are coded as Java interfaces. The WSDL representation of each is generated automatically by the GLUE environment. However, GLUE has not been extended to support the serviceType extension; hence, the aggregation of portTypes into services takes place in the Java implementation and the complete Web service description for each GridService is created automatically from the Java implementation. Given that the Grid Service Specification indicates that all GridServices should support the GridService portType, all other portTypes are constructed using interface extensions from the GridService interface. The basic GridService interface hierarchy is therefore as follows: interface com.fujitsu.arcon.gridservice.IGridService interface com.fujitsu.arcon.gridservice.IGridServiceFactory interface com.fujitsu.arcon.gridservice.IHandleMap interface com.fujitsu.arcon.gridservice.IJobGridService interface com.fujitsu.arcon.gridservice.ISteeringGridService interface com.fujitsu.arcon.gridservice.INotificationService interface com.fujitsu.arcon.gridservice.INotificationSink interface com.fujitsu.arcon.gridservice.INotificationSource interface com.fujitsu.arcon.gridservice.IRegistry 706 DAVID SNELLING 29.2.2.2 Grid Service implementations The implementation hierarchy parallels the interface hierarchy; there is an implementation of the GridService portType that all other portTypes extend. Although a GridService that implemented only the GridService portType could be created and used, it performs no useful function and therefore, no factory has been written for it. The following GridSer- vices can be created (or are created automatically) and accessed from the demonstrator’s client application. RegistryAndMapper: The HandleMap and Registry portTypes are implemented as a sin- gle GridService. This GridService is created automatically when the service starts up as a function of the hosting environment. GridServices created by the hosting environment are registered automatically with both the HandleMap and Registry when they are cre- ated. Also, there is a Registry portType for registering new, external GridServices with the Registry. GridServiceFactory: The factory is started automatically, but the client must enquire of the Registry to find the Grid Service Handle (GSH) of the factory and then contact the HandleMap to obtain a Grid Service Reference (GSR) with which to bind to the factory. This process could be streamlined; however, the goal was to expose the full functioning structure of the OGSA. The factory can construct a variety of JobGridServices and a NotificationService, as outlined below. NotificationService: Given a Grid Service Handle, this service will wait for any notified state change in the target GridService. The NotificationService implements the Notifica- tionSink portType and requires that the target GridService implements the Notification- Source portType. DummyJobGridService: This service mimics the functions of the JobGridService, but does not require access to a Unicore enabled site, nor does it support the Notification- Source portType. JobGridService: This GridService provides support for running a ‘black box’ application at a Unicore site. The serviceData describing these applications is instance specific, but made available as part of the factory’s serviceData. Each Unicore site may support many different applications, and the factory may query the Unicore sites for this serviceData. Note that, in this implementation, the factory’s version of this serviceData element is hard coded, so that the DummyJobGridService can function without reference to a Unicore site. The dynamic information from the Unicore site is available in the GridServiceDescription of the JobGridService after it has been created. Each ‘black box’ application accepts an input file and writes an output file. The names of these files are part of the GridServiceDescription, and the users may specify file names of their choosing. The GridService manages the renaming automatically. The application UNICORE AND THE OPEN GRID SERVICES ARCHITECTURE 707 may also access other resources at the site, but these are configured by the site and invisible to the user (effectively inside the ‘black box’). The input file may come from any Unicore site or the client and the output file may be sent to any Unicore site or returned to the client. SteeringGridService: This GridService directly extends the JobGridService with opera- tions that allow the user to fetch a sample file back to the client and write a control file to be read by the running application. The assumption is that the application occasionally inspects the control file and modifies its behaviour accordingly. The sample file may be written explicitly as a result of fields in the control file or according to some other policy, for example, every ten iterations. The client, included with the download of this demon- strator, assumes that the sample file is in GIF format and starts a viewer to display the contents. The same flexibility with respect to file names exists for the sample and control files as it does for the input and output files. 29.3 LESSONS This section briefly outlines the author’s thoughts on some issues that arose out of the process of implementing this OGSA interface to Unicore. Although arrived at indepen- dently, many of these ideas are not unique. Following the release of the first draft of the Grid Service Specification, a significant and active community of informal reviewers was formed. Many of the ideas and perspectives listed here can also be seen in the proceedings of this community, see Reference [6]. 29.3.1 Are GridService extensions to WSDL really needed? The author certainly feels that some of the proposed extensions are desirable, but from both the server and client-development points of view, many of the proposed extensions seem unnecessary. This project was carried out with only standard WSDL facilities on both the client and server sides and has resulted in a working demonstrator that could be deployed in a variety of contexts. However, several aspects of the OGSA model are missing. 29.3.1.1 ServiceType The serviceType extension provides a mechanism for aggregating and naming a collection of portTypes. Multiple serviceTypes can be combined to create new serviceTypes, and multiple implementations of a given serviceType can be constructed. In this implemen- tation, Java interfaces encapsulate the abstractions of the OGSA portTypes and therefore capture the serviceType structure, albeit in Java rather than (extended) WSDL. For each serviceType, the WSDL emitted by the platform is a single portType containing all the operations of the constituent portTypes (represented by Java interfaces) used to form the 708 DAVID SNELLING serviceType. The serviceType structure, retained at the Java level, is effectively flattened into a single portType at the WSDL level. For example, the RegistryAndHandleMap ser- viceType would contain three portTypes, that is, GridService, HandleMap, and Registry. The single WSDL portType constructed by the platform is semantically equivalent to what would be expressed in the extended serviceType representation, but the underlying structure is lost. The author’s belief is that a serviceType extension that supports serviceType inheritance should be a minimum extension to WSDL. Such an inheritance mechanism should support aggregation, versioning, extension (in the sense of adding additional operations through new portTypes), and implementation naming, see Section 29.3.1.2. 29.3.1.2 ServiceImplementation The related extension, serviceImplementation, provides a mechanism for naming a par- ticular implementation of a serviceType as part of the creation of a service. However, serviceImplementation extends the WSDL element service, which includes endpoint and binding information. The author favours separating the naming aspect from binding and endpoint aspects. Allowing serviceTypes to be extended, as recommended earlier, pro- vides an effective naming mechanism for particular implementations, while the standard WSDL service element provides the binding and endpoint details. The author believes that serviceImplementation can be dropped from the specification. 29.3.1.3 ServiceData Because serviceData can be obtained dynamically, through the findServiceData opera- tion in the required GridService portType, there is no requirement to expose service- Data as an extension to the WSDL representation of a serviceType. The typical pat- tern adopted in this implementation is to include some (usually static) serviceData ele- ments for one GridService as run-time serviceData elements in the Grid Service used to find (or create) the given GridService. In this way, serviceData that is needed to cre- ate a Grid Service, for example, is available from the factory as part of its run-time serviceData. Although not required, there are good reasons for making some serviceData available through the WSDL. In particular, it allows serviceData to remain associated with the serviceType to which it refers. Clearly not all serviceData elements can be defined when the WSDL is generated, however, wherever possible there are clear advantages of having it available. For example, if the list of applications supported by a given JobGridService were available as part of the WSDL, the client could decide if this service supported the desired application before creating the service. It is possible to obtain this information from the serviceData of the factory, as in our demonstrator, but this separates this important information from the GridService. Furthermore, serviceData is an ideal framework to capture increasing detail about a GridService as it evolves from interface, to implementation, to created instance, to running instance, to completed instance, and finally to destroyed GridService. In some cases UNICORE AND THE OPEN GRID SERVICES ARCHITECTURE 709 the content changes as the state of the service changes; in other cases new serviceData elements are added. At the transition from implementation to instance, serviceData allows site-specific details to be provided by the Unicore hosting environment. In particular, the list of supported applications at a given site is included in the factory’s serviceData. Also, details about the file names required by the application are provided to the client as the GridService is created. However, there are several places in the architecture where care must be taken to keep serviceData consistent with other structures in the model, in particular, the GSR. There are three different sources for a GSR. The client may have a copy of the GSR that was obtained from the factory when the GridService was created, one may be obtained from a HandleMap, or the client may query the serviceData of the GridService itself. In most cases the consistency of the GSR is not important, but with serviceData being generally available from several sources, consistency can become an issue. In general, serviceData is an essential part of the OGSA framework. The specification should make it clear what serviceData elements are required and when in the GridService life cycle they are relevant. Allowing serviceData as part of the WSDL of a serviceType is certainly desirable, but all serviceData should remain available dynamically from some source, be it the GridService itself or a registry or a factory. 29.3.1.4 Compatibility assertions There are many issues surrounding compatibility assertions. Since this demonstrator is the first GridService of any type (that the author had access to), it could not have been compatible with another implementation. During the analysis and design phases for this effort, many questions arose. These were also echoed in the OGSI mailing list and are detailed as follows: Who asserts the assertions? How do we believe the assertion? Does an assertion mean more semantically that ‘This one is like that one.’? Is compatibility asserted at the operation, port, service, or implementation level? Are compatibility assertions object-oriented programming by the back door? How are compatibility assertions composed? What is the meaning of composition of compatibility assertions? Recent discussions in the OGSI Working Group indicate that compatibility assertions will be removed from the specification and discussion postponed to some later time. The author concurs with this strategy. In the meantime, it is possible to experiment with compatibility issues in the context of GridServices. As part of GridService development, inclusion of a compatibility portType would provide information on the compatibility of one version to all earlier versions. There is no complex trust framework required, as the service provider is making assertions about his own implementation only. Similarly, as part of a registry or other discovery services, a compatibility portType could check that two distinct implementations were compatible. 710 DAVID SNELLING Such a service could be supported and run by a VO, or a corporate consortium, covering their own members and implementations. 29.3.1.5 PrimaryKey In this implementation no use could be found for the primaryKey portType. Initially, during development, it seemed to make sense as an internal handle, but it never made sense as an externally visible concept. As development proceeded, its internal role also vanished. The primaryKey portType should be removed from the specification. 29.3.2 Can a single GridService support multiple portTypes? One aspect of GridServices in general, and of this implementation in particular, is that multiple portTypes can be combined into a single serviceType. In this demonstrator, the Registry and the HandleMap are combined into a single serviceType. However, a single service possessing two distinct functions creates some complexities in the development of a clean client demonstrator, as well as raising questions of identity and equality within the GridService context. The bootstrap process for the client is based on knowing the address of an ‘interesting’ Registry. The Registry advertises a running factory for creating application steering Grid- Services. Following the procedure set out in the Grid Service Specification, the Registry’s or GridService’s GSH yields the handle for the home HandleMap. These two GSHs (the Registry and the HandleMap) and their mapped GSRs are different, but point to the same instance and therefore have the same serviceData. One of the serviceData elements is the GSH of the GridService. The GSH must appear exactly once in the serviceData and, in this case, can have two possible values. In the demonstrator, the serviceData contains the GSH of the Registry. In general, the issue of identity needs further discussion and some resolution needs to be arrived at in the context of GridServices. 29.3.3 Is the ‘Push Only’ notification framework sufficient? The OGSA notification framework is based around a subscribe/deliver model. However, the underlying assumptions in Unicore are based on periodic polling for changes in state. This was largely motivated by the need to deal with clients sitting behind firewalls, the norm with the majority of Unicore-related partners. 3 For this demonstrator, a NotificationService provides an operation that waits for a state change in a selected GridService and then returns that state to the client, that is, a ‘pull’-based model. To implement ‘push’-based notification back to the client, at least partial server-side functionality would need to be provided to the client, creating the same firewall issues that motivated the ‘pull’-based model in Unicore. This NotificationService (a NotificationSink) subscribes to state change notification with the nominated JobGridService (a NotificationSource), using the OGSA ‘push’-based 3 Incidentally, this is the only significant point where the architecture of OGSA and Unicore differ; otherwise a very direct mapping of concepts is possible. [...]... (2002) The Physiology of the Grid , Draft 2/3/2002, www.globus.org/research/papers/ogsa.pdf 3 Tuecke, S., Czaijkowski, K., Foster, I., Kesselman, C., Frey, J and Graham, A (2002) Grid Service Specification, Draft of 2/15/ 2002, www.globus.org/research/papers/gsspec.pdf 4 The Web Services Description Language, www.w3.org/TR/wsdl 5 Erwin, D and Snelling, D (2001) Unicore: A Grid Computing Environment LNCS... Snelling, D (2001) Unicore: A Grid Computing Environment LNCS 2150, EuroPar 2001, Springer 6 Global Grid Forum, OGSI Working Group, www.gridforum.org/ogsi-wg/ 7 Leymann, F., The Web Services Flow Language, www-4.ibm.com/software/solutions/webservices/pdf/WSFL.pdf 8 Grid Interoperability Project, IST-2001-32257, www .grid- interoperability.org ...UNICORE AND THE OPEN GRID SERVICES ARCHITECTURE 711 model Web Service-based bindings are used, even though both services are running in the same Java Virtual Machine (JVM) to demonstrate the principle Note that within the JobGridService implementation, a ‘pull’ model exists to obtain up-to-date information from the remote... Unicore and Globus provide this functionality and thus it will be expected of GridServices There are several approaches for achieving this, two of which are apparent in the differences between Unicore and Globus Workflow : Unicore includes a complete framework for workflow management in a batch job context The extent to which GridServices can be controlled with similar mechanisms is a current area of investigation... effort for this project so far, including the learning curve for Web Services, was not more than 2 person months (3 months elapsed) It is this accessibility of Web Services that we hope to see extended to GridServices The key requirement for this will be standardization, and for this reason the author stresses the importance of limiting the number and complexity of extensions to WSDL REFERENCES 1 Unicore... sites This is consistent with a Web Services model A Web service that needs to use subsequent services authenticates under its own credentials as a service provider to the subsidiary service Eventually, GridServices will need more flexibility than this Some of the approaches outlined in Section 29.2.1.4 above would allow extended flexibility There are many issues pertaining to security and trust frameworks... remote Unicore sites It seems likely that lightweight clients, running behind firewalls, will be needed in many instances Therefore, some thought should be given to establishing a framework within the Grid service specification that supports a ‘pull’-based model with the same scope as the ‘push’-based model currently described 29.3.4 Is security provision at binding level only adequate? In the case of . interface com.fujitsu.arcon.gridservice.IGridService interface com.fujitsu.arcon.gridservice.IGridServiceFactory interface com.fujitsu.arcon.gridservice.IHandleMap. com.fujitsu.arcon.gridservice.IJobGridService interface com.fujitsu.arcon.gridservice.ISteeringGridService interface com.fujitsu.arcon.gridservice.INotificationService