Building Interoperable Portals with Web Services

Final Project Report Contract No N62306-01-D-7110 Task Order No 03-011 Project ID: ET-03-011 Project Name Building Interoperable Portals with Web Services Principal Investigator: Mary P Thomas Reporting Period: October 1, 2002 - September 30, 2003 Building Interoperable Portals with Web Services Mary P Thomas2, Marlon Pierce2, Tomasz Haupt3 Texas Advanced Computing Center The University of Texas at Austin 10100 Burnet Road Austin, Texas 78758 mthomas@tacc.utexas.edu Community Grids Lab Indiana University 501 N Morton Street, Suite 224 Bloomington, IN 47404 marpierc@indiana.edu Keywords: portals, portlets, Jetspeed, Abstract We constructed an HPCMO Computing Portal, which acts as a single, centralized, secure gateway to all DoD resources, with a user interface based on the familiar HotPage The Jetspeed based portal is implemented using emerging technologies that team members are developing as part of currently deployed DoD technologies from existing portal projects Portal capabilities include portlets for user presentation, informational services (job, queue), a secure system based on Kerberos and/or Globus?), and DoD portal services such as those already deployed on the Gateway and MISS portals ARL, ASC, OKC, ERDC and NAVO Introduction The four MSRC environments, share a common security infrastructure supporting single log-in to distributed, heterogeneous resources, and are a prime example of a distributed computing environment Distributed high performance computing environments, and particularly Grid environments, are inherently complex for end users (employing multiple queuing systems, different file systems, job management, network latencies) Web portals and problem solving environments represent solutions to many of these problems, and the HPCMO has funded several such projects However, there is no current plan for interoperability between these systems, the sites where they are operating, or solutions that integrate these various portals into a common user interface Commercial and academic developers are converging on the use of “Web Services” for the interoperability that is required for integration This is based around the Internet standards XML, SOAP (Simple Object Access Protocol), WSDL (Web Services Definition Language) and the UDDI (Uniform Description, Discovery and Integration) service repository Preliminary investigations indicate that these standards are an appropriate way to build interoperable grid portals, and Thomas, Pierce, Fox and Haupt are all actively working on integrating web services technologies into their portals projects For example, all team members are participating in the GGF/GCE Interoperable Web Services Testbed, and are using experiences gained and technologies shared to further advance their projects We therefore identify the following requirements First, PET-sponsored resources must be able to work together operationally within MSRC production environments through inter-system protocols, such as web services, to avoid duplication of effort and to share services Second, the use of web servicebased portals should be extended to include support for early users of the HPCMO portals Third, browser interfaces to various services need to be integrated into a common portal environment Finally the experiences and best practices (from both industry and academia) of portal development should be integrated into production DoD environments Technical Approach A recent document, published by the GCE-RG (and on which Thomas and Pierce are coauthors), titled “Overview of Grid Computing Environments,” describes portal technologies and best practices in terms of architectural principles – multi-tier service-based model, role of metadata, workflow, tools and core functionalities that lead to aggregation portals The HPCMO portal architecture is based on the generalized principles laid out in this document Additionally, the portal project leverages efforts on currently funded activities for technology sources by using the emerging technologies that team members are developing as part of currently deployed DoD technologies from existing portal projects This list includes utilization of built-in Jetspeed portal capabilities (login, customization, etc.) as well as portlets for user presentation, a secure system based on Kerberos and/or Globus, and DoD portal services such as those already deployed on the Gateway and MISS portals ARL, ASC, OKC, ERDC and NAVO 3 Portal Technologies We have adopted the portlet/container approach originally advocated by Geoffrey Fox and Pierce for the OKC This approach divides the portal into a host container and content components We base our project around the Jetspeed project from Jakarta, which serves as our portlet reference implementation The basic system architecture is illustrated in Figure The portlet/container approach has several advantages: The container portal implements standard services like authentication, user management, account creation, access control, and customized user views (among several others) Content can be plugged into the portal in well defined ways, and can additionally be accessed controlled, so that only certain users get to see certain portlets Portal content can be indefinitely expanded and managed within a single, reusable container Note that the term “portlet” actually refers to the Java code that creates/manages content from a particular source; it does not refer to the content itself For simplicity, we sometimes refer to a particular piece of portal content as being a “portlet”, although this is not precise The first point is important to the HPCMP because there is a growing desire to provide Kerberos-secured Web content (the Information Environment portal and ASC’s intranet are two examples.) By following the approach of ET011, these services not need to be reinvented and can be easily deployed in new applications As illustrated in the screen shots below, we also (trivially) added MSRC web pages (Figure 2) and XML and Weather portlets (Figure 3) Thus the ET011 project has produced a computing portal that can also be used to manage general HPCMP and third party web content The display, arrangement, and even existence of these pages are at the discretion of the portal user and administrator The Jetspeed container provides a number of other useful features for account creation and management For example, we can (by editing property files) disable automatic account creation and allow only portal administrators to enable new accounts All account communications are handled via email This is one simple example of a standard service needed by current and future DOD portals Using the ET011 approach, these services never have to be reinvented Figure Jetspeed provides a component architecture for computing portals Jetspeed actually provides abstract interfaces as well as service implementations, so we are free to replace, for example the authentication mechanism We did this in order to support Kerberos+SecurID logins Pierce has developed an administrators’ guide for customizing Jetspeed access restrictions that is available on request Figure Portlets for HPCMP web content can be easily added with customized display arrangements Jetspeed allows content to be added in two basic ways: Local content can be created using either JavaServer Pages or Velocity These pages are placed in Jetspeed template directories The GPIR portlets (described below) are deployed this way Remote content can be pulled into the portal by using IFramePortlets These very simple portlets just use HTML IFrames to load remote URLs In addition to the above two approaches, we have developed two additional portlet types The DODWebFormPortlet is capable of managing session state, SSL secure connections, HTML Forms, and shared Java objects (when loaded from a Jetspeed subdirectory on the same server) We use this portlet type to simplify deployment of “legacy” web pages from the Gateway project We extended this portlet type to also allow file uploading and downloading for file transfer pages These portlets are used to verify authentication information Figure Third party technical and informational content can also be provided Authenticated users are mapped into an XML-based metadata description that contains the user’s real name, home directory at a particular MSRC, email address, and MSRC account These are currently edited by hand for each new user This information is needed by the File Management and File Portal Services 4.1 Informational Services – the GridPortal Information Repository (GPIR) 4.2 Security – Kerberos Integration For the Gateway project, Pierce developed security components that can be used to perform a “Web kinit” through an HTML form When combined with JSP/servlet session maintenance, this allows the user to maintain a secure session state that can be shared among several pages The Web kinit creates a ticket for each user on the Web server that can also be used to invoke a remote process (access to the file system on an HPC system, invocation of a remote process such as a remote queuing system launch, etc.) A successful login procedure creates an instance of our AuthenticationBean object that is shared with pages that need to interact with remote resources The login page is shown in Figure User enters principal name, password, and SecurID here We also have “Login” portlets that can be used as the first page for loading a sequence of linked pages within the same portlet This approach is appropriate for Jetspeed portlets that don’t want to use Kerberos in the portal login itself but want to provide a Kerberized portlet The installed portal also uses SSL for secure connections between the user’s browser and the Web server Remote resources are accessed with either Kerberized rsh/rcp or ssh/scp 4.3 File Management We have developed portlets that allow users to interact with their files on HPC resources through the portal Users can pick available hosts, navigate directories, view and download files, upload files, and create directories We tested these for ASC machines HPC04, HPC05, and HPC08 and the ARL machine Zornig The code for these pages can be easily modified to interact with other MSRC machines Select desired remote host with this menu The shown file interface is for HPC04 at ASC Files on Zornig (at ARL) can be viewed and managed through the same interface Other MSRC hosts can be easily added 4.4 Job Monitors We ported original code developed by the Gateway web portal for interacting with the GRD queuing system to work with PBS, LoadLeveler, and LSF We tested these with the hosts Zornig, Herman, Adele, HPC04, HPC05, and HPC08 Note Adele, Herman, and HPC05 have been decommissioned The code for these particular host monitors can be easily modified to support additional hosts matching the above queuing systems As shown in the figure, a user can select from ASC, ARL (and other) HPC machines 4.x Application Descriptor Services The original Gateway project developed a number of job submission services for working with ARL resources These include simple interfaces for ANSYS, ABAQUS, and ZNS Support for Fluent (2D and 3D versions) was added during the project We also ported ANSYS to work with ARL, although the age of our test databases prohibited full testing The underlying services need by these job submission services (file transfer, remote process execution, script generation, session archiving and recovery) work successfully and can be applied by developers to new submission pages 10 Deployment Notes In developing this project, we have attempted to simplify deployment as well as develop/combine services The portal is code is installed on gridportal.asc.hpc.mil under the jakarta-tomcat-4.1.24/webapps directory of the Gateway account All source code for applications is placed under the ~gateway/install_base directory and includes the following subdirectories • jetspeed-mods: these contain the modified actions (need for login), some additional property settings (needed for the Harvesters), and some modified page templates 11 • GPIR_Source: source code for GPIR portlets and services • InfoAggregators: source code for the GPIR Harvester portlet content and various Perl scripts for GRD, LoadLeveler, LSF, and PBS • Instance_Portlets: JSP pages for job monitoring, submission, file browsing • JavaBeans: Java implementations of many portal services, including remote process execution, file management, user management, and authentication • Portlets: Java code for portlet extensions • UIServerBeans: Contains some simple action management JavaBeans that we use for transferring control between Instance_Portlets pages Throughout the project we have used Apache-Ant to simplify compilation and deployment Each of the above directories contains build.xml and build.properties files to be used with Ant to deploy changes into a particular installation directory By changing the properties file to point to different directory paths, the source code for ET011 can be ported to other accounts and host Web servers Known Issues: The portal software at this time needs some thorough testing by friendly users The Web services and MySQL database used by the GPIR system need to be deployed to the ASC web server More complete user and administrator guides need to be written The ASC server currently uses a “fake” certificate generated with the Java keytool This should be replaced with a real certificate We used the instructions from http://jakarta.apache.org/tomcat/tomcat-4.0-doc/ssl-howto.html to this Note when the keytool promts for user name, the host name (gridportal.asc.hpc.mil) should actually be used For some portlets to work, the web server certificate needs to be added to the java keystore This is explained in the document usingHTTPwith_java.pdf that is included in the install_base directory File transfer of some binary files was not correct Sizes of transferred files are correctly retained, but the mime type settings may need to be tweaked For some 12 files (PDF) this is not a problem, but for others (tar files), this is Note file upload is a two stage multipart mime encoded process: the first stage goes through the portlet and the second goes through the proxied JSP page Single step transfers (through a standalone web page) of binary files work correctly, so the problem should be resolvable Acknowledgment This publication made possible through support provided by DoD High Performance Computing Modernization Program (HPCMP) Programming Environment and Training (PET) activities through Mississippi State University under the terms of Agreement No N62306-01-D-7110 Additional support provided by the National Science Foundation, grant NSF-ACI-975249 (Marlon, Tomasz, need grant numbers from you both for DoD projects) Views, opinions, and/or findings contained in this report are those of the authors and should not be construed as an official DoD position, policy or decision unless so designated by other official documentation and no official endorsement should be inferred 13 References G Fox, D Gannon, M Pierce, M Thomas Overview of Grid Computing Environments Document last accessed on September 30, 2003 at http://forge.gridforum.org/projects/ggf-editor/document/GFD-I.9/en/1 14 Figure 1: 15 .. .Building Interoperable Portals with Web Services Mary P Thomas2, Marlon Pierce2, Tomasz Haupt3 Texas Advanced Computing... operationally within MSRC production environments through inter-system protocols, such as web services, to avoid duplication of effort and to share services Second, the use of web servicebased portals. .. an appropriate way to build interoperable grid portals, and Thomas, Pierce, Fox and Haupt are all actively working on integrating web services technologies into their portals projects For example,