grid computing esis analysis of its impact on business and the shift to a services infrastructure

GRID COMPUTING: ESI’s ANALYSIS OF ITS IMPACT ON BUSINESS AND THE SHIFT TO A SERVICES INFRASTRUCTURE Robert Cohen, Ph.D., Fellow Economic Strategy Institute bcohen@bway.net National Academy of Sciences GUIRR February 24, 2004 WHAT IS DRAMATICALLY NEW ABOUT ABOUT GRIDS? a a a a GRIDS BEGIN AS A WAY TO ACCELERATE COMPUTING THEY EVOLVE TO A TECHNOLOGY THAT MAKES APPLICATIONS INFRASTRUCTURE AWARE THIS IMBEDS WEB SERVICES WITHIN A NEW INTELLIGENT LAYER BETWEEN APPLICATIONS AND COMPUTE/STORAGE RESOURCES THIS CREATES THE “SERVICES GRID” • PRIORITIZING & AUTOMATING THE USE OF COMPUTE/STORAGE RESOURCES AND • AUTOMATING COMPLEX BUSINESS PROCESSES WHY GRIDS ARE REVOLUTIONARY a THEY PROVIDE A STRUCTURE FOR THE NEW SERVICES ECONOMY - INCLUDING SERVICES AND CONTROL IN MANUFACTURING • LET CORPORATIONS MANAGE CRITICAL STANDARDS ACROSS THE FIRM: CREDIT RISK, SIX SIGMA • SUPPORT COLLABORATIVE PRODUCT DEVELOPMENT IN THE FIRM & WITH PARTNERS • ANALYTICAL CAPABILITIES OF A NEW ORDER REDUCE FINANCIAL RISK, PROVIDE A TYPE OF “SIX SIGMA” FOR SERVICE FIRMS • CREATE WAY TO SUPPORT SERVICE-BASED PRODUCTS GRIDS: BREAKING INTO THE SERVICES SIDE OF BUSINESS a IDC: – Grids hold significant potential, especially if the technology can break out from the HPC marketplace and capture the attention of commercial users – If this "chasm" can be crossed, revenue sources will largely shift from hardware in the HPC market to a more software and services based model in the commercial market a THE MANAGEMENT AND SERVICES SIDE OF GRID IS FAR GREATER THAN DIRECT COMPUTER SERVICES – HUGE REDUCTION IN RISK FACING BANKS, CORPORATIONS – GREATER CONTROL OF PRODUCT DEVELOPMENT, SUPPLY CHAINS, FINANCIAL PERFORMANCE What is a Grid? - Ian Foster, “What is the Grid? A Three Point Checklist” a Coordinates resources that are not subject to centralized control … – integrates and coordinates resources and users that live within different control domains—for example, the user’s desktop vs central computing; – It addresses the issues of security, policy, payment, membership, and so forth that arise in these settings Otherwise, we are dealing with a local management system a Uses standard, open, general-purpose protocols and interfaces – built from multi-purpose protocols and interfaces that address such fundamental issues as authentication, authorization, resource discovery, and resource access – these protocols and interfaces must be standard and open Otherwise, we are dealing with an application specific system a Delivers nontrivial qualities of service – its constituent resources are to be used in a coordinated fashion to deliver various qualities of service, relating to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands – the utility of the combined system is significantly greater than that of the sum of its parts 30000 Does not include downloads from: NMI, UK eScience, EU Datagrid, IBM, Platform, etc GT 2.0 Released 25000 Physiology of the Grid Paper Released 20000 Anatomy of the Grid Paper Released NSF & European Commission Initiate Many New Grid Projects DARPA, NSF, and DOE begin funding Grid work NASA begins funding Grid work, DOE adds support Significant Commercial Interest in Grids 15000 10000 GT 1.0.0 Released Early Application Successes Reported 5000 D o w n lo a d s p e r M o n th fro m ftp g lo b u s o rg Globus Toolkit® History 1997 1998 1999 2000 2001 2002 Globus Open Source Grid Software G T G T G T G T G T Community Scheduler Framework [contribution] Delegation Service Python WS Core [contribution] C WS Core Community Authorization Service OGSA-DAI [Tech Preview] WS Authentication Authorization Reliable File Transfer Grid Resource Allocation Mgmt (WS GRAM) Monitoring & Discovery System (MDS4) Java WS Core GridFTP Grid Resource Allocation Mgmt (Pre-WS GRAM) Monitoring & Discovery System (MDS2) C Common Libraries Pre-WS Authentication Authorization Web Services Components Components Replica Location Service XIO Credential Management Security Data Management Non-WS Execution Management Information Services Common Runtime Services Oriented Infrastructure as the Next Phase in the Evolution of Grids Infrastructure is Globus that is “completely” Web Service-ized Source: Rich Miller, Univa The SOI links to applications using Grid Services, Applications are linked by Web Services What is a Cluster vs a Grid? Gentzsch, “Grid Computing: A New Technology for the Advanced Web” GRID: FLEXIBLE, HIGH-PERFORMANCE ACCESS TO ALL SIGNIFICANT RESOURCES On-demand creation of powerful virtual computing systems Source: Dietmar Erwin, “UNICORE and EUROGRID: Grid Computing in Europe,” EUROGRID 10 and UNICORE Web Services Spending, Software and Related Costs, 2003-2010 14000 Millions of Dollars 12000 TELECOM SERVICES AUTOS HEALTH CARE AEROSPACE PHARMACEUTICALS FINANCIAL SVCS COMPUTERS SEMICONDUCTORS 10000 8000 6000 4000 2000 2003 2004 2006 2008 2010 IDC FIGURES FOR WEB SERVICES SPENDING FOR ALL US INDUSTRIES ARE 2007: $21 BILLION; 2008: $28 BILLION THE IDC FIGURES ARE NEARLY DOUBLE THE ESI ESTIMATES, LARGELY BECAUSE IDC ASSUMES A MORE RAPID DEPLOYMENT OF WEB SERVICES THE IDC STUDY MAY ALSO ASSUME 27 THAT MORE INDUSTRIES, SUCH AS THE RETAIL SECTOR, WILL BE EARLY ADOPTERS OF WEB SERVICES WE HAVE NOT INCLUDED THAT SECTOR IN OUR ANALYSIS 28 GM, FORD AND THE SERVICE GRID a ADVANTAGE: GOING BEYOND ENTERPRISE GRIDS TO CONNECT TO SUPPLIERS & CUSTOMERS • GM, FORD USED BIG CLUSTERS, GRIDS FOR CRASH SIMULATIONS, COMPLEX DESIGN NEEDS • SERVICE GRID WILL GET SOFTWARE FIXES TO CUSTOMERS WITH “HIGHLY ELECTRONIC” CARS & INFORMATION TO DEALERS • NEW ISSUE: CORRUPTION OF SOFTWARE FOR ENGINE CONTROLS; IMPOSSIBLE TO FIX WITHOUT SERVICE GRID • SERVICE GRID SENDS OUT FIX AFTER THE GLITCH IS ANALYZED DEALERS, CUSTOMERS GET FIX OVER HIGH SPEED CONNECTIONS FOR THE SERVICE GRID 29 BANK OF AMERICA, JPMORGANCHASE AND RISK ANALYSIS a ADVANTAGE: TREMENDOUS COST SAVINGS ACROSS A BANK CLUSTERS AND GRIDS EMPLOYED TO ESTIMATE A BANK’S RISK EXPOSURE EVERY DAY LOWERING RISK RESERVES CAN SAVE BILLIONS • BANKS INITIALLY USED CLUSTERS & GRIDS TO RUN RISK SIMULATION MODELS FOR TRADING GROUPS • ANALYZING OVERALL RISK PERMITS ONLY THE TOP 10 US BANKS TO REDUCE THEIR CAPITAL ALLOCATION FOR RISK • US, INTERNATIONAL REGULATORS PERMIT USE OF SOPHISTICATED MATHEMATICAL MODELS TO DO SOPHISTICATED RISK ANALYSIS ACROSS ALL PARTS OF BANKS THIS WOULD NOT BE POSSIBLE WITHOUT VERY LARGE CLUSTERS & PARALLEL PROCESSING • REDUCING THESE COSTS CUTS OVERALL OPERATING COSTS • SYSTEMS DIFFICULT TO REPLICATE: NEED ANALYTIC SKILLS & SOPHISTICATED COMPUTER SYSTEMS 30 CADENCE/ SYNOPSYS: GREATER INSIGHT INTO THE DETAILS OF CHIP DEVELOPMENT a KEY COMPETITIVE ADVANTAGE: VAST COMPUTE POWER CREATES ABILITY TO EXAMINE DESIGN AT FORMERLY IMPOSSIBLE LEVELS – ALSO, PUTTING DISPARATE DESIGN GROUPS IN SEVERAL LOCATIONS IN A SINGLE DESIGN TEAM REDUCES TIME TO MARKET, EXTENDS ANALYTIC CAPABILITIES TO ALL PARTS OF FIRM a ADDITIONAL INNOVATIONS & SAVINGS: – INITIAL SURGE IN PRODUCTIVITY (UP 70-90%) DUE TO FAR MORE EFFICIENT BATCH PROCESSING OF COMPLEX JOBS; ROI: MONTH – GREATLY REDUCED TIME TO FINAL PRODUCT, TIME TO RETRIEVE AND USE INFORMATION, COST OF DESIGNER TIME – MORE SOPHISTICATED MANAGEMENT OF EXPENSIVE LICENSES – ABILITY TO PACKAGE GRID CAPABILITIES WITH LICENSES AS PART OF A NEW SOFTWARE-SERVICE PRODUCT 31 INDUSTRY LEVEL CHANGES DUE TO ADOPTION OF GRIDS AND WEB SERVICES Auto Industry Financial Services Productivity Gain – Grids & Web Services 25% 26% Output Increase 18% 10% Price change -15% -12% Increase in Spending on Communications Services 437% 97% 32 GRIDS AND BANDWIDTH DEMAND a a a a EARLY CLUSTERS - BEHIND FIREWALL, VIRTUALIZATION OF COMPUTE RESOURCES AND STORAGE - COST GAINS EARLY GRIDS - “BEYOND” FIREWALL, VIRTUALIZATION OF NETWORK, HIGHER LEVEL OF CONTROL AND MANAGEMENT LATER GRIDS - INTERCONNECTED ORGANIZATIONS, VIRTUAL NETWORKS & HIGHER LEVELS OF VIRTUALIZATION OF COMPUTE RESOURCES & STORAGE TELCOS’ ROLE - MANAGE GRID NETWORKS, PROVIDE SERVICES BEYOND VPNs; ARE THESE JUST VIRTUAL LAN SERVICES WITH ETHERNET/LIGHTWAVE CONNECTIVITY? 33 GRID “Computing” – The BT Vision APPLICATIONS ENTERPRISE A WANS LANS APPLICATION “VIRTUAL” RUNTIME SPACE SERVICE PROVIDER IT RESOURCE “POOL” RESOURCE BROKER N/W RESOURCE “POOL” GRID TOOLKIT AAI Modules BT • GRID progressively turns all Enterprise ICT resources virtual • Traditional Enterprise relationships with Telcos and IT companies replaced • BT provides network-centric application delivery services to Enterprise – Distributable Applications delivered across “Pool” of virtual ICT resources – Non-distributable applications through dedicated resources from “Pool” – Vertically integrated delivery includes network connectivity – Resource management of Virtual Network is the key success factor 34 ON-DEMAND COMPUTING a a a ON-DEMAND COMPUTING IS ONE OF THE GREATEST BENEFITS OF GRIDS IN ON-DEMAND COMPUTING, A FIRM DOES NOT BUILD ITS OWN GRID THAT INCLUDES IN-HOUSE COMPUTERS AND DATA STORAGE A GRID SERVICES FIRM PROVIDES THE COMPUTE POWER AND DATA STORAGE CAPABILITIES TO SUPPORT A FIRM’S GRID AND BUNDLES THEM WITH NETWORK CONNECTIONS THROUGH A SERVICE PROVIDER(S) • RELYING ON OUTSOURCING, MEANS THAT A FIRM NO LONGER NEEDS TO PURCHASE ALL THE COMPUTERS AND STORAGE REQUIRED TO ESTABLISH ITS OWN GRID AND TO PAY FOR COMMUNICATIONS SERVICES TO LINK ALL PARTS OF ITS CORPORATE GRID a LARGE, INTEGRATED FIRMS BELIEVE THERE ARE CLEAR COST SAVINGS IN USING O ON-DEMAND COMPUTING VS PURCHASING EQUIPMENT AND BUILDING THEIR OWN GRID 35 36 BANDWIDTH GROWTH PROJECTIONS FOR INDIVIDUAL FIRMS, 2003-2006 AND BEYOND Firm Bandwidth Growth per year, 2003-4 Bandwidth Growth per year, 2005-6 Bandwidth Growth per year, after 2006 Large pharma 15% 30% 30-35% Large pharma 40% 60% 75% 300% over period 10-15% >10-15% 10% 40% 30% Large semiconductor Large auto 37 Forecast Bandwidth Monthly Demand: Baseline AT&T/RHK vs Grid Forecast The Use of Grids and Web Services Results in a Substantial Increase in Bandwidth Demand: The examples are derived from two separate studies 30 Exabytes 25 20 Baseline Baseline+Grid 15 One of the early goals of the ESI study was to evaluate how grids affected the demand for broadband access These charts provide some illustrations from the research 10 2002 2003 2004 2005 2006 2007 2008 2009 The economic model provides us with a way to see how much spending for communications services will shift from its assumed pattern through 2010 In many of the early adopter industries we studied, spending on communications services more than doubled by 2010 We believe that this is likely to reflect new spending on broadband access Pharmaceutical Industry Spending on Communications Services - Impact of Grids and Web Services 2.5 Billions of 1.5 Dollars 0.5 2002 Baseline Spending Forecast 2005 2008 2010 Spending for Grids and Web Services 38 Auto Industry Spending on Communications Services Baseline Spending and the Impact of Grids and Web Services 3000.0 2500.0 2000.0 Millions of 1500.0 Dollars 1000.0 500.0 0.0 2002 Baseline Spending Forecast 2005 2008 2010 Additional Spending - Grids and Web Services 39 Financial Services Industry Spending on Communications Services - Impact of Grids and Web Services Billions of Dollars 70 60 50 40 30 20 10 2002 Baseline Spending Forecast 2005 2008 2010 Additional Spending for Grids and Web Services 40 CONCLUSIONS a a a a COLLABORATION USING GRIDS CAN LEVERAGE THE BENEFITS OF COMPUTE POWER AND USE EXCESS COMPUTE CAPACITY THAT IS AVAILABLE NOW MOST CXOs ARE NOT AWARE OF THIS CHANGE RFID HAS GOTTEN MORE PRESS THAN GRIDS GRIDS CREATE THE POTENTIAL FOR NEW BUSINESS MODELS THEY WILL FACILITATE SMALLER AND MEDIUM SIZED FIRMS (SUPPLIERS) WORKING EFFECTIVELY WITH LARGER FIRMS LARGE EFFICIENCY GAINS WILL MAKE EARLY ADOPTER FIRMS MORE DYNAMIC THAN OTHER FIRMS IN THE ECONOMY A MAJOR BREAKTHROUGH WILL OCCUR AS FIRMS USE GRIDS TO CHANGE THE WAY THEY DO BUSINESS - CHANGING BASIC BUSINESS PROCESSES, SUCH AS PRODUCT DESIGN, INVESTMENT POLICIES WITH SUCH CHANGES, PRODUCTIVITY GAINS WILL BE DRAMATIC, BUT COMPETITIVE CHANGES WILL BE EVEN GREATER FIRMS THAT USE GRIDS WILL CREATE NEW BUSINESS MODELS 41 ...WHAT IS DRAMATICALLY NEW ABOUT ABOUT GRIDS? a a a a GRIDS BEGIN AS A WAY TO ACCELERATE COMPUTING THEY EVOLVE TO A TECHNOLOGY THAT MAKES APPLICATIONS INFRASTRUCTURE AWARE THIS IMBEDS WEB SERVICES. .. 2010 Grid Adoption 23 AEROSPACE AND HOSPITALS HAVE LOWER PRICES Prices - Aerospace Comparison of baseline and Adoption of Grids Prices - Hospitals 1.39 Comparison of baseline and Adoption of Grids... -5.0% Changes due to the Adoption of Web Services Changes due to the Adoption of Grids and Web Services 18 EXAMPLES OF ADOPTION FIGURES FOR FINANCIAL SERVICES AND AUTOS The main difference is the