10 2 Parallel Computer Architecture functional unit. But using even more functional units provides little additional gain [35, 99] because of dependencies between instructions and branching of control flow. 4. Parallelism at process or thread level: The three techniques described so far assume a single sequential control flow which is provided by the compiler and which determines the execution order if there are dependencies between instruc- tions. For the programmer, this has the advantage that a sequential programming language can be used nevertheless leading to a parallel execution of instructions. However, the degree of parallelism obtained by pipelining and multiple func- tional units is limited. This limit has already been reached for some time for typical processors. But more and more transistors are available per processor chip according to Moore’s law. This can be used to integrate larger caches on the chip. But the cache sizes cannot be arbitrarily increased either, as larger caches lead to a larger access time, see Sect. 2.7. An alternative approach to use the increasing number of transistors on a chip is to put multiple, independent processor cores onto a single processor chip. This approach has been used for typical desktop processors since 2005. The resulting processor chips are called multicore processors. Each of the cores of a multi- core processor must obtain a separate flow of control, i.e., parallel programming techniques must be used. The cores of a processor chip access the same mem- ory and may even share caches. Therefore, memory accesses of the cores must be coordinated. The coordination and synchronization techniques required are described in later chapters. A more detailed description of parallelism by multiple functional units can be found in [35, 84, 137, 164]. Section 2.4.2 describes techniques like simultaneous multi- threading and multicore processors requiring an explicit specification of parallelism. 2.2 Flynn’s Taxonomy of Parallel Architectures Parallel computers have been used for many years, and many different architec- tural alternatives have been proposed and used. In general, a parallel computer can be characterized as a collection of processing elements that can communicate and cooperate to solve large problems fast [14]. This definition is intentionally quite vague to capture a large variety of parallel platforms. Many important details are not addressed by the definition, including the number and complexity of the processing elements, the structure of the interconnection network between the processing ele- ments, the coordination of the work between the processing elements, as well as important characteristics of the problem to be solved. For a more detailed investigation, it is useful to make a classification according to important characteristics of a parallel computer. A simple model for such a clas- sification is given by Flynn’s taxonomy [52]. This taxonomy characterizes parallel computers according to the global control and the resulting data and control flows. Four categories are distinguished: 2.2 Flynn’s Taxonomy of Parallel Architectures 11 1. Single-Instruction, Single-Data (SISD): There is one processing element which has access to a single program and data storage. In each step, the processing element loads an instruction and the corresponding data and executes the instruc- tion. The result is stored back in the data storage. Thus, SISD is the conventional sequential computer according to the von Neumann model. 2. Multiple-Instruction, Single-Data (MISD): There are multiple processing ele- ments each of which has a private program memory, but there is only one com- mon access to a single global data memory. In each step, each processing element obtains the same data element from the data memory and loads an instruction from its private program memory. These possibly different instructions are then executed in parallel by the processing elements using the previously obtained (identical) data element as operand. This execution model is very restrictive and no commercial parallel computer of this type has ever been built. 3. Single-Instruction, Multiple-Data (SIMD): There are multiple processing ele- ments each of which has a private access to a (shared or distributed) data memory, see Sect. 2.3 for a discussion of shared and distributed address spaces. But there is only one program memory from which a special control processor fetches and dispatches instructions. In each step, each processing element obtains from the control processor the same instruction and loads a separate data element through its private data access on which the instruction is performed. Thus, the instruction is synchronously applied in parallel by all processing elements to different data elements. For applications with a significant degree of data parallelism, the SIMD approach can be very efficient. Examples are multimedia applications or com- puter graphics algorithms to generate realistic three-dimensional views of computer-generated environments. 4. Multiple-Instruction, Multiple-Data (MIMD): There are multiple processing elements each of which has a separate instruction and data access to a (shared or distributed) program and data memory. In each step, each processing element loads a separate instruction and a separate data element, applies the instruction to the data element, and stores a possible result back into the data storage. The processing elements work asynchronously with each other. Multicore processors or cluster systems are examples for the MIMD model. Compared to MIMD computers, SIMD computers have the advantage that they are easy to program, since there is only one program flow, and the synchronous execution does not require synchronization at program level. But the synchronous execution is also a restriction, since conditional statements of the form if (b==0) c=a; else c = a/b; must be executed in two steps. In the first step, all processing elements whose local value of b is zero execute the then part. In the second step, all other process- ing elements execute the else part. MIMD computers are more flexible, as each processing element can execute its own program flow. Most parallel computers 12 2 Parallel Computer Architecture are based on the MIMD concept. Although Flynn’s taxonomy only provides a coarse classification, it is useful to give an overview of the design space of parallel computers. 2.3 Memory Organization of Parallel Computers Nearly all general-purpose parallel computers are based on the MIMD model. A further classification of MIMD computers can be done according to their memory organization. Two aspects can be distinguished: the physical memory organization and the view of the programmer of the memory. For the physical organization, computers with a physically shared memory (also called multiprocessors) and com- puters with a physically distributed memory (also called multicomputers) can be distinguished, see Fig. 2.2 for an illustration. But there also exist many hybrid orga- nizations, for example providing a virtually shared memory on top of a physically distributed memory. computers with memory shared computers with distributed memory MIMD computer systems M ulticomputer systems shared computers with virtually memory parallel and distributed Multiprocessor systems Fig. 2.2 Forms of memory organization of MIMD computers From the programmer’s point of view, it can be distinguished between comput- ers with a distributed address space and computers with a shared address space. This view does not necessarily need to conform with the physical memory. For example, a parallel computer with a physically distributed memory may appear to the programmer as a computer with a shared address space when a corresponding programming environment is used. In the following, we have a closer look at the physical organization of the memory. 2.3.1 Computers with Distributed Memory Organization Computers with a physically distributed memory are also called distributed mem- ory machines (DMM). They consist of a number of processing elements (called nodes) and an interconnection network which connects nodes and supports the transfer of data between nodes. A node is an independent unit, consisting of pro- cessor, local memory, and, sometimes, periphery elements, see Fig. 2.3 (a) for an illustration. 2.3 Memory Organization of Parallel Computers 13 MP DMA RRR RRR RRR computer with distributed memory interconnection network with a hypercube as a) b) P = processor M = local memory c) DMA (direct memory access) d) PM e) R = Router MP DMA Router P MM P P M NN N N N N N N N interconnection network interconnection network node consisting of processor and local memory with DMA connections to the network external input channels external output channels N = node consisting of processor and local memory Fig. 2.3 Illustration of computers with distributed memory: (a) abstract structure, (b) computer with distributed memory and hypercube as interconnection structure, (c) DMA (direct memory access), (d) processor–memory node with router, and (e) interconnection network in the form of a mesh to connect the routers of the different processor–memory nodes Program data is stored in the local memory of one or several nodes. All local memory is private and only the local processor can access the local memory directly. When a processor needs data from the local memory of other nodes to perform local computations, message-passing has to be performed via the interconnection network. Therefore, distributed memory machines are strongly connected with the message-passing programming model which is based on communication between cooperating sequential processes and which will be considered in more detail in 14 2 Parallel Computer Architecture Chaps. 3 and 5. To perform message-passing, two processes P A and P B on different nodes A and B issue corresponding send and receive operations. When P B needs data from the local memory of node A, P A performs a send operation containing the data for the destination process P B . P B performs a receive operation specifying a receive buffer to store the data from the source process P A from which the data is expected. The architecture of computers with a distributed memory has experienced many changes over the years, especially concerning the interconnection network and the coupling of network and nodes. The interconnection network of earlier multicom- puters were often based on point-to-point connections between nodes. A node is connected to a fixed set of other nodes by physical connections. The structure of the interconnection network can be represented as a graph structure. The nodes repre- sent the processors, the edges represent the physical interconnections (also called links). Typically, the graph exhibits a regular structure. A typical network structure is the hypercube which is used in Fig. 2.3(b) to illustrate the node connections; a detailed description of interconnection structures is given in Sect. 2.5. In networks with point-to-point connection, the structure of the network determines the possible communications, since each node can only exchange data with its direct neighbor. To decouple send and receive operations, buffers can be used to store a message until the communication partner is ready. Point-to-point connections restrict paral- lel programming, since the network topology determines the possibilities for data exchange, and parallel algorithms have to be formulated such that their communi- cation fits the given network structure [8, 115]. The execution of communication operations can be decoupled from the proces- sor’s operations by adding a DMA controller (DMA – direct memory access) to the nodes to control the data transfer between the local memory and the I/O controller. This enables data transfer from or to the local memory without participation of the processor (see Fig. 2.3(c) for an illustration) and allows asynchronous communica- tion. A processor can issue a send operation to the DMA controller and can then continue local operations while the DMA controller executes the send operation. Messages are received at the destination node by its DMA controller which copies the enclosed data to a specific system location in local memory. When the processor then performs a receive operation, the data are copied from the system location to the specified receive buffer. Communication is still restricted to neighboring nodes in the network. Communication between nodes that do not have a direct connection must be controlled by software to send a message along a path of direct inter- connections. Therefore, communication times between nodes that are not directly connected can be much larger than communication times between direct neighbors. Thus, it is still more efficient to use algorithms with communication according to the given network structure. A further decoupling can be obtained by putting routers into the network, see Fig. 2.3(d). The routers form the actual network over which communication can be performed. The nodes are connected to the routers, see Fig. 2.3(e). Hardware- supported routing reduces communication times as messages for processors on remote nodes can be forwarded by the routers along a preselected path without 2.3 Memory Organization of Parallel Computers 15 interaction of the processors in the nodes along the path. With router support, there is not a large difference in communication time between neighboring nodes and remote nodes, depending on the switching technique, see Sect. 2.6.3. Each physical I/O channel of a router can be used by one message only at a specific point in time. To decouple message forwarding, message buffers are used for each I/O channel to store messages and apply specific routing algorithms to avoid deadlocks, see also Sect. 2.6.1. Technically, DMMs are quite easy to assemble since standard desktop computers can be used as nodes. The programming of DMMs requires a careful data layout, since each processor can directly access only its local data. Non-local data must be accessed via message-passing, and the execution of the corresponding send and receive operations takes significantly longer than a local memory access. Depending on the interconnection network and the communication library used, the difference can be more than a factor of 100. Therefore, data layout may have a significant influ- ence on the resulting parallel runtime of a program. Data layout should be selected such that the number of message transfers and the size of the data blocks exchanged are minimized. The structure of DMMs has many similarities with networks of workstations (NOWs) in which standard workstations are connected by a fast local area net- work (LAN). An important difference is that interconnection networks of DMMs are typically more specialized and provide larger bandwidths and lower latencies, thus leading to a faster message exchange. Collections of complete computers with a dedicated interconnection network are often called clusters. Clusters are usually based on standard computers and even standard network topologies. The entire cluster is addressed and programmed as a single unit. The popularity of clusters as parallel machines comes from the availabil- ity of standard high-speed interconnections like FCS (Fiber Channel Standard), SCI (Scalable Coherent Interface), Switched Gigabit Ethernet, Myrinet, or InfiniBand, see [140, 84, 137]. A natural programming model of DMMs is the message-passing model that is supported by communication libraries like MPI or PVM, see Chap. 5 for a detailed treatment of MPI. These libraries are often based on standard protocols like TCP/IP [110, 139]. The difference between cluster systems and distributed systems lies in the fact that the nodes in cluster systems use the same operating system and can usually not be addressed individually; instead a special job scheduler must be used. Several cluster systems can be connected to grid systems by using middleware software like the Globus Toolkit, see www.globus.org [59]. This allows a coordinated collab- oration of several clusters. In grid systems, the execution of application programs is controlled by the middleware software. 2.3.2 Computers with Shared Memory Organization Computers with a physically shared memory are also called shared memory ma- chines (SMMs); the shared memory is also called global memory. SMMs consist 16 2 Parallel Computer Architecture Fig. 2.4 Illustration of a computer with shared memory: (a) abstract view and (b) implementation of the shared memory with memory modules MM PPPP interconnection network interconnection network memory modules shared memory (a) (b) of a number of processors or cores, a shared physical memory (global memory), and an interconnection network to connect the processors with the memory. The shared memory can be implemented as a set of memory modules. Data can be exchanged between processors via the global memory by reading or writing shared variables. The cores of a multicore processor are an example for an SMM, see Sect. 2.4.2 for a more detailed description. Physically, the global memory usually consists of sep- arate memory modules providing a common address space which can be accessed by all processors, see Fig. 2.4 for an illustration. A natural programming model for SMMs is the use of shared variables which can be accessed by all processors. Communication and cooperation between the processors is organized by writing and reading shared variables that are stored in the global memory. Accessing shared variables concurrently by several processors should be avoided since race conditions with unpredictable effects can occur, see also Chaps. 3 and 6. The existence of a global memory is a significant advantage, since communi- cation via shared variables is easy and since no data replication is necessary as is sometimes the case for DMMs. But technically, the realization of SMMs requires a larger effort, in particular because the interconnection network must provide fast access to the global memory for each processor. This can be ensured for a small number of processors, but scaling beyond a few dozen processors is difficult. A special variant of SMMs are symmetric multiprocessors (SMPs). SMPs have a single shared memory which provides a uniform access time from any processor for all memory locations, i.e., all memory locations are equidistant to all processors [35, 84]. SMPs usually have a small number of processors that are connected via a central bus which also provides access to the shared memory. There are usually no private memories of processors or specific I/O processors, but each processor has a private cache hierarchy. As usual, access to a local cache is faster than access to the global memory. In the spirit of the definition from above, each multicore processor with several cores is an SMP system. SMPs usually have only a small number of processors, since the central bus provides a constant bandwidth which is shared by all processors. When too many processors are connected, more and more access collisions may occur, thus increas- ing the effective memory access time. This can be alleviated by the use of caches and suitable cache coherence protocols, see Sect. 2.7.3. The maximum number of processors used in bus-based SMPs typically lies between 32 and 64. Parallel programs for SMMs are often based on the execution of threads. A thread is a separate control flow which shares data with other threads via a global address 2.3 Memory Organization of Parallel Computers 17 space. It can be distinguished between kernel threads that are managed by the operating system and user threads that are explicitly generated and controlled by the parallel program, see Sect. 3.7.2. The kernel threads are mapped by the oper- ating system to processors for execution. User threads are managed by the specific programming environment used and are mapped to kernel threads for execution. The mapping algorithms as well as the exact number of processors can be hidden from the user by the operating system. The processors are completely controlled by the operating system. The operating system can also start multiple sequential programs from several users on different processors, when no parallel program is available. Small-size SMP systems are often used as servers, because of their cost- effectiveness, see [35, 140] for a detailed description. SMP systems can be used as nodes of a larger parallel computer by employing an interconnection network for data exchange between processors of different SMP nodes. For such systems, a shared address space can be defined by using a suitable cache coherence protocol, see Sect. 2.7.3. A coherence protocol provides the view of a shared address space, although the physical memory might be distributed. Such a protocol must ensure that any memory access returns the most recently written value for a specific memory address, no matter where this value is physically stored. The resulting systems are also called distributed shared memory (DSM) architectures. In contrast to single SMP systems, the access time in DSM systems depends on the location of a data value in the global memory, since an access to a data value in the local SMP memory is faster than an access to a data value in the memory of another SMP node via the coherence protocol. These systems are therefore also called NUMAs (non-uniform memory access), see Fig. 2.5. Since single SMP sys- tems have a uniform memory latency for all processors, they are also called UMAs (uniform memory access). 2.3.3 Reducing Memory Access Times Memory access time has a large influence on program performance. This can also be observed for computer systems with a shared address space. Technological develop- ment with a steady reduction in the VLSI (very large scale integration) feature size has led to significant improvements in processor performance. Since 1980, integer performance on the SPEC benchmark suite has been increasing at about 55% per year, and floating-point performance at about 75% per year [84], see Sect. 2.1. Using the LINPACK benchmark, floating-point performance has been increasing at more than 80% per year. A significant contribution to these improvements comes from a reduction in processor cycle time. At the same time, the capacity of DRAM chips that are used for building main memory has been increasing by about 60% per year. In contrast, the access time of DRAM chips has only been decreasing by about 25% per year. Thus, memory access time does not keep pace with processor performance improvement, and there is an increasing gap between processor cycle time and memory access time. A suitable organization of memory access becomes 18 2 Parallel Computer Architecture PP n21 P (a) cache ehcacehcac memory P 1n P 2 P MM M n21 (b) interconnection network processing elements interconnection network 1 P 2 P n P 1 C 2 C n C MMM n21 (c) processing elements 1n n21 2 PP P CC C Cache (d) Processor interconnection network processing elements Fig. 2.5 Illustration of the architecture of computers with shared memory: (a) SMP – symmet- ric multiprocessors, (b) NUMA – non-uniform memory access, (c) CC-NUMA – cache-coherent NUMA, and (d) COMA – cache-only memory access more and more important to get good performance results at program level. This is also true for parallel programs, in particular if a shared address space is used. Reducing the average latency observed by a processor when accessing memory can increase the resulting program performance significantly. Two important approaches have been considered to reduce the average latency for memory access [14]: the simulation of virtual processors by each physical processor (multithreading) and the use of local caches to store data values that are accessed often. We give now a short overview of these approaches in the following. 2.3 Memory Organization of Parallel Computers 19 2.3.3.1 Multithreading The idea of interleaved multithreading is to hide the latency of memory accesses by simulating a fixed number of virtual processors for each physical processor. The physical processor contains a separate program counter (PC) as well as a separate set of registers for each virtual processor. After the execution of a machine instruc- tion, an implicit switch to the next virtual processor is performed, i.e., the virtual processors are simulated by the physical processor in a round-robin fashion. The number of virtual processors per physical processor should be selected such that the time between the executions of successive instructions of a virtual processor is sufficiently large to load required data from the global memory. Thus, the memory latency will be hidden by executing instructions of other virtual processors. This approach does not reduce the amount of data loaded from the global memory via the network. Instead, instruction execution is organized such that a virtual processor accesses requested data not before their arrival. Therefore, from the point of view of a virtual processor, memory latency cannot be observed. This approach is also called fine-grained multithreading, since a switch is performed after each instruction. An alternative approach is coarse-grained multithreading which switches between virtual processors only on costly stalls, such as level 2 cache misses [84]. For the programming of fine-grained multithreading architectures, a PRAM-like program- ming model can be used, see Sect. 4.5.1. There are two drawbacks of fine-grained multithreading: • The programming must be based on a large number of virtual processors. There- fore, the algorithm used must have a sufficiently large potential of parallelism to employ all virtual processors. • The physical processors must be specially designed for the simulation of virtual processors. A software-based simulation using standard microprocessors is too slow. There have been several examples for the use of fine-grained multithreading in the past, including Dencelor HEP (heterogeneous element processor) [161], NYU Ultracomputer [73], SB-PRAM [1], Tera MTA [35, 95], as well as the Sun T1 and T2 multiprocessors. For example, each T1 processor contains eight processor cores, each supporting four threads which act as virtual processors [84]. Section 2.4.1 will describe another variation of multithreading which is simultaneous multithreading. 2.3.3.2 Caches A cache is a small, but fast memory between the processor and main memory. A cache can be used to store data that is often accessed by the processor, thus avoiding expensive main memory access. The data stored in a cache is always a subset of the data in the main memory, and the management of the data elements in the cache is done by hardware, e.g., by employing a set-associative strategy, see [84] and Sect. 2.7.1 for a detailed treatment. For each memory access issued by the processor, the hardware first checks whether the memory address specified currently resides . techniques for selecting paths through networks and switching techniques for message forwarding over a given path. Section 2.7 considers memory hierarchies of sequential and parallel plat- forms and. the resources of parallel platforms and to exchange data and information between these resources. Interconnection networks also play an impor- tant role in multicore processors for the connection. In the last decades, high-performance research included new developments in parallel hard- ware and software technologies, and a steady progress in parallel high-performance computing can be observed.