Memory Hierarchy and Cache Design

Memory Hierarchy and Cache Design The following sources are used for preparing these slides: • Lecture 14 from the course Computer architecture ECE 201 by Professor Mike Schulte • Lecture from William Stallings, Computer Organization and Architecture, Prentice Hall; 6th edition, July 15, 2002 • Lecture from the course Systems Architectures II by Professors Jeremy R Johnson and Anatole D Ruslanov • Some of figures are from Computer Organization and Design: The Hardware/Software Approach, Third Edition, by David Patterson and John Hennessy, are copyrighted material (COPYRIGHT 2004 MORGAN KAUFMANN PUBLISHERS, INC ALL RIGHTS RESERVED) The Big Picture: Where are We Now? • The Five Classic Components of a Computer • Memory is usually implemented as: – Dynamic Random Access Memory (DRAM) - for main memory – Static Random Access Memory (SRAM) - for cache Processor Input Control Memory Datapath Output Technology Trends (from 1st lecture) Capacity Logic: Speed (latency) 2x in years 2x in years DRAM: 4x in years 2x in 10 years Disk: 4x in years 2x in 10 years DRAM Year 1980 1000:1! Size 64 Kb 2:1! Cycle Time 250 ns 1983 256 Kb 220 ns 1986 Mb 190 ns 1989 Mb 165 ns 1992 16 Mb 145 ns 1995 64 Mb 120 ns 1998 256 Mb 100 ns 2001 Gb 80 ns Who Cares About Memory? Processor-DRAM Memory Gap (latency) 100 10 µProc 60%/yr “Moore’s Law” (2X/1.5yr) Processor-Memory Performance Gap: (grows 50% / year) DRAM DRAM 9%/yr (2X/10 yrs) CPU 198 198 198 198 198 198 198 198 198 898 199 199 199 199 199 199 599 199 199 199 200 Performance 1000 Time Memory Hierarchy Memory technology Typical access time $ per GB in 2004 SRAM 0.5–5 ns $4000–$10,000 DRAM 50–70 ns $100–$200 Magnetic disk 5,000,000–20,000,000 ns $0.50–$2 CPU Processor Level Levels in the memory hierarchy Increasing distance from the CPU in access time Level Data are transferred Level n Size of the memory at each level Memory • SRAM: – Value is stored on a pair of inverting gates – Very fast but takes up more space than DRAM (4 to transistors) • DRAM: – Value is stored as a charge on capacitor (must be refreshed) – Very small but slower than SRAM (factor of to 10) Memory Cell Operation Dynamic RAM • Bits stored as charge in capacitors • Charges leak • Need refreshing even when powered • Simpler construction • Smaller per bit • Less expensive • Need refresh circuits • Slower • Main memory • Essentially analogue – Level of charge determines value Dynamic RAM Structure DRAM Operation • Address line active when bit read or written – Transistor switch closed (current flows) • Write – Voltage to bit line » High for low for – Then signal address line » Transfers charge to capacitor • Read – Address line selected » transistor turns on – Charge from capacitor fed via bit line to sense amplifier » Compares with reference value to determine or – Capacitor charge must be restored Calculation II • • • • • If clock speed is doubled but memory speed remains same: Instruction miss cycles = I x 100% x 2% x 80 = 1.60 x I Data miss cycles = I x 36% x 4% x 80 = 1.16 x I Total miss cycles = 1.60 x I + 1.16 x I = 2.76 x I CPI = + 2.76 = 4.76 • PerfFast / PerfSlow = ( I x 3.38 x L ) / ( I x 4.76 x L/2 ) = 1.41 • Conclusion: Relative cache penalties increase as the machine becomes faster Reducing Cache Misses with a more Flexible Replacement Strategy • In a direct mapped cache a block can go in exactly one place in cache • In a fully associative cache a block can go anywhere in cache • A compromise is to use a set associative cache where a block can go into a fixed number of locations in cache, determined by: (Block number) mod (Number of sets in cache) Direct mapped Block # Data Tag Search Set associative Set # Data Tag Search Fully associative Data Tag Search Example • Three small word caches: Direct mapped, two-way set associative, fully associative • How many misses in the sequence of block addresses: 0, 8, 0, 6, 8? • How does this change with words, 16 words? Locating a Block in Cache • Check the tag of every cache block in the appropriate set • Address consists of parts • Replacement strategy: tag index block offset E.G Least Recently Used (LRU) Address 31 30 12 11 10 8 22 Index V Tag Data V 3210 Tag Data V Tag Data V Tag 253 254 255 22 4-to-1 multiplexor Program gcc Assoc Data I miss rate D miss rate Combined rate 2.0% 1.7% 1.9% 1.6% 1.4% 1.5% 1.6% 1.4% 1.5% Hit Data 32 Effect of associativity on performance 15% KB 12% KB 9% KB 6% KB 16 KB 32 KB 3% 64 KB 128 KB One-way Two-way Associativity Four-way Eight-way Size of Tags vs Associativity • Increasing associativity requires more comparators, as well as more tag bits per cache block • Assume a cache with 4K 4-word blocks and 32 bit addresses • Find the total number of sets and the total number of tag bits for a – direct mapped cache – two-way set associative cache – four-way set associative cache – fully associative cache Size of Tags vs Associativity • Total cache size 4K x words/block x bytes/word = 64Kb • Direct mapped cache: – – – – 16 bytes/block  28 bits for tag and index # sets = # blocks Log(4K) = 12 bits for index  16 bits for tag Total # of tag bits = 16 bits x 4K locations = 64 Kbits • Two-way set-associative cache: – – – – – 32 bytes / set 16 bytes/block  28 bits for tag and index # sets = # blocks /  2K sets Log(2K) = 11 bits for index  17 bits for tag Total # of tag bits = 17 bits x location / set x 2K sets = 68 Kbits Size of Tags vs Associativity • Four-way set-associative cache: – 64 bytes / set – 16 bytes/block  28 bits for tag and index – # sets = # blocks /  1K sets – Log(1K) = 10 bits for index  18 bits for tag – Total # of tag bits = 18 bits x location / set x 1K sets = 72 Kbits • Fully associative cache: – set of K blocks  28 bits for tag and index – Index = bits  tag will have 28 bits – Total # of tag bits = 28 bits x 4K location / set x set = 112 Kbits Reducing the Miss Penalty using Multilevel Caches • To further reduce the gap between fast clock rates of CPUs and the relatively long time to access memory additional levels of cache are used (level two and level three caches) • The primary cache is optimized for a fast hit rate, which implies a relatively small size • A secondary cache is optimized to reduce the miss rate and penalty needed to go to memory • Example: – – – – – – Assume CPI = (with all hits) and GHz clock 100 ns main memory access time 2% miss rate for primary cache Secondary cache with ns access time and miss rate of 5% What is the total CPI with and without secondary cache? How much of an improvement does secondary cache provide? Reducing the Miss Penalty using Multilevel Caches • The miss penalty to main memory: 100 ns / ns per cycle = 500 cycles • For the processor with only L1 cache: Total CPI = + 2% x 500 = 11 • The miss penalty to access L2 cache: ns / ns per cycle = 25 cycles • If the miss is satisfied by L2 cache, then this is the only miss penalty • If the miss has to be resolved by the main memory, then the total miss penalty is the sum of both • For the processor with both L1 and L2 caches: Total CPI = + 2% x 25 + 0.5% x 500 = • The performance ratio: 11 / = 2.8! Memory Hierarchy Framework • Three Cs used to model our memory hierarchy – – – Compulsory misses » Cold-start misses caused by the first access to a block » Solution is to increase the block size Capacity misses » Caused when the cache is full and block needs to be replaced » Solution is to enlarge the cache Conflict misses » Collision misses caused when multiple blocks compete for the same set, in the case of set-associative and fully-associative mappings » Solution is to increase associativity Design Tradeoffs • • As in everything in engineering, multiple design tradeoffs exist when discussing memory hierarchies There are many more factors involved, but the presented ones are the most important and accessible ones Example • A computer system contains a main memory of 32K 16-bit words It has also a 4Kword cache divided into 4-line sets with 64 words per line The processor fetches words from locations 0, 1, 2,…, 4351 in that order sequentially 10 times The cache is 10 times faster than the main memory Assume LRU policy With no cache Fetch time = (10 passes) (68 blocks/pass) (10T/block) = 6800T With cache Fetch time + = = (68) (11T) (9) (48) (T) + (9) (20) (11T) 3160T Improvement = 2.15 first pass other passes Modern Systems • Questions • What is the difference between DRAM and SRAM in terms of applications? • What is the difference between DRAM and SRAM in terms of characteristics such as speed, size and cost? • Explain why one type of RAM is considered to be analog and the other digital • What is the distinction between spatial and temporal locality? • What are the strategies for exploring spatial and temporal locality? • What is the difference among direct mapping, associative mapping and set-associative mapping? • List the fields of the direct memory cache • List the fields of associative and set- associative caches ... Size • Mapping Function • Replacement Algorithm • Write Policy • Block Size • Number of Caches Relationship of Caches and Pipeline Memory D-$ I-$ WB Data MUX RD MEM/WB RD Data Memory RD ALU MUX ID/EX... write-back caches » Hope subsequent writes to the block hit in cache – No-write allocate » The block is modified in memory, but not brought into the cach » Used with write-through caches » Writes... 1.54% 1.66% 256 KB 1.15% 1.17% 1.13% 1.13% LRU 4.39% 1.39% 1.12% 8-way Random 4.96% 1.53% 1.12% For caches with low miss rates, random is almost as good as LRU Q4: What Happens on a Write? • Write