Operating system

Table of Contents Main Page Table of content Copyright International Computer Science Series 11 Preface 13 Computer systems curriculum 15 Audience 17 An outline of the contents 18 Order of presentation 21 Further study 22 Objective 23 Instructor's guide 24 Acknowledgements 25 Trademark notice 26 Chapter System design requirements 27 1.1 Inherently concurrent systems 29 1.2 Supporting potentially concurrent applications 39 1.3 Architectures for software systems 43 1.4 Defining a concurrent system 50 1.5 Systems implementation requirements 52 1.6 Security, protection and fault tolerance in system design 54 Part I: System design: Technology and principles 56 Chapter System structure and dynamic execution 57 2.1 System structure: Conceptual views 58 2.2 System structure: Implementational views 61 2.3 The process concept 69 2.4 Operating system functions 72 2.5 Operating system invocation and protection 74 2.6 Operating system structure 76 2.7 Object structuring within and supported by operating systems 80 2.8 Distributed object systems, platforms and middleware 83 2.9 Security and protection 84 2.10 Summary 86 Chapter The hardware interface, I/O and communications 89 3.1 Overview 90 3.2 Device interfacing 91 3.3 Exceptions 101 3.4 Multiprocessors 105 3.5 User-level input and output 107 3.6 Communications management 110 3.7 Communications networks, interfaces and drivers 112 3.8 Communications software 115 3.9 Communications handling within and above an operating system 120 3.10 Summary 122 Chapter Support for processes 124 Terminology 125 4.1 Use of processes in systems 126 4.2 Processes and processors 128 4.3 Process state 130 4.4 Synchronizing with the hardware: Events and the WAIT operation 132 4.5 The process data structure 135 4.6 Scheduling: General approaches 136 4.7 Scheduling for shared-memory multiprocessors 139 4.8 Process scheduling to meet real-time requirements 140 4.9 Process abstraction and implementation 143 4.10 Operating system structure and placement of processes 145 4.11 Multi-threaded process implementation 147 4.12 Processes in languages, runtime systems and operating systems 149 4.13 Process state in language systems and operating systems 150 4.14 Sequential programs with system calls 154 4.15 Evolution of concurrency in programming languages 155 4.16 Implementation of processes in language systems 164 4.17 Thread package architectures 167 4.18 Java threads and exceptions 170 4.19 Summary 173 178 5.1 Memory management 179 5.2 The memory hierarchy 180 5.3 The address space of a process 181 5.4 Segmented virtual memory 185 5.5 Paged virtual memory 189 5.6 Combining segmentation and paging 192 5.7 Operating system data structures 194 5.8 An example of a memory management unit (MMU) 198 5.9 Memory management in system design 201 5.10 Summary 202 Chapter File management 204 6.1 File management 205 6.2 An overview of filing system functions 206 6.3 File and directory structure 208 6.4 The filing system interface 211 6.5 The filing system implementation 214 6.6 Modern file system design 222 6.7 Network-based file servers 225 6.8 Integrating virtual memory and storage 232 6.9 Summary 235 Chapter Fundamentals of distributed systems 237 7.1 Introduction 238 7.2 Evolution of distributed systems for the workplace 239 7.3 Personal computing 241 7.4 Model and software architecture 242 7.5 Special characteristics of distributed systems 243 7.6 Time in distributed systems 244 7.7 Naming 248 7.8 Mobile users, computers and objects 253 7.9 Summary 254 Chapter Security 256 8.1 Scenarios and security requirements 257 8.2 Threats and risk analysis 259 8.3 Approaches to encryption 261 8.4 Algorithms 264 8.5 Protocols 267 8.6 Examples 272 8.7 Summary 276 Chapter Memory management Part II: Concurrency control in main memory 278 Chapter System structure 280 9.1 Processes sharing an address space 281 9.2 Processes in separate address spaces 283 9.3 Sharing the operating system 285 9.4 Summary of process placement in the two models 287 9.5 Requirements for process interaction 289 9.6 Types of process interaction 291 9.7 A process interaction 293 9.8 Definition of single concurrent actions 297 Chapter 10 Low-level synchronization: Implementation 300 10.1 Process synchronization compared with event signal and wait 302 10.2 Mutual exclusion 303 10.3 Hardware support for mutual exclusion 306 10.4 Semaphores 310 311 315 320 Chapter 11 Low-level synchronization: Algorithms 323 11.1 Introduction 324 11.2 An example of semaphores in system design: The THE system 325 11.3 The producerconsumer, bounded buffer problem 327 11.4 Safety and liveness properties 332 11.5 The multiple readers, single writer problem 333 11.6 Limitations of semaphores 337 11.7 Eventcounts and sequencers 338 11.8 POSIX threads 342 11.9 Summary 347 11.10 Case study with exercises: Management of a disk block cache 348 Chapter 12 IPC with shared memory 355 12.1 Critical regions in programming languages 356 12.2 Monitors 358 12.3 Synchronization at the granularity of operations 371 12.4 Summary 376 Chapter 13 IPC and system structure 380 13.1 Styles of inter-process communication 381 13.2 System structure and IPC with shared memory 383 13.3 System structure and IPC without shared memory 384 13.4 Systems where shared-memory communication is appropriate 385 13.5 Systems where shared-memory communication is not appropriate 386 13.6 Examples from classical UNIX 387 13.7 Overview of inter-process communication 389 13.8 Duality of system structures 391 13.9 Naming 394 13.10 Summary 395 Chapter 14 IPC without shared memory 397 14.1 Introduction 398 14.2 Use of files for common data 399 14.3 UNIX pipes 400 14.4 Asynchronous message passing 402 14.5 Variations on basic message passing 405 14.6 Implementation of asynchronous message passing 411 14.7 Synchronous message passing 413 14.8 Message passing in programming languages 415 10.5 Styles of use of semaphores 10.6 Implementation of semaphore operations 10.7 Summary 14.9 Multi-threading in clients and servers 419 14.10 Summary 420 Chapter 15 Crash resilience and persistent data 422 15.1 Crashes 423 15.2 A model of a crash 424 15.3 Crash resilience or failure transparency 425 15.4 Idempotent (repeatable) operations 426 15.5 Atomic operations on persistent objects 427 15.6 Implementation of atomic operations 428 15.7 Non-volatile memory 431 15.8 A single operation on persistent data 432 15.9 Database management systems' requirements on operating systems 434 15.10 Summary 435 Chapter 16 Distributed IPC 437 16.1 Introduction 438 16.2 Special characteristics of distributed systems 439 16.3 Distributed IPC: Message passing 440 16.4 Integration of IPC with communications 443 16.5 Java's sockets and streams 446 16.6 Distributed programming paradigms 450 16.7 Remote procedure call (RPC) 452 16.8 RPClanguage integration 457 16.9 Java's RMI: RPC in the general object model 462 16.10 Critique of synchronous invocation 467 16.11 Naming, location and binding 469 16.12 Summary of Part II 473 Part III: Transactions 476 477 478 479 480 482 484 487 17.7 Summary 488 Chapter 18 Resource allocation and deadlock 490 18.1 Requirements for dynamic allocation 491 18.2 Deadlock 492 18.3 Livelock and starvation 494 18.4 Conditions for deadlock to exist 495 18.5 The dining philosophers problem 497 18.6 Object allocation graphs 499 18.7 Data structures and algorithms for deadlock detection 501 18.8 Deadlock avoidance 504 18.9 Information on releasing objects: Multiphase processes 506 18.10 Distributed deadlocks 507 18.11 Summary 509 Chapter 19 Transactions 512 19.1 Introduction 513 19.2 Transaction specification and programming 516 19.3 The definition of serializability and consistency 518 19.4 The ACID properties of transactions 520 19.5 Indicating specific orderings of transactions 521 19.6 A system model for transaction processing 522 Chapter 17 Composite operations 17.1 Composite operations 17.2 Composite operations in main memory 17.3 Composite operations involving main memory and persistent memory 17.4 Concurrent execution of composite operations 17.5 Potential problems 17.6 Crashes 19.7 Dependency graphs for transactions 527 19.8 Histories and serialization graphs 530 19.9 Dealing with aborts: More about the property of isolation 533 19.10 Summary 537 Chapter 20 Concurrency control 539 20.1 Introduction 540 20.2 Concurrent composite operations in main memory only 541 20.3 Structure of transaction management systems 544 20.4 Concurrency control through locking 545 20.5 Time-stamp ordering (TSO) 548 20.6 Optimistic concurrency control (OCC) 552 20.7 Summary 559 563 21.1 Requirements for recovery 564 21.2 The object model, object state and recovery 565 21.3 Concurrency, crashes and the properties of transactions 566 21.4 Logging and shadowing for crash resilience 567 21.5 Use of a recovery log 568 21.6 Idempotent undo and redo operations 571 21.7 Transaction states on a failure 572 21.8 An algorithm for recovery 573 21.9 Location databases for mobile objects 575 21.10 Summary 576 Chapter 22 Distributed transactions 578 22.1 An object model for distributed systems 579 22.2 Distributed transaction processing 580 22.3 Communication 582 22.4 Concurrency control: Two-phase locking (2PL) 583 22.5 Concurrency control: Time-stamp ordering (TSO) 584 22.6 Optimistic concurrency control (OCC) 585 22.7 Commit and abort in a distributed system 586 22.8 Atomic commitment: The two-phase commit (2PC) protocol 588 22.9 Two-phase validation for OCC 590 22.10 Summary 592 Chapter 23 Distributed computations 594 23.1 Introduction 595 23.2 Process groups 596 23.3 Consistency of data replicas 599 23.4 Ordering message delivery 602 23.5 Distributed, 'N'-process mutual exclusion 604 23.6 Summary of Part III 607 Part IV: Case Studies 609 Chapter 24 Classical UNIX 611 24.1 Introduction 612 24.2 Evolution of UNIX 613 24.3 System structure and the UNIX kernel 614 24.4 File system interfaces 618 24.5 File system implementation 620 24.6 Process creation, termination and scheduling 625 24.7 IPC: Pipes and signals 633 24.8 Summary 637 Chapter 25 LINUX, Solaris and contemporary UNIX 639 25.1 Introduction 640 25.2 Kernel structure 641 Chapter 21 Recovery 25.3 SVr4 IPC 646 25.4 Sockets and file subsystem integrated IPC 650 25.5 Memory management 656 25.6 Multiprocessor scheduling 659 25.7 Summary 661 663 26.1 Mechanisms for extensibility 664 26.2 Compile-time specialization 666 26.3 Microkernel operating systems 668 26.4 Downloadable code 672 26.5 Exokernels and vertical structuring 674 26.6 Nested virtual machines 679 26.7 Extensible virtual machines 682 26.8 Summary 687 Chapter 27 Windows 2000 688 27.1 Introduction to Windows 2000 689 27.2 System structure 691 27.3 The object model and object manager 694 27.4 The kernel 701 27.5 Processes, threads, fibres and concurrency control 704 27.6 The I/O subsystem 709 27.7 The NT filing system, NTFS 713 27.8 Networking 716 27.9 Summary 718 Chapter 28 The World Wide Web 720 28.1 A simple web-client, web-server interaction 721 28.2 Naming web pages 723 28.3 Communication using HTTP 724 28.4 Document representation 725 28.5 Executing programs at server and client 727 28.6 Security 729 28.7 Concurrency control 730 28.8 Scalability issues 731 28.9 Web-based middleware: XML and SOAP 732 28.10 Summary 734 Chapter 29 Middleware 736 29.1 Middleware paradigms 737 29.2 Java middleware 740 29.3 OMG and OMA 746 29.4 CORBA 750 29.5 ODMG 755 29.6 COM, DCOM and NET 756 29.7 Message-oriented middleware (MOM) 759 29.8 Summary 763 Chapter 30 Transaction processing monitors and systems 765 30.1 Transaction processing monitors 766 30.2 Introduction to some electronic funds transfer (EFT) applications 771 30.3 International inter-bank payments: SWIFT 774 30.4 Authentication by PIN 775 30.5 The international automatic teller machine (ATM) network service 778 30.6 Load and traffic in TP systems 781 30.7 Summary and trends 782 Appendix Evolution of computer systems 784 A.1 Evolution and exploitation of technology 785 Chapter 26 Extensible systems A.2 Operating systems and distributed operating systems 793 A.3 Databases 798 A.4 Concurrency control 800 Bibliography 806 Glossary 822 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design / 850 [ Team LiB ] • Table of Contents Operating Systems: Concurrent and Distributed Software Design By Jean Bacon, Tim Harris Publisher: Addison Wesley Pub Date: March 11, 2003 ISBN: 0-321-11789-1 Pages: 720 Today's software systems rely on the concepts of concurrency, modularity and distribution, both within the design of the operating system and those systems that it supports The modern approach of this book acknowledges that distributed systems are now commonplace and a reader is more likely to be using one than a centralized time-sharing system This avoids the focus on centralized systems, instead placing the operating system within its context and providing an overview of system components and their relationships Java is used throughout to illustrate object-oriented concepts, concurrent algorithms and distributed programming Core OS concepts and functions are explained Covers the latest developments in OS theory and practice OS case studies in UNIX, LINUX, Windows 2000/NT,JVM Objects, concurrency and transactions are unifying themes Incorporates Java throughout the book System security is covered in a full chapter Integrated approach to database and distributed system design Further case studies on Web services, Middleware and Transaction Processing [ Team LiB ] / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design / 850 [ Team LiB ] • Table of Contents Operating Systems: Concurrent and Distributed Software Design By Jean Bacon, Tim Harris Publisher: Addison Wesley Pub Date: March 11, 2003 ISBN: 0-321-11789-1 Pages: 720 Copyright International Computer Science Series Preface Computer systems curriculum Audience An outline of the contents Order of presentation Further study Objective Instructor's guide Acknowledgements Trademark notice Chapter System design requirements Section 1.1 Inherently concurrent systems Section 1.2 Supporting potentially concurrent applications Section 1.3 Architectures for software systems Section 1.4 Defining a concurrent system Section 1.5 Systems implementation requirements Section 1.6 Security, protection and fault tolerance in system design Part I System design: Technology and principles Chapter System structure and dynamic execution Section 2.1 System structure: Conceptual views Section 2.2 System structure: Implementational views Section 2.3 The process concept Section 2.4 Operating system functions Section 2.5 Operating system invocation and protection Section 2.6 Operating system structure Section 2.7 Object structuring within and supported by operating systems Section 2.8 Distributed object systems, platforms and middleware Section 2.9 Security and protection Section 2.10 Summary Chapter The hardware interface, I/O and communications Section 3.1 Overview Section 3.2 Device interfacing Section 3.3 Exceptions Section 3.4 Multiprocessors Section 3.5 User-level input and output / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design / 850 Section 3.6 Communications management Section 3.7 Communications networks, interfaces and drivers Section 3.8 Communications software Section 3.9 Communications handling within and above an operating system Section 3.10 Summary Chapter Support for processes Terminology Section 4.1 Use of processes in systems Section 4.2 Processes and processors Section 4.3 Process state Section 4.4 Synchronizing with the hardware: Events and the WAIT operation Section 4.5 The process data structure Section 4.6 Scheduling: General approaches Section 4.7 Scheduling for shared-memory multiprocessors Section 4.8 Process scheduling to meet real-time requirements Section 4.9 Process abstraction and implementation Section 4.10 Operating system structure and placement of processes Section 4.11 Multi-threaded process implementation Section 4.12 Processes in languages, runtime systems and operating systems Section 4.13 Process state in language systems and operating systems Section 4.14 Sequential programs with system calls Section 4.15 Evolution of concurrency in programming languages Section 4.16 Implementation of processes in language systems Section 4.17 Thread package architectures Section 4.18 Java threads and exceptions Section 4.19 Summary Chapter Memory management Section 5.1 Memory management Section 5.2 The memory hierarchy Section 5.3 The address space of a process Section 5.4 Segmented virtual memory Section 5.5 Paged virtual memory Section 5.6 Combining segmentation and paging Section 5.7 Operating system data structures Section 5.8 An example of a memory management unit (MMU) Section 5.9 Memory management in system design Section 5.10 Summary Chapter File management Section 6.1 File management Section 6.2 An overview of filing system functions Section 6.3 File and directory structure Section 6.4 The filing system interface Section 6.5 The filing system implementation Section 6.6 Modern file system design Section 6.7 Network-based file servers Section 6.8 Integrating virtual memory and storage Section 6.9 Summary Chapter Fundamentals of distributed systems Section 7.1 Introduction Section 7.2 Evolution of distributed systems for the workplace Section 7.3 Personal computing Section 7.4 Model and software architecture Section 7.5 Special characteristics of distributed systems Section 7.6 Time in distributed systems Section 7.7 Naming Section 7.8 Mobile users, computers and objects / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 836 / 850 MSNA Multi-Service Network Architecture (Cambridge) MTBF Mean Time Between Failures multi-threaded process Used in systems where a process is the unit of resource allocation and a thread is the unit of execution on a processor The threads of the process share its resources Name service Maps names to locations See also [YPS] NFS SUN's Network File System NIST National Institute of Standards and Technology (US) node A component computer of a distributed system, like host NNTP Network News Transfer Protocol NSA National Security Agency (US) 836 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 837 / 850 NT New Technology NTP Network Time Protocol NVRAM Non-Volatile Random Access Memory Object-based OS An operating system which is object structured internally as well as supporting object-structured software Object-support OS An operating system which is designed to support object-structured software OCC Optimistic Concurrency Control, a method of concurrency control for transactions ODBMS Object-oriented Database Management System ODL Object Definition Language ODMG Object Data Management Group 837 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 838 / 850 ODMG-93 Object-oriented database standard OLTP On-Line Transaction Processing OM Object Manager OMA Object Management Architecture OMG Object Management Group OO Object-oriented OOM Object-oriented Middleware OQL Object Query Language ORB Object Request Broker 838 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 839 / 850 OS Operating System OSF Open Software Foundation: a consortium of vendors led by IBM, DEC and Hewlett-Packard OSI Open Systems Inter-connection OTS Object Transaction Service (OMG-CORBA) PC Personal Computer Program Counter PCC Proof Carrying Code PGP Pretty Good Privacy PIN Personal Identification Number PKI 839 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 840 / 850 Public Key Infrastructure POSIX IEEE Standard 1000.3, Portable Operating System Interface for Computer (UNIX) Environments PSR Processor Status Register PTBR Page Table Base Register pthreads POSIX threads PTT Post, Telephone and Telegraph: the bodies within a country that are licensed to provide public data transmission services QoS Quality of Service RAM Random Access Memory RARA Request–Acknowledge–Reply–Acknowledge 840 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 841 / 850 RFC Request For Comment RFE Return From Exception RFP Request For Proposal RISC Reduced Instruction Set Computer RM Resource Manager RMI (Java's) Remote Method Invocation RMS Rate Monotonic Scheduling – for real-time systems RPC Remote Procedure Call RRA Request–Reply– Acknowledge 841 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 842 / 850 RSA Rivest, Shamir and Adleman algorithm SAP Service Access Point SAR Segmentation And Reassembly SASOS Single Address Space Operation System SCB Stream Control Block SCSI Small Computer System Interface A standard I/O bus server A computer or program designed to perform computations on behalf of other programs, its clients SFID System File IDentifier SGML Standard Generalized Mark-up Language 842 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 843 / 850 SHA Secure Hash Algorithm SHS Secure Hash Standard SIMD Single Instruction Multiple Data SISD Single Instruction Single Data SITA Société Internationale de Télécommunications Aéronautiques site A component computer of a distributed system, like host and node A local internetwork at some geographical locality, such as the Cambridge site SLA Single Logical Action SMB Server Message Block SMT Simultaneous Multi-Threading 843 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 844 / 850 SMTP Simple Mail Transport Protocol SOAP Simple Object Access Protocol socket An abstraction for a communication end-point SPARC Scalable Processor ARChitecture, a RISC architecture SPKI Simple Public Key Infrastructure SQL Structured Query Language SR1 Single-copy Serializability SSL Secure Socket Layer SWIFT Society for Worldwide Inter-bank Financial Telecommunications 844 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 845 / 850 TAL Typed Assembly Language TAS Test-And-Set Tcl-Tk Tool command language and the Tk X11 toolkit TCP Transmission Control Protocol thread A name for a unit of execution on a processor; See also [multi-threaded process] TLB Translation Lookaside Buffer TLS Transport Layer Security TP Transaction Processing TPPC Transaction Processing Performance Council 845 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 846 / 850 TPS Transaction Processing System TS Tuple Space TSO Time-Stamp Ordering, a method of concurrency control for transactions UDP User Datagram Protocol UFID User File IDentifier UI UNIX International: a consortium of vendors led by AT&T UID Unique Identifier ULTRIX Digital Equipment Corporation's version of UNIX UML Unified Modelling Language 846 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 847 / 850 UNC Universal Naming Convention Unicode A 16-bit character encoding scheme UPS Uninterruptable Power Supply – used to maintain a supply of power to equipment when the mains has failed URI Uniform Resource Identifier URL Universal Resource Locator URN Uniform Resource Name USD User-Safe Disk user A human operator of a computer system; compare client UTC Universal Coordinated Time 847 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 848 / 850 VAX A CISC processor proprietary to the Digital Equipment Corporation VC Virtual Circuit VLSI Very Large Scale Integration VM Virtual Machine Virtual Memory VME A backplane bus standard, IEEE standard 104 VMM Virtual Machine Monitor W3C World Wide Web Consortium WAN Wide Area Network WDM 848 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 849 / 850 Windows Driver Model WORM Write Once Read Multiple, an optical storage technology WWW (W3) World Wide Web XDR External Data Representation A standard for representing data types so that they can be exchanged between different types of computer XML eXtensible Markup Language XSL eXtensible Style Language XVM eXtensible Virtual Machine YPS Yellow Pages Service, maps names to attributes 2PL Two-Phase Locking: a method of concurrency control for transactions 849 / 850 Addison Wesley : Operating Systems: Concurrent and Distributed Software Design 850 / 850 2PC Two-Phase Commit: an atomic commitment protocol 3PC Three-Phase Commit: an atomic commitment protocol [ Team LiB ] 850 / 850 ... 2.4 Operating system functions Section 2.5 Operating system invocation and protection Section 2.6 Operating system structure Section 2.7 Object structuring within and supported by operating systems... by) operating systems Real-time systems need specially tailored operating systems Dedicated communications-handling computers need specially tailored operating systems Database management systems... operating system as a major component), how to comprehend existing systems and how to design new systems One can't write certain kinds of system in certain languages above certain operating systems