An Introduction to the Analysis of Algorithms - Michael Soltys, World Scientific Publishing (2012)

An Introduction to the Analysis of Algorithms 2nd Edition 8439hc.9789814401159-tp.indd 25/5/12 5:35 PM This page intentionally left blank An Introduction to the Analysis of Algorithms 2nd Edition Michael Soltys McMaster University, Canada World Scientific NEW JERSEY • LONDON 8439hc.9789814401159-tp.indd • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TA I P E I • CHENNAI 25/5/12 5:35 PM Published by World Scientific Publishing Co Pte Ltd Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library AN INTRODUCTION TO THE ANALYSIS OF ALGORITHMS 2nd Edition Copyright © 2012 by World Scientific Publishing Co Pte Ltd All rights reserved This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA In this case permission to photocopy is not required from the publisher ISBN-13 978-981-4401-15-9 ISBN-10 981-4401-15-3 Printed in Singapore Patrick - An Intro to the Analysis (2nd Ed).pmd 3/20/2012, 7:32 PM April 3, 2012 10:24 World Scientific Book - 9in x 6in To my family v soltys˙alg April 3, 2012 10:24 World Scientific Book - 9in x 6in This page intentionally left blank soltys˙alg April 3, 2012 10:24 World Scientific Book - 9in x 6in Preface This book is an introduction to the analysis of algorithms, from the point of view of proving algorithm correctness Our theme is the following: how we argue mathematically, without a burden of excessive formalism, that a given algorithm does what it is supposed to do? And why is this important? In the words of C.A.R Hoare: As far as the fundamental science is concerned, we still certainly not know how to prove programs correct We need a lot of steady progress in this area, which one can foresee, and a lot of breakthroughs where people suddenly find there’s a simple way to something that everybody hitherto has thought to be far too difficult1 Software engineers know many examples of things going terribly wrong because of program errors; their particular favorites are the following two2 The blackout in the American North-East during the summer of 2003 was due to a software bug in an energy management system; an alarm that should have been triggered never went off, leading to a chain of events that climaxed in a cascading blackout The Ariane 5, flight 501, the maiden flight of the rocket in June 4, 1996, ended with an explosion 40 seconds into the flight; this $500 million loss was caused by an overflow in the conversion from a 64-bit floating point number to a 16-bit signed integer While the goal of absolute certainty in program correctness is elusive, we can develop methods and techniques for reducing errors The aim of this book is modest: we want to present an introduction to the analysis of algorithms—the “ideas” behind programs, and show how to prove their correctness From An Interview with C.A.R Hoare, in [Shustek (2009)] two examples come from [van Vliet (2000)], where many more instances of spectacular failures may be found These vii soltys˙alg April 3, 2012 10:24 viii World Scientific Book - 9in x 6in An Introduction to the Analysis of Algorithms The algorithm may be correct, but the implementation itself might be flawed Some syntactical errors in the program implementation may be uncovered by a compiler or translator—which in turn could also be buggy— but there might be other hidden errors The hardware itself might be faulty; the libraries on which the program relies at run time might be unreliable, etc It is the task of the software engineer to write code that works given such a delicate environment, prone to errors Finally, the algorithmic content of a piece of software might be very small; the majority of the lines of code could be the “menial” task of interface programming Thus, the ability to argue correctly about the soundness of an algorithm is only one of many facets of the task at hand, yet an important one, if only for the pedagogical reason of learning to argue rigorously about algorithms We begin this book with a chapter of preliminaries, containing the key ideas of induction and invariance, and the framework of pre/post-conditions and loop invariants We also prove the correctness of some classical algorithms, such as the integer division algorithm, and Euclid’s procedure for computing the greatest common divisor of two numbers We present three standard algorithm design techniques in eponymous chapters: greedy algorithms, dynamic programming and the divide and conquer paradigm We are concerned with correctness of algorithms, rather than, say, efficiency or the underlying data structures For example, in the chapter on the greedy paradigm we explore in depth the idea of a promising partial solution, a powerful technique for proving the correctness of greedy algorithms We include online algorithms and competitive analysis, as well as randomized algorithms with a section on cryptography Algorithms solve problems, and many of the problems in this book fall under the category of optimization problems, whether cost minimization, such as Kruskal’s algorithm for computing minimum cost spanning trees— section 2.1, or profit maximization, such as selecting the most profitable subset of activities—section 4.4 The book is sprinkled with problems; most test the understanding of the material, but many include coding in the Python programming language The reader is expected to learn Python on their own (a great source to so is [Downey (2008)]—the PDF can be downloaded for free from the web) One of the advantages of this programming language is that it is easy to start writing small snippets of code that work—and most of the coding in this book falls into the “small snippet” category The solutions to most problems are included in the “Answers to selected problems” at the end of each chapter The solutions to most of the programming exercises will be soltys˙alg April 3, 2012 10:24 World Scientific Book - 9in x 6in Preface soltys˙alg ix available for download from the author’s web page (a quick Google search will reveal the URL) The intended audience for this book consists of undergraduate students in computer science and mathematics The book is very self-contained: the first chapter, Preliminaries, reviews induction and the invariance principle It also introduces the aforementioned ideas of pre/post-conditions, loop invariants and termination—in other words, it sets the mathematical stage for the rest of the book Not much mathematics is assumed (besides some tame forays into linear algebra and number theory), but a certain penchant for discrete mathematics is considered helpful This book draws on many sources First of all, [Cormen et al (2009)] is a fantastic reference for anyone who is learning algorithms I have also used as reference the elegantly written [Kleinberg and Tardos (2006)] A classic in the field is [Knuth (1997)], and I base my presentation of online algorithms on the material in [Borodin and El-Yaniv (1998)] I have learned greedy algorithms, dynamic programming and logic from Stephen A Cook at the University of Toronto Appendix B, a digest of relations, is based on hand-written lecture slides of Ryszard Janicki No book on algorithms is complete without a short introduction to the “big-Oh” notation Consider functions from N to N We say that g(n) ∈ O(f (n)) if there exist constants c, n0 such that for all n ≥ n0 , g(n) ≤ cf (n), and the little-oh notation, g(n) ∈ o(f (n)), which denotes that limn→∞ g(n)/f (n) = We also say that g(n) ∈ Ω(f (n)) if there exist constants c, n0 such that for all n ≥ n0 , g(n) ≥ cf (n) Finally, we say that g(n) ∈ Θ(f (n)) if it is the case that g(n) ∈ O(f (n)) ∩ Ω(f (n)) The ubiquitous floor and ceil functions are defined, respectively, as follows: bxc = max{n ∈ Z|n ≤ x} and dxe = min{n ∈ Z|n ≥ x} Finally, bxe refers to the “rounding-up” of x, and it is defined as bxe = bx + 21 c We have the usual Boolean connectives: ∧ is “and,” ∨ is “or” and ¬ is “not.” We also use → as Boolean implication, i.e., x → y is logically equivalent to ¬x ∨ y, and ↔ is Boolean equivalence, and α ↔ β expresses ((α → β) ∧ (β → α)) We use “⇒” to abbreviate the word “implies,” i.e., 2|x ⇒ x is even, while “6⇒” abbreviates “does not imply.” .. .An Introduction to the Analysis of Algorithms 2nd Edition 8439hc.9789814401159-tp.indd 25/5/12 5:35 PM This page intentionally left blank An Introduction to the Analysis of Algorithms. .. In this case permission to photocopy is not required from the publisher ISBN-13 97 8-9 8 1-4 40 1-1 5-9 ISBN-10 98 1-4 40 1-1 5-3 Printed in Singapore Patrick - An Intro to the Analysis (2nd Ed).pmd 3/20/2012,... reducing errors The aim of this book is modest: we want to present an introduction to the analysis of algorithms? ? ?the “ideas” behind programs, and show how to prove their correctness From An Interview