P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 Real-Time Image and Video Processing: From Research to Reality P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 Copyright © 2006 by Morgan & Claypool All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews, without the prior permission of the publisher. Real-Time Image and Video Processing: From Research to Reality Nasser Kehtarnavaz and Mark Gamadia www.morganclaypool.com 1598290525 paper Kehtarnavaz/Gamadia Real-Time Image and Video Processing 1598290533 ebook Kehtarnavaz/Gamadia Real-Time Image and Video Processing DOI 10.2200/S00021ED1V01Y200604IVM005 A Publication in the Morgan & Claypool Publishers’ SYNTHESIS LECTURES ON IMAGE, VIDEO & MULTIMEDIA PROCESSING Lecture #5 First Edition 10987654321 Printed in the United States of America P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 Real-Time Image and Video Processing: From Research to Reality Nasser Kehtarnavaz and Mark Gamadia University of Texas at Dallas, USA SYNTHESIS LECTURES ON IMAGE, VIDEO & MULTIMEDIA PROCESSING #5 M &C Morgan & Claypool Publishers P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 iv ABSTRACT This book presents an overview of the guidelines and strategies for transitioning an image or video processing algorithm from a research environment into a real-time constrained en- vironment. Such guidelines and strategies are scattered in the literature of various disciplines including image processing, computer engineering, and software engineering, and thus have not previously appeared in one place. By bringing these strategies into one place, the book is intended to serve the greater community of researchers, practicing engineers, industrial profes- sionals, who are interested in taking an image or video processing algorithm from a research environment to an actual real-time implementation on a resource constrained hardware plat- form. These strategies consist of algorithm simplifications, hardware architectures, and software methods. Throughout the book, carefully selected representative examples from the literature are presented to illustrate thediscussed concepts. After reading the book, the readers are exposed to a wide variety of techniques and tools, which they can then employ for designing a real-time image or video processing system of interest. KEYWORDS Real-time image and video processing, Real-time implementation strategies, Algorithmic sim- plifications for real-time image and video processing, Hardware platforms for real-time image and video processing, Software methods for real-time image and video processing P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 v Contents 1. Real-Time Image and Video Processing Concepts . 1 1.1 Introduction 1 1.2 Parallelism in Image/Video Processing Operations 1 1.2.1 Low-Level Operations 3 1.2.2 Intermediate-Level Operations 4 1.2.3 High-Level Operations 5 1.2.4 Matrix–Vector Operations 5 1.3 Diversity of Operations in Image/Video Processing 5 1.4 Definition of “Real-Time” 6 1.4.1 Real-time in Perceptual Sense 6 1.4.2 Real-time in Software Engineering Sense 7 1.4.3 Real-time in Signal Processing Sense 8 1.4.4 Misinterpretation of Concept of Real-time 8 1.4.5 Challenges in Real-time Image/Video Processing 9 1.5 Historical Perspective 9 1.5.1 History of Image/Video Processing Hardware Platforms 9 1.5.2 Growth in Applications of Real-time Image/Video Processing 11 1.6 Trade-Off Decisions 11 1.7 Chapter Breakdown 12 2. Algorithm Simplification Strategies 15 2.1 Introduction 15 2.2 Core Simplification Concepts 16 2.2.1 Reduction in Number of Operations 16 2.2.2 Reduction in Amount of Data 18 2.2.3 Simplified Algorithms. . . 19 2.3 Examples of Simplifications 19 2.3.1 Reduction in Number of Operations 20 2.3.2 Reduction of Data 24 2.3.3 Simple Algorithms 29 2.4 Summary 31 P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 vi CONTENTS 3. Hardware Platforms for Real-Time Image and Video Processing 33 3.1 Introduction 33 3.2 Essential Hardware Architecture features . . . 34 3.3 Overview of Currently Available processors . . 35 3.3.1 Digital Signal Processors . . . 35 3.3.2 Field Programmable Gate Arrays 37 3.3.3 Multicore Embedded System-on-Chip 38 3.3.4 General-Purpose Processors . . 39 3.3.5 Graphics Processing Unit. . . 40 3.4 Example Systems 41 3.4.1 DSP-Based Systems . . . 41 3.4.2 FPGA-Based Systems 43 3.4.3 Hybrid Systems 48 3.4.4 GPU-Based Systems 50 3.4.5 PC-Based Systems 52 3.5 Revolutionary Technologies 53 3.6 Summary 54 4. Software Methods for Real-Time Image and Video Processing 55 4.1 Introduction 55 4.2 Elements of Software Platform . . . 55 4.2.1 Programming Languages . . . 56 4.2.2 Software Architecture Design . . 60 4.2.3 Real-time Operating System 60 4.3 Memory Management 61 4.3.1 Memory Performance Gap . 61 4.3.2 Memory Hierarchy 61 4.3.3 Organization of Image Data in Memory 62 4.3.4 Spatial Locality and Cache Hits/Misses 63 4.3.5 Memory Optimization Strategies 63 4.4 Software Optimization 66 4.4.1 Profiling 66 4.4.2 Compiler Optimization Levels 66 4.4.3 Fixed-Point Versus Floating-Point Computations and Numerical Issues 67 4.4.4 Optimized Software Libraries 69 P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 CONTENTS vii 4.4.5 Precompute Information. . 69 4.4.6 Subroutines Versus In-Line Code 69 4.4.7 Branch Predication 70 4.4.8 Loop Transformations 70 4.4.9 Packed Data Processing 71 4.5 Examples of Software Methods 71 4.5.1 Software Design 72 4.5.2 Memory Management . . . 74 4.5.3 Software Optimization 76 4.6 Summary 78 5. The Road Map 81 5.1 Recommended Road Map 81 5.2 Epilog 82 References 83 About the Authors 97 P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 ix Preface The relentless progression of Moore’s Law coupled with the establishment of international standards for digital multimedia has served as the catalyst behind the ubiquitous dissemination of digital information in our everyday lives in the form of digital audio, digital images, and more recently, digital video. Nowadays, entire music libraries can be stored on portable MP3 players, allowing listening to favorite songs wherever one goes. Digital cameras and camera-equipped cell phones are enabling easy capturing, storing, and sharing valuable moments through digital images and video. Set-top boxes are being used to pause, record, and stream live television signal over broadband networks to different locations, while smart camera systems are providing peace of mind through intelligent scene surveillance. Of course, all of these innovative multimedia products would not have materialized without efficient, optimized implementations of practical signal and image processing algorithms on embedded platforms, where constraints are placed not only on system size, cost, and power consumption, but also on the interval of time in which processed information must be made available. While digital audio processing presents its own implementation difficulties, the processing of digital images and video is challenging primarily due to the fact that vast amounts of data must be processed on platforms having limited computational resources, memory, and power consumption. Another challenge is that the al- gorithms for processing digital images and video are developed and prototyped on desktop PCs or workstations, which are considered to be, in contrast to portable embedded devices, resource unlimited platforms. Adding to this the fact that the vast majority of algorithms developed to process digital images and video are quite computationally intensive, one requires to resort to specialized processors, judicious trade-off decisions to reach an accepted solution, or even aban- doning a complex algorithm for a simpler, less computationally complex algorithm. Noting that there are many competing hardware platforms with their own advantages and disadvantages, it is rather difficult to navigate the road from research to reality without some guidelines. Real- Time Image and Video Processing: From Research to Reality is intended to provide such guidelines and help bridge the gap between the theory and the practice of image and video processing by providing a broad overview of proven algorithmic, hardware, software tools and strategies. This book is intended to serve the greater community of researchers, practicing engineers, and industrial professionals who deal with designing image and video processing systems and are asked to satisfy strict system design constraints on performance, cost, and power consumption. P1: IML/FFX P2: IML/FFX QC: IML/FFX T1: IML MOBK019-FM MOBK019-Kehtarnavaz.cls May 9, 2006 16:51 [...]... attached to the maximum bounded response time into what is known as hard realtime, firm real- time, and soft real- time Hard real- time refers to the case where if a real- time deadline is missed, it is deemed to be a complete failure Firm real- time refers to the case in which a certain amount of missed real- time deadlines is acceptable and does not constitute failure Finally, soft real- time refers to the... by the term real- time, ” an elusive term that is often used to describe a wide variety of image/ video processing systems and algorithms From the literature, it can be derived that there are three main interpretations of the concept of real- time, ” namely real- time in the perceptual sense, real- time in the software engineering sense, and real- time in the signal processing sense 1.4.1 Real- time in Perceptual... needs to be performed across all the pixels in the input image 3 P1: IML/FFX P2: IML/FFX MOBK01 9-0 1 MOBK019-Kehtarnavaz.cls 4 QC: IML/FFX T1: IML May 9, 2006 16:51 REAL- TIME IMAGE AND VIDEO PROCESSING: FROM RESEARCH TO REALITY (a) (b) Input image Output image Input image Output image (c) Input image Output image FIGURE 1.2: Parallelism in low-level (a) point, (b) neighborhood, and (c) global image/ video. .. IML/FFX P2: IML/FFX MOBK01 9-0 1 MOBK019-Kehtarnavaz.cls 12 QC: IML/FFX T1: IML May 9, 2006 16:51 REAL- TIME IMAGE AND VIDEO PROCESSING: FROM RESEARCH TO REALITY adaptability, flexibility, and total system cost are important aspects of a design and in practice, one usually has to trade one aspect for another [35] In real- time image/ video processing systems, speed is critical and thus trade-offs such as speed... algorithm in real- time P1: IML/FFX P2: IML/FFX QC: IML/FFX MOBK01 9-0 1 MOBK019-Kehtarnavaz.cls T1: IML May 9, 2006 16:51 REAL- TIME IMAGE AND VIDEO PROCESSING CONCEPTS 1.4.5 Challenges in Real- time Image/ Video Processing Bearing in mind the above argument, developing a real- time image/ video processing system can be quite a challenge The solution often ends up as some combination of hardware and software... computation of local histogram statistics was performed recursively and then generalized to N-dimensional images This provided a remarkable speedup, 63 ms as compared to 2 min, leading to real- time color histogram matching P1: IML/FFX P2: IML/FFX MOBK01 9-0 2 MOBK019-Kehtarnavaz.cls 22 QC: IML/FFX T1: IML May 9, 2006 16:52 REAL- TIME IMAGE AND VIDEO PROCESSING: FROM RESEARCH TO REALITY 2.3.1.2 Reduction... strategies to achieve algorithmic simplifications along with relevant examples from the literature to exhibit successful applications of the strategies P1: IML/FFX P2: IML/FFX MOBK01 9-0 2 MOBK019-Kehtarnavaz.cls 16 QC: IML/FFX T1: IML May 9, 2006 16:52 REAL- TIME IMAGE AND VIDEO PROCESSING: FROM RESEARCH TO REALITY 2.2 CORE SIMPLIFICATION CONCEPTS Examining the literature on real- time image and video processing. .. hardware, and software tools to properly transition algorithms from research to reality 1.6 TRADE-OFF DECISIONS Designing real- time image/ video processing systems is a challenging task indeed Given a fixed amount of hardware, certain design trade-offs will most certainly have to be made during the course of transitioning an algorithm from a research development environment to an actual real- time operation... been to compile in one place the guidelines one needs to know in order to take an algorithm from a research environment into an actual real- time constrained implementation Real- time image and video processing has long played a key role in industrial inspection systems and will continue to do so while its domain is being expanded into multimedia-based consumer electronics products, such as digital and. .. operations at the back end [1] A typical image/ video processing chain combines the three levels of operations into a complete system, as shown in Figure 1.3, where row (a) shows the image/ video processing chain, and 5 P1: IML/FFX P2: IML/FFX MOBK01 9-0 1 MOBK019-Kehtarnavaz.cls 6 QC: IML/FFX T1: IML May 9, 2006 16:51 REAL- TIME IMAGE AND VIDEO PROCESSING: FROM RESEARCH TO REALITY capture: reduction detection . publisher. Real- Time Image and Video Processing: From Research to Reality Nasser Kehtarnavaz and Mark Gamadia www.morganclaypool.com 1598290525 paper Kehtarnavaz /Gamadia Real- Time Image and Video Processing 1598290533. real- time, ” namely real- time in the percep- tual sense, real- time in the software engineering sense, and real- time in the signal processing sense. 1.4.1 Real- time in Perceptual Sense Real- time. advantages and disadvantages, it is rather difficult to navigate the road from research to reality without some guidelines. Real- Time Image and Video Processing: From Research to Reality is intended to