ARCHITECTURES FOR COMPUTER VISION From Algorithm to Chip with Verilog Hong Jeong CuuDuongThanCong.com CuuDuongThanCong.com ARCHITECTURES FOR COMPUTER VISION CuuDuongThanCong.com CuuDuongThanCong.com ARCHITECTURES FOR COMPUTER VISION FROM ALGORITHM TO CHIP WITH VERILOG Hong Jeong Pohang University of Science and Technology, South Korea CuuDuongThanCong.com This edition first published 2014 © 2014 John Wiley & Sons Singapore Pte Ltd Registered office John Wiley & Sons Singapore Pte Ltd., Fusionopolis Walk, #07-01 Solaris South Tower, Singapore 138628 For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com All Rights Reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as expressly permitted by law, without either the prior written permission of the Publisher, or authorization through payment of the appropriate photocopy fee to the Copyright Clearance Center Requests for permission should be addressed to the Publisher, John Wiley & Sons Singapore Pte Ltd., Fusionopolis Walk, #07-01 Solaris South Tower, Singapore 138628, tel: 65-66438000, fax: 65-66438008, email: enquiry@wiley.com Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not be available in electronic books Designations used by companies to distinguish their products are often claimed as trademarks All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners The Publisher is not associated with any product or vendor mentioned in this book This publication is designed to provide accurate and authoritative information in regard to the subject matter covered It is sold on the understanding that the Publisher is not engaged in rendering professional services If professional advice or other expert assistance is required, the services of a competent professional should be sought Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose It is sold on the understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the author shall be liable for damages arising herefrom If professional advice or other expert assistance is required, the services of a competent professional should be sought Library of Congress Cataloging-in-Publication Data Jeong, Hong Architectures for computer vision : from algorithm to chip with Verilog / Hong Jeong pages cm Includes bibliographical references and index ISBN 978-1-118-65918-2 (cloth) Verilog (Computer hardware description language) Computer vision I Title II Title: From algorithm to chip with Verilog TK7885.7.J46 2014 621.39–dc23 2014016398 Set in 9/11pt Times by Aptara Inc., New Delhi, India 2014 CuuDuongThanCong.com Contents About the Author Preface xi xiii Part One VERILOG HDL 1.1 1.2 1.3 1.4 Introduction Computer Architectures for Vision Algorithms for Computer Vision Computing Devices for Vision Design Flow for Vision Architectures Problems References 3 10 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 Verilog HDL, Communication, and Control The Verilog System Hello, World! Modules and Ports UUT and TB Data Types and Operations Assignments Structural-Behavioral Design Elements Tasks and Functions Syntax Summary Simulation-Synthesis Verilog System Tasks and Functions Converting Vision Algorithms into Verilog HDL Codes Design Method for Vision Architecture Communication by Name Reference Synchronous Port Communication Asynchronous Port Communication Packing and Unpacking Module Control Procedural Block Control Problems References 11 11 12 14 17 17 20 22 25 27 29 30 33 36 38 40 44 50 51 55 61 62 CuuDuongThanCong.com Contents vi 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 Processor, Memory, and Array Image Processing System Taxonomy of Algorithms and Architectures Neighborhood Processor BP Processor DP Processor Forward and Backward Processors Frame Buffer and Image Memory Multidimensional Array Queue Stack Linear Systolic Array Problems References 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 Verilog Vision Simulator Vision Simulator Image Format Conversion Line-based Vision Simulator Principle LVSIM Top Module LVSIM IO System LVSIM RAM and Processor Frame-based Vision Simulator Principle FVSIM Top Module FVSIM IO System FVSIM RAM and Processor OpenCV Interface Problems References 63 63 64 66 68 70 73 74 76 77 79 81 87 88 89 90 91 98 100 102 105 109 111 112 116 122 125 128 Part Two VISION PRINCIPLES 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 Energy Function Discrete Labeling Problem MRF Model Energy Function Energy Function Models Free Energy Inference Schemes Learning Methods Structure of the Energy Function Basic Energy Functions Problems References 131 132 132 135 136 138 139 141 142 144 147 147 6.1 6.2 Stereo Vision Camera Systems Camera Matrices 151 151 153 CuuDuongThanCong.com Contents vii 6.3 6.4 6.5 6.6 6.7 6.8 6.9 6.10 6.11 6.12 6.13 Camera Calibration Correspondence Geometry Camera Geometry Scene Geometry Rectification Appearance Models Fundamental Constraints Segment Constraints Constraints in Discrete Space Constraints in Frequency Space Basic Energy Functions Problems References 156 158 162 163 165 167 169 171 172 176 179 180 180 7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10 7.11 7.12 7.13 7.14 7.15 Motion and Vision Modules 3D Motion Direct Motion Estimation Structure from Optical Flow Factorization Method Constraints on the Data Term Continuity Equation The Prior Term Energy Minimization Binocular Motion Segmentation Prior Blur Diameter Blur Diameter and Disparity Surface Normal and Disparity Surface Normal and Blur Diameter Links between Vision Modules Problems References 183 184 187 188 191 192 197 197 201 203 205 205 207 208 209 210 212 213 Part Three VISION ARCHITECTURES 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 8.10 Relaxation for Energy Minimization Euler–Lagrange Equation of the Energy Function Discrete Diffusion and Biharminic Operators SOR Equation Relaxation Equation Relaxation Graph Relaxation Machine Affine Graph Fast Relaxation Machine State Memory of Fast Relaxation Machine Comparison of Relaxation Machines Problems References CuuDuongThanCong.com 219 220 224 225 226 231 234 236 238 240 242 243 244 Contents viii 9.1 9.2 9.3 9.4 9.5 9.6 Dynamic Programming for Energy Minimization DP for Energy Minimization N-best Parallel DP N-best Serial DP Extended DP Hidden Markov Model Inside-Outside Algorithm Problems References 247 247 254 255 256 260 265 273 274 10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 Belief Propagation and Graph Cuts for Energy Minimization Belief in MRF Factor System Belief in Pairwise MRF System BP in Discrete Space BP in Vector Space Flow Network for Energy Function Swap Move Algorithm Expansion Move Algorithm Problems References 277 278 280 283 285 288 291 295 299 300 Part Four VERILOG DESIGN 11 11.1 11.2 11.3 11.4 11.5 11.6 11.7 11.8 11.9 11.10 11.11 11.12 11.13 Relaxation for Stereo Matching Euler–Lagrange Equation Discretization and Iteration Relaxation Algorithm for Stereo Matching Relaxation Machine Overall System IO Circuit Updation Circuit Circuit for the Data Term Circuit for the Differential Circuit for the Neighborhood Functions for Saturation Arithmetic Functions for Minimum Argument Simulation Problems References 305 305 307 308 309 309 312 314 317 319 320 321 323 324 325 326 12 12.1 12.2 12.3 12.4 12.5 12.6 12.7 Dynamic Programming for Stereo Matching Search Space Line Processing Computational Space Energy Equations DP Algorithm Architecture Overall Scheme 327 327 330 331 333 334 337 338 CuuDuongThanCong.com Belief Propagation for Stereo Matching 437 Mi ϕ Mi 14 + Mi 14 x Figure 14.7 Building the result vector The operations are designed by the following code Listing 14.9 The framework: processor.v (9/12) //determining the disparity wire [‘DATA_BITS - 1:0] result; assign result = argmin(mess_av(data, mess_4av (mess_in[0],mess_in[1],mess_in[2],mess_in[3]))); //final message always @(posedge clock) begin ‘ifdef LEFT //left mode res[y+yy][3*(‘WIDTH - - (x+xx))]