Intelligent Vision Systems for Industry Bruce G Batchelor and Paul F Whelan Intelligent Vision Systems for Industry Bruce G Batchelor University of Wales, Cardiff Paul F Whelan Dublin City University © Bruce G Batchelor, Paul F Whelan 2002 To Caroline, Brian and Phyllis Paul Finally, brothers, whatever is true, whatever is noble, whatever is right, whatever is pure, whatever is lovely, whatever is admirable - if anything is excellent or praise worthy - think about such things Letter to Philippians IV For my dear wife, Eleanor, our children, Helen and David, my Mother, Ingrid, and late Father, Ernest Bruce V Preface During the period 1970 - 1990, Japan taught Europe and America the importance of quality in manufactured goods The West learned the hard way: markets were quickly lost to companies whose names were hitherto unknown Many long established and well respected Western companies were unable to meet the challenge and consequently failed to survive Those that did were often faced with difficult years, as their share of the market shrank Most companies in Europe and America have largely come to terms with this and now realise that quality has a vital role in establishing and maintaining customer loyalty In the present climate of opinion, any technology which improves or simply guarantees product quality is welcome Machine vision is a relatively new technology, which has much to offer manufacturing industry in improving product quality and safety, as well as enhancing process efficiency and operational safety Machine vision owes its rising popularity to one major factor: optical sensing is inherently clean, safe (because it a non-contacting technology) and very versatile It is possible to certain things using vision (both human and machine) that no other known sensing method can achieve - imagine trying to sense stains, rust or surface corrosion by any other means Designing a machine vision system is like assembling a jigsaw Among other component technologies machine vision involves the digitisation, manipulation and analysis of images, usually within a computer, a subject which is also covered by the terms image processing and computer vision However, we must emphasise that machine vision, computer vision and image processing are not synonymous None is a subset of either of the others Computer vision is a branch of Computer Science, while machine vision is an area of specialisation within Systems Engineering Notice, in particular, the use of the words "Science" and "Engineering" here Machine vision does not necessarily imply the use of a VI computer; specialised image processing hardware is often used to obtain higher processing speeds than a conventional computer can achieve Machine vision system for industry first received serious attention in the mid1970s, although the proposal that a video system be used for industrial inspection was first made in the 1930s Throughout the early 1980s, the subject developed slowly, with a steady contribution being made by the academic research community, but with only limited industrial interest being shown It seemed in the mid-1980s that there would be a major boost to progress, with serious interest being shown in vision systems by the major American automobile manufacturers Then, came a period of serious disillusionment in the USA, with a large number of small vision companies failing to survive In the late 1980s and early 1990s, interest has grown markedly, due largely to significant progress being made in making fast, dedicated image digitisation and processing hardware In the mid1990s, the role of the general purpose processor is being revised, with the modern RISC processors offering high processing speed on a standard computing platform Throughout this period, academic workers have been steadily proving feasibility in a very wide range of products, representing all of the major branches of manufacturing industry Soon after starting work, machine vision is seen as a confusing jumble of disconnected ideas Industrial image processing systems, which necessarily form part of a vision system, have developed very considerably in the last decade In addition, there have been major advances in other component technologies: image sensors, specialised lighting units, lenses and advisor (CAD) programs, which guide a vision engineer through the initial stages of the design process However, systems integration remains the key factor for the successful design and operation of a machine vision system Having separated the subjects of machine vision and image processing, our first task in this book is to introduce the reader to the basic concepts of image processing, as they apply to our subject (Chapter 2) There are numerous techniques for manipulating images that are either not used, at all, or are used very infrequently in machine vision Wherever there are problems of computational speed, machine vision systems engineers will either seek another solution, or avoid the problem entirely Standard image processing techniques are able to VII achieve some remarkable results but they could not be described as being intelligent By adding a certain level of intelligence, through the integration of image processing software and the AI language Prolog, we are able to certain things that would otherwise be impossible For example, analysing the image of a standard ("analogue") clock, in order to tell the time is one such task that could not be solved using "traditional" image processing methods working on their own We shall, therefore, devote much of the discussion in this book to explaining how intelligence can be provided to image processing, or alternatively, how Artificial Intelligence can be given "eyes" All of this is done with one goal in mind: to improve the prospects for installing machine vision systems in factories Eventually, the pieces fit together However, if one piece is missing, the result is imperfect; system integration is incomplete There is a serious bottleneck in the design of machine vision systems: a high level of skilled man-power is needed to achieve an effective design To illustrate the problem, consider the case of just one organisation, which has over 60000 products That company operates a policy which tries to maintain at least 25% of its sales on products that are not more than years old Simple arithmetic shows that over 10 new product are being introduced by that one company alone, every working day If we were to use a machine vision system on only 1% of those new product lines, we would need to design, build, install and test a new system once every weeks At the moment, the design process typically takes several months and there are simply not enough machine vision engineers to provide that kind of level of support, even for that one company We desperately need more welleducated machine vision systems engineers We also need improved design tools By claiming that machine vision is a flexible technology, without having the manpower available to fulfil that boast is simply foolish Such "overselling" of the virtues of machine vision technology was largely responsible for the collapse in credibility and confidence in the mid-1980s, to which we referred earlier We need both improved educational material and better engineering tools, if we are to meet the challenge that this subject imposes upon us (Chapter 3) Working in such a flexible and potentially beneficial technology carries responsibilities, because it is upon our shoulders that its future development and exploitation lies The user interface is all important, since this will either make a system acceptable or damn it to certain failure For this reason, we shall discuss the prospects for using multi-media interfaces, including, hypertext, speech synthesis, VIII speech recognition and natural language understanding In Chapter 4, we also discuss the use of multi-camera and multi-processor systems, since it is clear that considerable advantage can be obtained from the use of systems that are able to communicate information about what they see to each other System integration is complete; all elements are now in place The result is perfect (machine) vision Industrial machine vision systems would be virtually useless if it were not possible to control external devices, such as lamps, cameras, lenses, robots, etc A good deal of attention will therefore be paid to this topic (Chapter 5) We devote a whole chapter (Chapter 6) to the task of recognising coloured objects The approach we take here is one which has not always found favour with Colour Scientists - but it works! We conclude by discussing several case studies, which may seem to concentrate on unimportant tasks, such as recognising playing cards, telling the time, etc However, all of the applications that we discuss in Chapter reflect industrially important tasks, in a way which allows us to write freely about the technical issues, without violating commercial confidence It is customary in many areas of public writing to use so called gender-neutral phrases, such as "he / she", "his / her” , "s/he" etc We regard these as being both clumsy and counter-productive In this book, we use the words, "he" and "him" in the traditional way, to include both sexes, without claiming precedence for either This is done to improve clarity and to avoid placing women after men, as "he / she" does While many of the ideas the ideas outlined in this book can be implemented on a range of computers, an integrated software package, called PIP (Prolog Image Processing), has been designed specifically for this and runs on Macintosh computers Readers who wish to gain access to PIP should contact Bruce Batchelor or Andrew Jones at the University of Wales, Cardiff An interactive image processing package for the Windows environment, without Prolog, has also been developed This is called MvT (Machine Vision Tutorial) and is available from Paul Whelan at Dublin City University For current information on our research and the status of the PIP and MvT software packages, please consult our WWW sites Bruce G Batchelor Department of Computer Science Paul F Whelan School of Electronic Engineering IX PO Box 916, University of Wales Cardiff, CF2 3XF, Wales UK bruce@cs.cf.ac.uk http://www.cs.cf.ac.uk/ Dublin City University Glasnevin, Dublin Ireland whelanp@eeng.dcu.ie http://www.eeng.dcu.ie/~whelanp/home X XI Acknowledgements It is our pleasure to acknowledge and thank all of the many people who have helped us to formulate our ideas through many fruitful discussions and who have given us so much encouragement These include Frederick Waltz, Mike Snyder, John Chan, Nicky Johns, Derek Molloy, Don Braggins, Robert Churchhouse and Nick Fiddian We also wish to thank the following for their specific help and encouragement: • Clive Spenser, Logic Programming Associates Ltd., London, England, UK for kindly supplying several copies of MacProlog for use in our software development program • Andrew Jones and Ralf Hack for their contribution to Appendix D • Andrew Jones, Ralf Hack, Steve Palmer and Eric Griffiths for their contribution to the development of the PIP software package • Michael Daley for developing the MMB interfacing hardware • Stephen Cooper, formally of Uppsala University, for his design for the interface between MacProlog and Hypercard • Tommy McGowan for his work in developing the Prolog speech synthesis interface • Ken McClannon for his work on machine vision in process control • John Miller and his colleagues for porting the Lighting Advisor to the DOS/Windows platforms • Mark Graves for his Darkroom program which acted as the inspiration for the Machine Vision Tutorial (MvT) software package • Prof Charles McCorkell, Head of the School of Electronic Engineering, Dublin City University, for his support of the Vision Systems Laboratory at Dublin City University and for enabling Bruce Batchelor to visit Dublin on a regular basis • The Commission of the European Union (Agro-Industry Research Programme) for their financial support, which helped to develop the Prolog Image Processing (PIP) software (Grant no AIR2-CT93-1054) • Eolas (Forbairt) and the British Council, whose financial support enabled us to establish the initial link between the University of Wales and Dublin City University Many of the ideas that are outlined in this book were formed during our time working with industry Therefore we would like to acknowledge all our industrial colleagues and thank them for their permission to use some of the material cited in this book Special thanks are due to Nicholas Pinfield of Springer-Verlag for his commitment to this book Chapter - Legends for the Half Tone Images Image 7.5.1 Using filtering and thesholding to analyse a simple clock, which produces a high contrast between the hands and a plain face (a) Original image (b) The image processing sequence [wri, 2•(3•lnb, neg)), rea, sub, thr(140)] was applied to (a) (c) [blb,xor] was applied to (b) (d) [blb, xor] was applied to (c) (e) Only blobs with an area in excess of 200 pixels have been retained (f) [2•skw,2•exw] was applied to (e) Image 7.5.2 The Hough transform applied to simulated and real clock faces (a) Original image of a simple clock (simulated) (b) [enc, thr] applied to (a) (c) [huf, neg, sqr] applied to (a) The sub-sequence [neg, sqr] was used simply to improve visibility of some of the minor detail (d) The two major peaks in (c) were detected automatically and the corresponding lines reconstructed This shows that these peaks correspond to the minute and hour hands (e) The medial axis transformation [ske, mdl] applied to (b) (f) [huf, neg, sqr] applied to (e) Notice that the peaks are better defined than in (c) (g) Same processing as in (d) but this time applied to (f) (h) The Hough transform method was applied to locate the minute hand on the real clock Although the minute hand has been correctly located, the fluorescent tip of the hour hand makes this method unreliable Image 7.5.3 The Polar-to-Cartesian axis transformation [ptc] applied to a simulated and real clock faces (a) Original (grey-scale) image of a simple clock face (b) The operator ptc applied to (a) (c) The command sequence [yxt, rin, csh, wgx, sub, thr, bed, yxt] applied to (b) (d) Original image from a real clock (e) ptc applied to (d) (f) As in (c), processing applied to (d) (g) Another clock design (h) ptc applied to (g) The minute and hour hands are responsible for the two major peaks (i) As in (c), processing applied to (h) Image 7.5.4 (a) Original clock face (b) Circular scan which intersects the hands but avoids the printing on the face (c) Intensity plotted against angular position around the circular scan in (b) (d) The hour “tick” marks can be identified easily, using a circular scan, followed by simple filtering and thresholding The image shown here was obtained using the Cartesian-topolar co-ordinate axis transformation (e) Difference image obtained by subtracting two images of the clock, taken at [9:10] and at [9:37] (f) Multilevel thresholding produces a “cleaner” picture which is easy to analyse (g) Image of the face of the clock, without the hands (obtained using the max operator, applied to two images, taken at [9:10] and [9:37]) (h) Difference image, obtained by subtracting the image in (g) from the (unprocessed) image of the clock taken, at [9:41] (i) Simple thresholding applied to (h) Image 7.6.1 Inspecting bread slices from a lidded tin loaf (a) Silhouette of the slice (b) Axis of minimum second moment [dpa] (c) Slice after reorientation, so that the principal axis is vertical (d) Difference between the minimum enclosing rectangle and (c) (e) A similar result to (d) can be obtained by computing the convex deficiency of (a) [cvd] Image 7.6.2 Analysing the silhouette of a slice of bread from a non-lidded tin loaf (a) Silhouette (b) Outer edge [bed] (c) Centroid and principal axis (axis of minimum second moment) [dpa] (d) Hough transform [bed,huf] (e) Hough transform image enhanced for easier viewing [bed,huf,sqr,neg] (f) Line corresponding to brightest point in (d) Inverse Hough transform applied (g) Three lines obtain by applying the inverse Hough transform to the three principal peaks in (d) (h) Overspill found from lines drawn in (g) (i) Points of high curvature (corners) are highlighted (j) Corners have been used to "cut" the edge contour (k) After removing very small blobs in (j) The three straight segments and curved top of the slice have been isolated Image 7.6.3 Projecting a single light stripe onto a bread roll, using a diode laser.(a) Image obtained in ambient light (b) Improved image obtained in darkened room (c) Light stripe has been reduced to one-pixel wide arc (Note discontinuities) Image 7.6.4 Projecting many (white) light stripes onto a bread roll Image 7.6.5 Projecting coloured light stripes onto a bread roll (a) Original image There were light stripes in a repeating pattern (b) Edge contours derived from (a) Image 7.6.6 Depth maps (a) Depth map of a croissant (b) sca(3) applied to (a) reveals the height contours (c) Intensity plot (d) Cornish pastie (e) Intensity plot (horizontal section) (f) Intensity plot (vertical section) (g) Depth map of a loaf (h) Height contours of (g) (i) Intensity plot of (g) Image 7.6.7 Structured lighting applied to an engineering component (zinc die-casting) (a) Depth map (b) Height profile, across horizontal line in (a) (c) Height contours ... purpose vision systems (Section 2.9 discusses commercial vision systems. ) These systems mainly concentrated on the software aspect of the vision task, and due to the generality of such systems, vision. .. machine vision in inspection, robotic control and automated assembly Machine vision is an umbrella term used to describe many different types of vision systems, but in general, machine vision systems. .. the genuine design and systems problems involved in the implementation of industrial vision systems [SIM-81] One of the reasons for the current growth of machine vision systems in manufacturing