509 16 IMAGE FEATURE EXTRACTION An image feature is a distinguishing primitive characteristic or attribute of an image. Some features are natural in the sense that such features are defined by the visual appearance of an image, while other, artificial features result from specific manipu- lations of an image. Natural features include the luminance of a region of pixels and gray scale textural regions. Image amplitude histograms and spatial frequency spec- tra are examples of artificial features. Image features are of major importance in the isolation of regions of common property within an image (image segmentation) and subsequent identification or labeling of such regions (image classification). Image segmentation is discussed in Chapter 16. References 1 to 4 provide information on image classification tech- niques. This chapter describes several types of image features that have been proposed for image segmentation and classification. Before introducing them, however, methods of evaluating their performance are discussed. 16.1. IMAGE FEATURE EVALUATION There are two quantitative approaches to the evaluation of image features: prototype performance and figure of merit. In the prototype performance approach for image classification, a prototype image with regions (segments) that have been indepen- dently categorized is classified by a classification procedure using various image features to be evaluated. The classification error is then measured for each feature set. The best set of features is, of course, that which results in the least classification error. The prototype performance approach for image segmentation is similar in nature. A prototype image with independently identified regions is segmented by a Digital Image Processing: PIKS Inside, Third Edition. William K. Pratt Copyright © 2001 John Wiley & Sons, Inc. ISBNs: 0-471-37407-5 (Hardback); 0-471-22132-5 (Electronic) 510 IMAGE FEATURE EXTRACTION segmentation procedure using a test set of features. Then, the detected segments are compared to the known segments, and the segmentation error is evaluated. The problems associated with the prototype performance methods of feature evaluation are the integrity of the prototype data and the fact that the performance indication is dependent not only on the quality of the features but also on the classification or seg- mentation ability of the classifier or segmenter. The figure-of-merit approach to feature evaluation involves the establishment of some functional distance measurements between sets of image features such that a large distance implies a low classification error, and vice versa. Faugeras and Pratt (5) have utilized the Bhattacharyya distance (3) figure-of-merit for texture feature evaluation. The method should be extensible for other features as well. The Bhatta- charyya distance (B-distance for simplicity) is a scalar function of the probability densities of features of a pair of classes defined as (16.1-1) where x denotes a vector containing individual image feature measurements with conditional density . It can be shown (3) that the B-distance is related mono- tonically to the Chernoff bound for the probability of classification error using a Bayes classifier. The bound on the error probability is (16.1-2) where represents the a priori class probability. For future reference, the Cher- noff error bound is tabulated in Table 16.1-1 as a function of B-distance for equally likely feature classes. For Gaussian densities, the B-distance becomes (16.1-3) where u i and represent the feature mean vector and the feature covariance matrix of the classes, respectively. Calculation of the B-distance for other densities is gener- ally difficult. Consequently, the B-distance figure of merit is applicable only for Gaussian-distributed feature data, which fortunately is the common case. In prac- tice, features to be evaluated by Eq. 16.1-3 are measured in regions whose class has been determined independently. Sufficient feature measurements need be taken so that the feature mean vector and covariance can be estimated accurately. BS 1 S 2 ,() p x S 1 ()p x S 2 ()[] 12⁄ xd ∫ ln–= p x S i () PPS 1 ()PS 2 ()[] 12⁄ BS 1 S 2 ,()–{}exp≤ PS i () BS 1 S 2 ,() 1 8 u 1 u 2 –() T Σ ΣΣ Σ 1 Σ ΣΣ Σ 2 + 2 1 – u 1 u 2 –() 1 2 1 2 Σ ΣΣ Σ 1 Σ ΣΣ Σ 2 + Σ ΣΣ Σ 1 12⁄ Σ ΣΣ Σ 2 12⁄ ln+= Σ ΣΣ Σ i AMPLITUDE FEATURES 511 TABLE 16.1-1 Relationship of Bhattacharyya Distance and Chernoff Error Bound 16.2. AMPLITUDE FEATURES The most basic of all image features is some measure of image amplitude in terms of luminance, tristimulus value, spectral value, or other units. There are many degrees of freedom in establishing image amplitude features. Image variables such as lumi- nance or tristimulus values may be utilized directly, or alternatively, some linear, nonlinear, or perhaps noninvertible transformation can be performed to generate variables in a new amplitude space. Amplitude measurements may be made at spe- cific image points, [e.g., the amplitude at pixel coordinate , or over a neighborhood centered at ]. For example, the average or mean image amplitude in a pixel neighborhood is given by (16.2-1) where W = 2w + 1. An advantage of a neighborhood, as opposed to a point measure- ment, is a diminishing of noise effects because of the averaging process. A disadvan- tage is that object edges falling within the neighborhood can lead to erroneous measurements. The median of pixels within a neighborhood can be used as an alternative amplitude feature to the mean measurement of Eq. 16.2-1, or as an additional feature. The median is defined to be that pixel amplitude in the window for which one-half of the pixels are equal or smaller in amplitude, and one-half are equal or greater in amplitude. Another useful image amplitude feature is the neighborhood standard deviation, which can be computed as (16.2-2) B Error Bound 1 1.84 × 10 –1 2 6.77 × 10 –2 4 9.16 × 10 –3 6 1.24 × 10 –3 8 1.68 × 10 –4 10 2.27 × 10 –5 12 2.07 × 10 –6 S 1 S 2 ,() Fjk,() jk,() jk,() WW× Mjk,() 1 W 2 Fj mk n+,+() nw –= w ∑ mw –= w ∑ = WW× Sjk,() 1 W Fj mk n+,+()Mj mk n+,+()–[] 2 nw –= w ∑ mw –= w ∑ 12⁄ = 512 IMAGE FEATURE EXTRACTION In the literature, the standard deviation image feature is sometimes called the image dispersion. Figure 16.2-1 shows an original image and the mean, median, and stan- dard deviation of the image computed over a small neighborhood. The mean and standard deviation of Eqs. 16.2-1 and 16.2-2 can be computed indirectly in terms of the histogram of image pixels within a neighborhood. This leads to a class of image amplitude histogram features. Referring to Section 5.7, the first-order probability distribution of the amplitude of a quantized image may be defined as (16.2-3) where denotes the quantized amplitude level for . The first-order his- togram estimate of P(b) is simply FIGURE 16.2-1. Image amplitude features of the washington_ir image. ( a ) Original ( b ) 7 × 7 pyramid mean ( c ) 7 × 7 standard deviation ( d ) 7 × 7 plus median Pb() P R Fjk,()r b =[]= r b 0 bL1–≤≤ AMPLITUDE FEATURES 513 (16.2-4) where M represents the total number of pixels in a neighborhood window centered about , and is the number of pixels of amplitude in the same window. The shape of an image histogram provides many clues as to the character of the image. For example, a narrowly distributed histogram indicates a low-contrast image. A bimodal histogram often suggests that the image contains an object with a narrow amplitude range against a background of differing amplitude. The following measures have been formulated as quantitative shape descriptions of a first-order histogram (6). Mean: (16.2-5) Standard deviation: (16.2-6) Skewness: (16.2-7) Kurtosis: (16.2-8) Energy: (16.2-9) Entropy: (16.2-10) Pb() Nb() M ≈ jk,() Nb() r b S M b≡ bP b() b 0 = L 1 – ∑ = S D σ b ≡ bb–() 2 Pb() b 0 = L 1 – ∑ 12⁄ = S S 1 σ b 3 bb–() 3 Pb() b 0 = L 1 – ∑ = S K 1 σ b 4 bb–() 4 Pb() b 0 = L 1 – ∑ 3–= S N Pb()[] 2 b 0 = L 1 – ∑ = S E Pb() 2 log Pb(){} b 0 = L 1 – ∑ –= 514 IMAGE FEATURE EXTRACTION The factor of 3 inserted in the expression for the Kurtosis measure normalizes S K to zero for a zero-mean, Gaussian-shaped histogram. Another useful histogram shape measure is the histogram mode, which is the pixel amplitude corresponding to the histogram peak (i.e., the most commonly occurring pixel amplitude in the window). If the histogram peak is not unique, the pixel at the peak closest to the mean is usu- ally chosen as the histogram shape descriptor. Second-order histogram features are based on the definition of the joint proba- bility distribution of pairs of pixels. Consider two pixels and that are located at coordinates and , respectively, and, as shown in Figure 16.2-2, are separated by r radial units at an angle with respect to the horizontal axis. The joint distribution of image amplitude values is then expressed as (16.2-11) where and represent quantized pixel amplitude values. As a result of the dis- crete rectilinear representation of an image, the separation parameters may assume only certain discrete values. The histogram estimate of the second-order dis- tribution is (16.2-12) where M is the total number of pixels in the measurement window and denotes the number of occurrences for which and . If the pixel pairs within an image are highly correlated, the entries in will be clustered along the diagonal of the array. Various measures, listed below, have been proposed (6,7) as measures that specify the energy spread about the diagonal of . Autocorrelation: (16.2-13) FIGURE 16.2-2. Relationship of pixel pairs. j,k r q m,n Fjk,() Fmn,() jk,() mn,() θ Pab,()P R Fjk,()r a Fmn,()r b =,=[]= r a r b r θ,() Pab,() Nab,() M ≈ Nab,() Fjk,() r a = Fmn,()r b = Pab,() Pab,() S A abP a b,() b 0 = L 1 – ∑ a 0 = L 1 – ∑ = AMPLITUDE FEATURES 515 Covariance: (16.2-14a) where (16.2-14b) (16.2-14c) Inertia: (16.2-15) Absolute value: (16.2-16) Inverse difference: (16.2-17) Energy: (16.2-18) Entropy: (16.2-19) The utilization of second-order histogram measures for texture analysis is consid- ered in Section 16.6. S C aa–()bb–()Pab,() b 0 = L 1 – ∑ a 0 = L 1 – ∑ = aaPab,() b 0 = L 1 – ∑ a 0 = L 1 – ∑ = bbPab,() b 0 = L 1 – ∑ a 0 = L 1 – ∑ = S I ab–() 2 Pab,() b 0 = L 1 – ∑ a 0 = L 1 – ∑ = S V ab– Pab,() b 0 = L 1 – ∑ a 0 = L 1 – ∑ = S F Pab,() 1 ab–() 2 + b 0 = L 1 – ∑ a 0 = L 1 – ∑ = S G Pab,()[] 2 b 0 = L 1 – ∑ a 0 = L 1 – ∑ = S T Pab,() 2 log Pab,(){} b 0 = L 1 – ∑ a 0 = L 1 – ∑ –= 516 IMAGE FEATURE EXTRACTION 16.3. TRANSFORM COEFFICIENT FEATURES The coefficients of a two-dimensional transform of a luminance image specify the amplitude of the luminance patterns (two-dimensional basis functions) of a trans- form such that the weighted sum of the luminance patterns is identical to the image. By this characterization of a transform, the coefficients may be considered to indi- cate the degree of correspondence of a particular luminance pattern with an image field. If a basis pattern is of the same spatial form as a feature to be detected within the image, image detection can be performed simply by monitoring the value of the transform coefficient. The problem, in practice, is that objects to be detected within an image are often of complex shape and luminance distribution, and hence do not correspond closely to the more primitive luminance patterns of most image trans- forms. Lendaris and Stanley (8) have investigated the application of the continuous two- dimensional Fourier transform of an image, obtained by a coherent optical proces- sor, as a means of image feature extraction. The optical system produces an electric field radiation pattern proportional to (16.3-1) where are the image spatial frequencies. An optical sensor produces an out- put (16.3-2) proportional to the intensity of the radiation pattern. It should be observed that and are unique transform pairs, but is not uniquely related to . For example, does not change if the origin of is shifted. In some applications, the translation invariance of may be a benefit. Angular integration of over the spatial frequency plane produces a spatial frequency feature that is invariant to translation and rotation. Representing in polar form, this feature is defined as (16.3-3) where and . Invariance to changes in scale is an attribute of the feature (16.3-4) F ω x ω y ,() Fxy,() i ω x x ω y y+()–{}exp xdyd ∞ – ∞ ∫ ∞ – ∞ ∫ = ω x ω y ,() M ω x ω y ,()F ω x ω y ,() 2 = F ω x ω y ,()Fxy,() M ω x ω y ,() Fxy,() M ω x ω y ,() Fxy,() M ω x ω y ,() M ω x ω y ,() M ω x ω y ,() N ρ() M ρθ,()θd 0 2π ∫ = θ arc ω x ω y ⁄{}tan= ρ 2 ω x 2 ω y 2 += P θ() M ρθ,()ρd 0 ∞ ∫ = TRANSFORM COEFFICIENT FEATURES 517 The Fourier domain intensity pattern is normally examined in specific regions to isolate image features. As an example, Figure 16.3-1 defines regions for the following Fourier features: Horizontal slit: (16.3-5) Vertical slit: (16.3-6) Ring: (16.3-7) Sector: (16.3-8) FIGURE 16.3-1. Fourier transform feature masks. M ω x ω y ,() S 1 m() M ω x ω y ,()ω x ω y dd ω y m() ω y m 1 + () ∫ ∞ – ∞ ∫ = S 2 m() M ω x ω y ,()ω x ω y dd ∞ – ∞ ∫ ω x m() ω x m 1 + () ∫ = S 3 m() M ρθ,()ρθdd 0 2π ∫ ρ m() ρ m 1 + () ∫ = S 4 m() M ρθ,()ρθdd θ m() θ m 1 + () ∫ 0 ∞ ∫ = 518 IMAGE FEATURE EXTRACTION For a discrete image array , the discrete Fourier transform (16.3-9) FIGURE 16.3-2. Discrete Fourier spectra of objects; log magnitude displays. ( a ) Rectangle ( b ) Rectangle transform ( c ) Ellipse ( d ) Ellipse transform ( e ) Triangle ( f ) Triangle transform Fjk,() F uv,() 1 N Fjk,() 2πi– N uxvy+() exp k 0 = N 1 – ∑ j 0 = N 1 – ∑ =