213 9 LINEAR PROCESSING TECHNIQUES Most discrete image processing computational algorithms are linear in nature; an output image array is produced by a weighted linear combination of elements of an input array. The popularity of linear operations stems from the relative simplicity of spatial linear processing as opposed to spatial nonlinear processing. However, for image processing operations, conventional linear processing is often computation- ally infeasible without efficient computational algorithms because of the large image arrays. This chapter considers indirect computational techniques that permit more efficient linear processing than by conventional methods. 9.1. TRANSFORM DOMAIN PROCESSING Two-dimensional linear transformations have been defined in Section 5.4 in series form as (9.1-1) and defined in vector form as (9.1-2) It will now be demonstrated that such linear transformations can often be computed more efficiently by an indirect computational procedure utilizing two-dimensional unitary transforms than by the direct computation indicated by Eq. 9.1-1 or 9.1-2. Pm 1 m 2 ,() Fn 1 n 2 ,()Tn 1 n 2 m 1 m 2 ,;,() n 2 1 = N 2 ∑ n 1 1 = N 1 ∑ = pTf= Digital Image Processing: PIKS Inside, Third Edition. William K. Pratt Copyright © 2001 John Wiley & Sons, Inc. ISBNs: 0-471-37407-5 (Hardback); 0-471-22132-5 (Electronic) 214 LINEAR PROCESSING TECHNIQUES Figure 9.1-1 is a block diagram of the indirect computation technique called gen- eralized linear filtering (1). In the process, the input array undergoes a two-dimensional unitary transformation, resulting in an array of transform coeffi- cients . Next, a linear combination of these coefficients is taken according to the general relation (9.1-3) where represents the linear filtering transformation function. Finally, an inverse unitary transformation is performed to reconstruct the processed array . If this computational procedure is to be more efficient than direct computation by Eq. 9.1-1, it is necessary that fast computational algorithms exist for the unitary transformation, and also the kernel must be reasonably sparse; that is, it must contain many zero elements. The generalized linear filtering process can also be defined in terms of vector- space computations as shown in Figure 9.1-2. For notational simplicity, let N 1 = N 2 = N and M 1 = M 2 = M. Then the generalized linear filtering process can be described by the equations (9.1-4a) (9.1-4b) (9.1-4c) FIGURE 9.1-1. Direct processing and generalized linear filtering; series formulation. Fn 1 n 2 ,() F u 1 u 2 ,() F ˜ w 1 w 2 ,() F u 1 u 2 ,()T u 1 u 2 w 1 w 2 ,;,() u 2 1 = M 2 ∑ u 1 1 = M 1 ∑ = T u 1 u 2 w 1 w 2 ,;,() Pm 1 m 2 ,() T u 1 u 2 w 1 w 2 ,;,() f ff f A N 2 []f= f ff f ˜ Tf ff f= pA M 2 [] 1 – f ff f ˜ = TRANSFORM DOMAIN PROCESSING 215 where is a unitary transform matrix, T is a linear filtering transform operation, and is a unitary transform matrix. From Eq. 9.1-4, the input and output vectors are related by (9.1-5) Therefore, equating Eqs. 9.1-2 and 9.1-5 yields the relations between T and T given by (9.1-6a) (9.1-6b) If direct processing is employed, computation by Eq. 9.1-2 requires oper- ations, where is a measure of the sparseness of T. With the generalized linear filtering technique, the number of operations required for a given operator are: Forward transform: by direct transformation by fast transformation Filter multiplication: Inverse transform: by direct transformation by fast transformation FIGURE 9.1-2. Direct processing and generalized linear filtering; vector formulation. A N 2 N 2 N 2 × M 2 N 2 × A M 2 M 2 M 2 × p A M 2 [] 1 – T A N 2 []f= TA M 2 [] 1 – T A N 2 []= T A M 2 []T A N 2 [] 1 – = k P M 2 N 2 () 0 k P 1≤≤ N 4 2N 2 2 Nlog k T M 2 N 2 M 4 2M 2 2 Mlog 216 LINEAR PROCESSING TECHNIQUES where is a measure of the sparseness of T . If and direct unitary transform computation is performed, it is obvious that the generalized linear filter- ing concept is not as efficient as direct computation. However, if fast transform algorithms, similar in structure to the fast Fourier transform, are employed, general- ized linear filtering will be more efficient than direct processing if the sparseness index satisfies the inequality (9.1-7) In many applications, T will be sufficiently sparse such that the inequality will be satisfied. In fact, unitary transformation tends to decorrelate the elements of T caus- ing T to be sparse. Also, it is often possible to render the filter matrix sparse by setting small-magnitude elements to zero without seriously affecting computational accuracy (1). In subsequent sections, the structure of superposition and convolution operators is analyzed to determine the feasibility of generalized linear filtering in these appli- cations. 9.2. TRANSFORM DOMAIN SUPERPOSITION The superposition operations discussed in Chapter 7 can often be performed more efficiently by transform domain processing rather than by direct processing. Figure 9.2-1a and b illustrate block diagrams of the computational steps involved in direct finite area or sampled image superposition. In Figure 9.2-1d and e, an alternative form of processing is illustrated in which a unitary transformation operation is per- formed on the data vector f before multiplication by a finite area filter matrix D or sampled image filter matrix B . An inverse transform reconstructs the output vector. From Figure 9.2-1, for finite-area superposition, because (9.2-1a) and (9.2-1b) then clearly the finite-area filter matrix may be expressed as (9.2-2a) 0 k T 1≤≤ k T 1= k T k P 2 M 2 ------- 2 Nlog– 2 N 2 ------ 2 Mlog–< qDf= q A M 2 [] 1 – D A N 2 []f= D A M 2 []DA N 2 [] 1 – = TRANSFORM DOMAIN SUPERPOSITION 217 FIGURE 9.2-1. Data and transform domain superposition. 218 LINEAR PROCESSING TECHNIQUES Similarly, (9.2-2b) If direct finite-area superposition is performed, the required number of computational operations is approximately , where L is the dimension of the impulse response matrix. In this case, the sparseness index of D is (9.2-3a) Direct sampled image superposition requires on the order of operations, and the corresponding sparseness index of B is (9.2-3b) Figure 9.2-1f is a block diagram of a system for performing circulant superposition by transform domain processing. In this case, the input vector k E is the extended data vector, obtained by embedding the input image array in the left cor- ner of a array of zeros and then column scanning the resultant matrix. Follow- ing the same reasoning as above, it is seen that (9.2-4a) and hence, (9.2-4b) As noted in Chapter 7, the equivalent output vector for either finite-area or sampled image superposition can be obtained by an element selection operation of k E . For finite-area superposition, (9.2-5a) and for sampled image superposition (9.2-5b) B A M 2 []BA N 2 [] 1 – = N 2 L 2 k D L N ---- 2 = M 2 L 2 k B L M ----- 2 = Fn 1 n 2 ,() JJ× k E Cf E A J 2 [] 1 – C A J 2 []f== C A J 2 []CA J 2 [] 1 – = qS1 J M() S1 J M() ⊗[]k E = gS2 J M() S2 J M() ⊗[]k E = TRANSFORM DOMAIN SUPERPOSITION 219 Also, the matrix form of the output for finite-area superposition is related to the extended image matrix K E by (9.2-6a) For sampled image superposition, (9.2-6b) The number of computational operations required to obtain k E by transform domain processing is given by the previous analysis for M = N = J. Direct transformation Fast transformation: If C is sparse, many of the filter multiplication operations can be avoided. From the discussion above, it can be seen that the secret to computationally effi- cient superposition is to select a transformation that possesses a fast computational algorithm that results in a relatively sparse transform domain superposition filter matrix. As an example, consider finite-area convolution performed by Fourier domain processing (2,3). Referring to Figure 9.2-1, let (9.2-7) where with for x, y = 1, 2, ., K. Also, let denote the vector representation of the extended spatially invariant impulse response array of Eq. 7.3-2 for J = K. The Fou- rier transform of is denoted as (9.2-8) These transform components are then inserted as the diagonal elements of a matrix (9.2-9) QS1 J M() []K E S1 J M() [] T = GS2 J M() []K E S2 J M() [] T = 3J 4 J 2 4J 2 2 Jlog+ J 2 A K 2 A K A K ⊗= A K 1 K --------W x 1 – ()y 1 – () = W 2πi– K ----------- exp≡ h E K() K 2 1× h E K() h hh h E K() A K 2 []h E K() = K 2 K 2 × H K() diag h hh h E K() 1()…h hh h E K() K 2 (),,[]= 220 LINEAR PROCESSING TECHNIQUES Then, it can be shown, after considerable manipulation, that the Fourier transform domain superposition matrices for finite area and sampled image convolution can be written as (4) (9.2-10) for N = M – L + 1 and (9.2-11) where N = M + L + 1 and (9.2-12a) (9.2-12b) Thus the transform domain convolution operators each consist of a scalar weighting matrix and an interpolation matrix that performs the dimensionality con- version between the - element input vector and the - element output vector. Generally, the interpolation matrix is relatively sparse, and therefore, transform domain superposition is quite efficient. Now, consider circulant area convolution in the transform domain. Following the previous analysis it is found (4) that the circulant area convolution filter matrix reduces to a scalar operator (9.2-13) Thus, as indicated in Eqs. 9.2-10 to 9.2-13, the Fourier domain convolution filter matrices can be expressed in a compact closed form for analysis or operational stor- age. No closed-form expressions have been found for other unitary transforms. Fourier domain convolution is computationally efficient because the convolution operator C is a circulant matrix, and the corresponding filter matrix C CC C is of diagonal form. Actually, as can be seen from Eq. 9.1-6, the Fourier transform basis vectors are eigenvectors of C (5). This result does not hold true for superposition in general, nor for convolution using other unitary transforms. However, in many instances, the filter matrices D , B , and C are relatively sparse, and computational savings can often be achieved by transform domain processing. DH M() P D P D ⊗[]= B P B P B ⊗[] H N() = P D uv,() 1 M --------- 1 W M u 1 – () – L 1 – () – 1 W M u 1 – () – – W N v 1 – () – – ----------------------------------------------------------------= P B uv,() 1 N -------- 1 W N v 1 – () – L 1 – () – 1 W M u 1 – () – – W N v 1 – () – – ----------------------------------------------------------------= H HH H K() PP⊗() N 2 M 2 C J H J() = FAST FOURIER TRANSFORM CONVOLUTION 221 Figure 9.2-2 shows the Fourier and Hadamard domain filter matrices for the three forms of convolution for a one-dimensional input vector and a Gaussian-shaped impulse response (6). As expected, the transform domain representations are much more sparse than the data domain representations. Also, the Fourier domain circulant convolution filter is seen to be of diagonal form. Figure 9.2-3 illustrates the structure of the three convolution matrices for two-dimensional convolution (4). 9.3. FAST FOURIER TRANSFORM CONVOLUTION As noted previously, the equivalent output vector for either finite-area or sampled image convolution can be obtained by an element selection operation on the extended output vector k E for circulant convolution or its matrix counterpart K E . FIGURE 9.2-2. One-dimensional Fourier and Hadamard domain convolution matrices. ( b ) Sampled data convolution Signal Fourier Hadamard ( a ) Finite length convolution ( c ) Circulant convolution 222 LINEAR PROCESSING TECHNIQUES This result, combined with Eq. 9.2-13, leads to a particularly efficient means of con- volution computation indicated by the following steps: 1. Embed the impulse response matrix in the upper left corner of an all-zero matrix, for finite-area convolution or for sampled infinite-area convolution, and take the two-dimensional Fourier trans- form of the extended impulse response matrix, giving FIGURE 9.2-3. Two-dimensional Fourier domain convolution matrices. Spatial domain Fourier domain ( a ) Finite-area convolution ( b ) Sampled image convolution (c) Circulant convolution JJ× JM≥ JN≥