Tài liệu 53 Image and Video Restoration pdf

A. Murat Tekalp. “Image and Video Restoration.” 2000 CRC Press LLC. <http://www.engnetbase.com>. ImageandVideoRestoration A.MuratTekalp UniversityofRochester 53.1Introduction 53.2Modeling Intra-FrameObservationModel • MultispectralObserva- tionModel • MultiframeObservationModel • Regularization Models 53.3ModelParameterEstimation BlurIdentification • EstimationofRegularizationParameters • EstimationoftheNoiseVariance 53.4Intra-FrameRestoration BasicRegularizedRestorationMethods • RestorationofIm- agesRecordedbyNonlinearSensors • RestorationofImages DegradedbyRandomBlurs • AdaptiveRestorationforRing- ingReduction • BlindRestoration(Deconvolution) • Restora- tionofMultispectralImages • RestorationofSpace-Varying BlurredImages 53.5MultiframeRestorationandSuperresolution MultiframeRestoration • Superresolution • Superresolution withSpace-VaryingRestoration 53.6Conclusion References 53.1 Introduction Digitalimagesandvideo,acquiredbystillcameras,consumercamcorders,orevenbroadcast-quality videocameras,areusuallydegradedbysomeamountofblurandnoise.Inaddition,mostelectronic camerashavelimitedspatialresolutiondeterminedbythecharacteristicsofthesensorarray.Common causesofblurareout-of-focus,relativemotion,andatmosphericturbulence.Noisesourcesinclude filmgrain,thermal,electronic,andquantizationnoise.Further,manyimagesensorsandmediahave knownnonlinearinput-outputcharacteristicswhichcanberepresentedaspointnonlinearities.The goalofimageandvideo(imagesequence)restorationistoestimateeachimage(frameorfield)asit wouldappearwithoutanydegradations,byfirstmodelingthedegradationprocess,andthenapplying aninverseprocedure.Thisisdistinctfromimageenhancementtechniqueswhicharedesignedto manipulateanimageinordertoproducemorepleasingresultstoanobserverwithoutmaking useofparticulardegradationmodels.Ontheotherhand,superresolutionreferstoestimatingan imageataresolutionhigherthanthatoftheimagingsensor.Imagesequencefiltering(restoration andsuperresolution)becomesespeciallyimportantwhenstillimagesfromvideoaredesired.This isbecausetheblurandnoisecanbecomeratherobjectionablewhenobservinga“freeze-frame”, althoughtheymaynotbevisibletothehumaneyeattheusualframerates.Sincemanyvideosignals encounteredinpracticeareinterlaced,weaddressthecasesofbothprogressiveandinterlacedvideo. c  1999byCRCPressLLC Theproblemofimagerestorationhassparkedwidespreadinterestinthesignalprocessingcommu- nityoverthe past20or 30years. Becauseimage restorationis essentiallyanill-posed inverseproblem whichisalsofrequentlyencounteredinvariousotherdisciplinessuchasgeophysics,astronomy,med- ical imaging, and computer vision, the literature that is related to image restoration is abundant. A concisediscussion of early results can befound inthebooks by Andrewsand Hunt[1] andGonzalez and Woods [2]. More recent developments are summarized in the book by Katsaggelos [3], and re- view papers byMeinel[4], Demoment [5], Sezan and Tekalp[6], andKaufmanand Tekalp[7]. Most recently, printinghigh-quality still images from video sources has become an important application for multi-frame restoration and superresolution methods. An in-depth coverage of video filtering methods can be found in the book D igital Video Processing by Tekalp [8]. This chapter summarizes key results in digital image and video restoration. 53.2 Modeling Every image restoration/superresolution algorithm is based on an observation model, which relates the observed degraded image(s) to the desired “ideal” image, and possibly a regularization model, whichconveystheavailablea priori informationabouttheideal image. Thesuccessofimage restora- tion and/or superresolution depends on how good the assumed mathematical models fit the actual application. 53.2.1 Intra-Frame Observation Model Letthe observed and ideal imagesbe sampled on the same 2-D lattice . Then, the observedblurred and noisy image can be modeled as g = s(Df ) + v (53.1) whereg,f ,andv denotevectorsrepresentinglexicographicalorderingofthesamplesoftheobserved image, ideal image, and a particular realization of the additive (random)noise process, respectively. The operator D is called the blur operator. The response of the image sensor to light intensity is represented by thememoryless mapping s(·), which is, in general, nonlinear. (Thisnonlinearity has often been ignored in the literature for algorithmdevelopment.) The blur may be space-invariant or space-variant. Forspace-invariant blurs, D becomesaconvo- lution operator, which has block-Toeplitz structure; and Eq. (53.1) can be expressed, in scalar form, as g ( n 1 ,n 2 ) = s    (m 1 ,m 2 )∈S d d ( m 1 ,m 2 ) f ( n 1 − m 1 ,n 2 − m 2 )   + v ( n 1 ,n 2 ) (53.2) where d(m 1 ,m 2 ) and S d denote the kernel and support of the operator D,respectively. The kernel d(m 1 ,m 2 ) is the impulse response of the blurring system, often called the point spread function (PSF). In case of space-variant blurs, the operator D does not have a particular structure; and the observation equation can be expressed as a superposition summation g ( n 1 ,n 2 ) = s    ( m 1 ,m 2 ) ∈ S d ( n 1 ,n 2 ) d ( n 1 ,n 2 ; m 1 ,m 2 ) f ( m 1 ,m 2 )   + v ( n 1 ,n 2 ) (53.3) where S d (n 1 ,n 2 ) denotes the support of the PSF at the pixel location(n 1 ,n 2 ). The noise isusually approximated by a zero-mean, white Gaussianrandom field whichis additive and independent of the image signal. In fact, it has been generally accepted that more sophisticated noise models do not, in general, lead to significantly improved restorations. c  1999 by CRC Press LLC 53.2.2 Multispectral Observation Model Multispectral images refer to image data with multiple spectral bands that exhibit inter-band cor- relations. An important class of multispectral images are color images with three spectral bands. Supposewe haveK spectralbands, each blurred by possibly adifferentPSF.Then, the vector-matrix model (53.1) can be extended to multispectral modeling as g = Df + v (53.4) where g . =    g 1 . . . g K    , f . =    f 1 . . . f K    , v . =    v 1 . . . v K    denote N 2 K × 1 vectors representing the multispectral observed, ideal, and noise data, respectively, stacked as composite vectors, and D . =    D 11 ··· D 1K . . . . . . . . . D K1 ··· D KK    is an N 2 K × N 2 K matrix representing the multispectral blur operator. In most applications, D is block diagonal, indicating no inter-band blurring. 53.2.3 Multiframe Observation Model Supposeasequenceofblurredandnoisyimagesg k (n 1 ,n 2 ),k = 1, ,L, correspondingtomultiple shots (from different angles) of a static scene sampled on a 2-D lattice or frames (fields) of video sampled (at different times) on a 3-D progressive (interlaced) lattice, is available. Then, we may be able to estimate a higher-resolution “ideal” still image f(m 1 ,m 2 ) (corresponding to one of the observed frames) sampled on a lattice, which has a higher sampling density than that of the input lattice. The main distinction between the multispectral and multiframe observation models is that here the observed images are subject to sub-pixel shifts (motion), possibly space-varying, which makeshigh-resolution reconstruction possible. Inthe case of video,wemayalsomodelblurring due to motion w ithin the aperture time to further sharpen images. To this effect, each observed image (frame or field) can be related to the desired high-resolution ideal still-image through the superposition summation [8] g k ( n 1 ,n 2 ) = s    ( m 1 ,m 2 ) ∈ S d ( n 1 ,n 2 ;k ) d k ( n 1 ,n 2 ; m 1 ,m 2 ) f ( m 1 ,m 2 )   + v k ( n 1 ,n 2 ) (53.5) wherethe supportof the summation overthe high-resolution grid (m 1 ,m 2 ) at a particular obser ved pixel(n 1 ,n 2 ; k) dependsonthemotion trajectoryconnectingthepixel(n 1 ,n 2 ; k) totheidealimage, thesizeofthesupportofthe low-resolutionsensor PSF h a (x 1 ,x 2 ) withrespecttothe hig h resolution grid, and whether there is additional optical (out-of-focus, motion, etc.) blur. Because the relative positionsoflow-andhigh-resolution pixelsingeneralvary byspatialcoordinates,the discretesensor PSFisspace-vary ing. Thesupportofthespace-varyingPSFisindicatedbytheshadedareainFig.53.1, wheretherectangle depictedbysolidlinesshowsthesupport of a low-resolutionpixeloverthe high- resolutionsensorarray. The shaded region corresponds to the area swept bythelow-resolutionpixel due to motion duringthe aperture time [8]. c  1999 by CRC Press LLC FIGURE 53.1: Illustration of the discrete system PSF. Note that the model (53.5) is invalid in case of occlusion. That is, each observed pixel (n 1 ,n 2 ; k) canbe expressedas alinearcombinationofseveraldesiredhig h-resolutionpixels(m 1 ,m 2 ),provided that (n 1 ,n 2 ; k) is connected to (m 1 ,m 2 ) by a motion trajectory. We assume that occlusion regions can be detected a priori using a proper motion estimation/segmentation algorithm. 53.2.4 Regularization Models Restorationisanill-posedproblemwhichcanberegularizedbymodelingcertainaspectsofthedesired “ideal” image. Images can be modeled as either 2-D deterministic sequences or random fields. A priori information about the ideal image can then be used to define hard or soft constraints on the solution. In the deterministic case, images are usually assumed to be members of an appropriate Hilbert space, such as aEuclidean space withthe usual inner productand norm. For example, in the context of set theoretic restoration, the solution can be restricted to be a member of a set consisting of all images satisfying a certain smoothness criterion [9]. On the other hand, constrained least squares (CLS) and Tikhonov-Miller regularization use quadratic functionals to impose smoothness constraints in an optimization framework. In the random case, models have been developed for the pdf of the ideal image in the context of maximuma posteriori(MAP)imagerestoration. Forexample, TrussellandHunt[10]haveproposed a Gaussian distribution with space-varying mean and stationary covariance as a model for the pdf of the image. Geman and Geman [11] proposed a Gibbsdistribution to model thepdf ofthe image. Alternatively, if the image is assumed to be a realization of a homogeneous Gauss-Markov random process,then it canbe statistically modeled through anautoregressive (AR) differenceequation [12] f ( n 1 ,n 2 ) =  ( m 1 ,m 2 ) ∈ S c c ( m 1 ,m 2 ) f ( n 1 − m 1 ,n 2 − m 2 ) + w ( n 1 ,n 2 ) (53.6) where {c(m 1 ,m 2 ) : (m 1 ,m 2 ) ∈ S c } denote the model coefficients, S c is the model support (which may be causal, semi-causal, or non-causal), and w(n 1 ,n 2 ) represents the modeling error which is Gaussian distributed. The model coefficients can be determined such that the modeling error has minimumvariance[12]. Extensionsof(53.6)toinhomogeneous Gauss-Markovfieldswasproposed by Jeng and Woods [13]. 53.3 Model Parameter Estimation Inthissection,wediscussmethodsforestimatingtheparameters thatareinvolvedin theobservation and regularization models for subsequent use in the restoration algorithms. c  1999 by CRC Press LLC 53.3.1 Blur Identification Blur identification refers to estimation of both the support and parameters of the PSF {d(n 1 ,n 2 ) : (n 1 ,n 2 ) ∈ S d }. It is a crucialelement of image restoration because the quality of restored images is highly sensitiveto errors in thePSF [14]. An early approach to blur identification has been based on the assumption that the original scene contains an ideal point source, and that its spread (hence the PSF) can be determined from the observed image. Rosenfeld and Kak [15] show that the PSF can alsobedeterminedfromanidealline source. These approachesare oflimiteduse in practicebecause a scene, in general, doesnot contain an ideal point or line source and the observation noise may not allow the measurement of a useful spread. Models for certain types of PSF can be derived using principles of optics, if the source of the blur is known [7]. For example, out-of-focus and motion blur PSF can be parameterized with afew parameters. Further,theyarecompletelycharacterizedbytheirzerosinthefrequency-domain. Power spectrumandcepstrum(Fouriertransformofthelogarithmofthepowerspectrum)analysismethods have been successfully applied in many cases to identify the location of these zero-crossings [ 16, 17]. Alternatively, Chang et al. [18] proposed a bispectrum analysis method, which is motivated by the fact that bispectrum is not affected, in principle, by the observation noise. However, the bispectral method requires much more data than the method based on the power spectrum. Note that PSFs, whichdonothavezerocrossingsinthe frequencydomain(e.g., GaussianPSFmodeling atmospheric turbulence), cannot be identified by these techniques. Yetanotherapproachforbluridentificationisthemaximumlikelihood(ML)estimationapproach. TheMLapproachaimstofind those parameter values (including, in pr inciple, theobservationnoise variance) that have most likely resulted in the observed image(s). Different implementations of the ML image and blur identification are discussed under a unifying framework [19]. Pavlovi ´ c and Tekalp [20] proposea practical method to find theML estimates of thepar ameters of a PSFbased on a continuous domain image formation model. Inmulti-frameimagerestoration,bluridentificationusingmorethanoneframeatatimebecomes possible. For example, the PSF of a possibly space-varying motion blur can be computed at each pixel from an estimate of the frame-to-frame motion vector at that pixel, provided that the shutter speed of the camera is known [21]. 53.3.2 Estimation of Regularization Parameters Regularizationmodelparameters aim to strikeabalancebetween the fidelityofthe restoredimage to the observed data and its smoothness. Various methods exist to identify regularization parameters, such as parametric pdf models, parametric smoothness constraints, and AR image models. Some restoration methods require the knowledge of the power spectrum of the ideal image, which can be estimated,forexample,fromanARmodeloftheimage. TheARparameterscan,inturn,beestimated from the observed image by a least squares [22] or an ML technique [63]. On the other hand, non-parametric spectral estimation is also possible through the application of periodogram-based methodstoaprototypeimage[69,23]. Inthecontextofmaximumaposteriori(MAP)methods,thea prioripdfisoftenmodeled by a parametricpdf,suchas a Gaussian [ 10] or a Gibbsian [11]. Standard methods for estimating these parameters do not exist. Methods for estimating the regularization parameter in the CLS, Tikhonov-Miller, and related formulations are discussed in [24]. 53.3.3 Estimation of the Noise Variance Almost all restoration algorithms assume that the observation noise is a zero-mean, white random process that is uncorrelated with the image. Then, the noise field is completely characterized by its variance, which is commonly estimated by the sample variance computed over a low-contrast local c  1999 by CRC Press LLC region of the observed image. As we will see in the following section, the noise variance plays an important role in defining constraints used in some of the restoration algorithms. 53.4 Intra-Frame Restoration Westartbyfirstlookingatsomebasicregularizedrestorationstrategies,inthecaseofanLSIblurmodel withnopointwisenonlinearity. Theeffectofthenonlinearmappings(.)isdiscussedinSection53.4.2. Methods that allow PSFs with a random components are summarized in Section 53.4.3. Adaptive restoration for ringing suppression and blind restoration are covered in Sections 53.4.4 and 53.4.5, respectively. Restoration of multispectral images and space-varyingblurred images are addressed in Sections 53.4.6 and 53.4.7, respectively. 53.4.1 Basic Regularized Restoration Methods When the mapping s(.) is ignored, it is evident from Eq. (53.1) that image restoration reduces to solving a set of simultaneous linear equations. If the matrix D is nonsingular (i.e., D −1 exists) and the vector g lies in the column space of D (i.e., there is no observation noise), then there exists a uniquesolutionwhichcanbefoundbydirect inversion(alsoknown as inversefiltering). In practice, however,wealmostalwayshaveanunderdetermined(duetoboundarytruncationproblem[14])and inconsistent(due to observation noise) setof equations. In this case,we resort to a minimum-norm least-squaressolution. A least squares (LS) solution (notunique when the columns of D arelinearly dependent) minimizes the norm-square of the residual J LS (f ) . =||g − Df || 2 (53.7) LS solution(s) with the minimum norm (energy) is (are) generally known as pseudo-inverse solu- tion(s) (PIS). Restorationbypseudo-inversionis oftenill-posed owingtothepresenceofobservationnoise[14]. This follows because the pseudo-inverse operator usually has some very large eigenvalues. For ex- ample, a typical blur transfer function has zeros; and thus, its pseudo-inverse attains very large magnitudes near these singularities as well as at high frequencies. This results in excessive amplifi- cation at these frequencies in the sensor noise. Regularized inversion techniques attempt to roll-off the transfer function of the pseudo-inverse filter at these frequencies to limit noise amplification. It follows that the regularized inverse deviates from the pseudo-inverse at these frequencies which leads to other types of artifacts, generally known as regularization artifacts [14]. Various strategies for regularized inversion (and how to achieve the right amount of regularization) are discussed in the following. Singular-Value Decomposition Method The pseudo-inverse D + can be computed using the singular value decomposition (SVD) [1] D + = R  i=0 λ −1/2 i z i u T i (53.8) where λ i denote the singular values, z i and u i are the eigenvectors of D T D and DD T , respectively, andR isthe rankofD. Clearly,reciprocation ofzerosingular-valuesisavoidedsince thesummation runs to R, the rank of D. Under the assumption that D is block-circulant (corresponding to a circular convolution), the PIS computed through Eq. (53.8) is equivalent to the frequency domain c  1999 by CRC Press LLC pseudo-inverse filtering D + (u, v) =  1/D(u, v) if D(u, v) = 0 0 if D(u, v) = 0 (53.9) where D(u, v) denotes the frequency response of the blur. This is because a block-circulant matrix can be diagonalized by a 2-D discrete Fourier transformation (DFT) [2]. Regularization of the PIS can then be achieved by truncating the singular value expansion (53.8) to eliminate all terms corresponding to small λ i (which are responsible for the noise amplification) at the expense of reduced resolution. Truncation str ategies are generally ad-hoc in the absence of additional information. Iterative Methods (Landweber Iterations) Several image restoration algorithmsare based on variations of the so-called Landweber itera- tions [25, 26, 27, 28, 31, 32] f k+1 = f k + RD T  g − Df k  (53.10) where R is a matrix that controls the rate of convergence of the iterations. There is no general way to select the best C matrix. If the system (53.1) is nonsingular and consistent (hardly ever the case), the iterations (53.10) willconverge to the solution. If, on the otherhand, (53.1) is underdetermined and/or inconsistent, then (53.10) converges to a minimum-norm least squares solution (PIS). The theory of this and other closely related algorithms are discussed by Sanz and Huang [26] and Tom et al. [27]. Kawata and Ichioka [28] are among the first to apply the Landweber-type iterations to image restoration, which they refer to as “reblurring” method. Landweber-type iterative restoration methods can be regularized by appropriately terminating the iterations before convergence, since the closer we are to the pseudo-inverse, the more noise amplification we have. A termination rule can be defined on the basis of the norm of the residual image signal [29]. Alternatively, soft and/or hard constraints can be incorporated into iterations to achieve regularization. Theconstrained iterations can be written as [30, 31] f k+1 = C  f k + RD T  g − Df k   (53.11) whereC is a nonexpansiveconstraint operator, i.e., ||C(f 1 ) − C(f 2 )||≤||f 1 − f 2 ||, to guarantee theconvergenceoftheiterations. ApplicationofEq.(53.11)toimagerestorationhasbeenextensively studied (see [31, 32] and the references therein). Constrained Least Squares Method Regularizedimagerestorationcanbeformulatedasaconstrainedoptimizationproblem,where a functional ||Q(f )|| 2 of the image is minimized subject to the constraint ||g − Df || 2 = σ 2 .Here σ 2 is a constant, which isusually setequal to the variance of the observation noise. The constrained least squares (CLS) estimate minimizes the Lagrangian [34] J CLS (f ) =||Q(f )|| 2 + α  ||g − Df || 2 − σ 2  (53.12) whereα istheLagrangemultiplier. TheoperatorQ ischosensuchthattheminimizationofEq.(53.12) enforces some desired property of the ideal image. For instance, if Q is selected as the Laplacian operator, smoothnessoftherestoredimageisenforced. TheCLSestimatecanbeexpressed,bytaking the derivative of Eq. (53.12) and setting it equal to zero, as [1] ˆ f =  D H D + γ Q H Q  −1 D H g (53.13) c  1999 by CRC Press LLC where H stands for Hermitian (i.e., complex-conjugate and transpose). The parameter γ = 1 α (the regularization parameter) must be such that the constraint ||g − Df || 2 = σ 2 is satisfied. It is often computed iteratively [2]. A sufficient condition for the uniqueness of the CLS solution is that Q −1 exists. For space-invariant blurs, the CLS solution canbe expressed in the frequency domain as [34] ˆ F (u, v) = D ∗ (u, v) |D(u, v)| 2 + γ |L(u, v)| 2 G(u, v) (53.14) where ∗ denotescomplexconjugation. AcloselyrelatedregularizationmethodistheTikhonov-Miller (T-M)regularization[33,35]. T-Mregularization has beenappliedtoimagerestoration[31, 32,36]. Recently, neural network structures implementingthe CLS or T-M image restoration have also been proposed [37, 38]. Linear Minimum Mean Square Error Method The linear minimum mean square error (LMMSE) method finds the linear estimate which minimizes the mean square error between the estimate and ideal image, using up to second order statistics of the ideal image. Assumingthat the ideal imagecan bemodeled bya zero-meanhomoge- neous random fieldand the bluris space-invariant, theLMMSE (Wiener) estimate, in thefrequency domain, is given by [8] ˆ F (u, v) = D ∗ (u, v) |D(u, v)| 2 + σ 2 v /|P (u, v)| 2 G(u, v) (53.15) where σ 2 v is the variance of the observation noise (assumed white) and |P (u, v)| 2 stands for the powerspectrum of the ideal image. The powerspectrumofthe ideal image is usually estimated from a prototype. It can be easily seen that the CLS estimate (53.14) reduces to the Wiener estimate by setting |L(u, v)| 2 = σ 2 v /|P (u, v)| 2 and γ = 1. A Kalman filter determines the causal (up to a fixed lag) LMMSE estimate recursively. It is based on a state-space representation of the image and observ ation models. In the first step of Kalman filtering, a prediction of the present state is formed using an autoregressive (AR) image model and the previous state of the system. In the second step, the predictions are updated on the basis of the observed image data to form the estimate of the present state. Woods and Ingle [39] applied 2-D reduced-updateKalmanfilter (RUKF)toimagerestoration, wherethe updateislimited toonly those state variables in a neighborhood of the present pixel. The main assumption here is that a pixel is insignificantly correlated with pixels outside a certain neighborhood about itself. More recently, a reduced-ordermodelKalmanfiltering(ROMKF),wherethestatevectoristruncatedtoasizethatison the order of the image modelsupport has beenproposed [40]. Other Kalmanfiltering formulations, including higher-dimensional state-space models to reduce the effective size of the state vector, have been reviewed in [7]. The complexity of higher-dimensional state-space model based formulations, however, limits their pr actical use. Maximum A posteriori Probability Method Themaximumaposterioriprobability(MAP)restorationmaximizestheaposterioriprobability density function (pdf) p(f |g), i.e., the likelihood of a realization of f being the ideal image given the observed data g. Through the application of the Bayes rule, we have p(f |g) ∝ p(g|f )p(f ) (53.16) wherep(g|f ) istheconditionalpdf of g givenf (relatedtothe pdf of the noise process)and p( f )is the a priori pdf oftheideal image. Weusuallyassume that the observation noise is Gaussian,leading c  1999 by CRC Press LLC to p(g|f ) = 1 ( 2π ) N/2 |R v | 1/2 exp  −1/2 ( g − Df ) T R −1 v ( g − Df )  (53.17) whereR v denotes the covariancematrix of the noise process. Unlike the LMMSEmethod, theMAP method uses complete pdf information. However, if both the image and noise are assumed to be homogeneous Gaussian random fields, the MAP estimate reduces to the LMMSE estimate, under a linear observation model. Trusselland Hunt [10] used non-stationarya prioripdf models, andproposeda modifiedform of thePicarditerationtosolvethe nonlinear maximizationproblem. They suggestedusingthe variance of the residualsignal as a criterionforconvergence. Geman and Geman[11] proposedusing a Gibbs randomfield modelforthea prioripdfoftheidealimage. Theyusedsimulatedannealingprocedures to maximize Eq. (53.16). It should be noted that the MAP procedures usually require significantly more computation compared to, for example, the CLS or Wiener solutions. Maximum Entropy Method A number of maximum entropy(ME) approacheshavebeen discussed in theliterature,which vary in the way that the ME principle is implemented. A common feature of all these approaches, however, is their computational complexity. Maximizing the entropy enforces smoothness of the restored image. (In the absence of constraints, the entropy is highest for a constant-valued image). One importantaspect of the ME approach is that the nonnegativity constraint isimplicitly imposed on the solution because the entropy is defined in terms of the logarithm of the intensity. Frieden was the first to apply the ME principle to image restoration [41]. In his formulation, the sum of the entropy of the image and noise, given by J ME1 (f ) =−  i f(i)ln f(i)−  i n(i) ln n(i) (53.18) is maximized subject to the constraints n = g − Df (53.19)  i f(i) = K . =  i g(i) (53.20) which enforce fidelity to the dataand a constantsum of pixel intensities. This approach requires the solution of a system of nonlinear equations. The number of equations and unknowns are on the order of the number of pixels in the image. The formulation proposed by Gull and Daniell [42] can be viewed as another form of Tikhonov regularization (or constrained least squares formulation), where the entropy of the image J ME2 (f ) =−  i f(i)ln f(i) (53.21) is the regularization functional. It is maximized subject to the following usual constraints ||g − Df || 2 = σ 2 v (53.22)  i f(i)= K . =  i g(i) (53.23) on the restored image. The optimization problem is solved using an ascent algorithm. Trussell [43] showed that in the case of a prior distribution defined in terms of the image entropy, the MAP solution is identical to the solution obtained by this ME formulation. Other ME formulations were also proposed [44, 45]. Note that all ME methods are nonlinear in nature. c  1999 by CRC Press LLC [...]... multispectral image restoration, when there is no inter-band blurring, may be to ignore the spectral correlations among different bands and restore each band independently, using one of the algorithms discussed above However, algorithms that are optimal for single-band imagery may no longer be so when applied to individual spectral bands For example, restoration of the red, green, and blue bands of a color image. .. and Tekalp, A.M., Efficient multiframe Wiener restoration of blurred and noisy image sequences, IEEE Trans Image Proc., 1(4), 453 476, 1992 [69] Sezan, M.I and Trussell, H.J., Use of a priori knowledge in multispectral image restoration, Proc IEEE ICASSP’89, Glasgow, Scotland, 1429–1432, 1989 [70] Ohyama, N., Yachida, M., Badique, E., Tsujiuchi, J and Honda, T., Least-squares filter for color image restoration, ... high-resolution image from noisy undersampled frames, IEEE Trans Acoust., Speech and Sign Proc., ASSP-38(6), 1013–1027, 1990 [83] Kim, S.P and Su, W.-Y., Recursive high-resolution reconstruction of blurred multiframe images, IEEE Trans Image Proc., 2(4), 534 539 , 1993 [84] Irani, M and Peleg, S., Improving resolution by image registration, CVGIP: Graphical Models and Image Proc., 53, 231–239, 1991 [85] Irani, M and. .. H.J., Prototype image constraints for set-theoretic image restoration, IEEE Trans Sign Proc., 39(10), 2275–2285, 1991 [24] Galatasanos, N.P and Katsaggelos, A.K., Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their relation, IEEE Trans Image Proc., 1(3), 322–336, 1992 [25] Trussell, H.J and Civanlar, M.R., The Landweber iteration and projection... comparison to LSI restoration 53. 4.5 Blind Restoration (Deconvolution) Blind restoration refers to methods that do not require prior identification of the blur and regularization model parameters Two examples are simultaneous identification and restoration of noisy blurred images [63] and image recovery from Fourier phase information [64] Lagendijk et al [63] applied the E-M algorithm to blind image restoration, ... dramatically improve restoration results [54, 55] 53. 4.3 Restoration of Images Degraded by Random Blurs Basic regularized restoration methods (reviewed in Section 53. 4.1) assume that the blur PSF is a deterministic function A more realistic model may be ¯ D=D+ D (53. 26) ¯ where D is the deterministic part (known or estimated) of the blur operator and D stands for the random component Random component may... Proc.: Image Comm., 5, 511–526, 1993 [80] Ur, H and Gross, D., Improved resolution from subpixel shifted pictures, CVGIP: Graphical Models and Image Processing, 54(3), 181–186, 1992 [81] Tsai, R.Y and Huang, T.S., Multiframe image restoration and registration, in Advances in Computer Vision and Image Processing, vol 1, Huang, T.S Ed., Jai Press, Greenwich, CT, 1984, 317–339 [82] Kim, S.P., Bose, N.K and. .. B.R and Kubler, O., Karhunen-Loeve multispectral image restoration, Part I: Theory, IEEE Trans Acoust Speech Sig Proc., ASSP-32(6), 592–599, 1984 [66] Galatsanos, N.P and Chin, R.T., Digital restoration of multi-channel images, IEEE Trans Acoust Speech Sig Proc., ASSP-37(3), 415–421, 1989 [67] Tekalp, A.M and Pavlovi´ , G., Multichannel image modeling and Kalman filtering for multic spectral image restoration, ... Trans Image Proc., 1(4), 496–504, 1992 c 1999 by CRC Press LLC [21] Trussell, H.J and Fogel, S., Identification and restoration of spatially variant motion blurs in sequential images, IEEE Trans Image Proc., 1(1), 123–126, 1992 [22] Kaufman, H., Woods, J.W., Dravida, S and Tekalp, A.M., Estimation and Identification of Two-Dimensional Images, IEEE Trans Aut Cont., 28, 745–756, 1983 [23] Sezan, M.I and. .. The method of FS has also been applied to image restoration [53] 53. 4.2 Restoration of Images Recorded by Nonlinear Sensors Image sensors and media may have nonlinear characteristics that can be modeled by a pointwise (memoryless) nonlinearity s(.) Common examples are photographic film and paper, where the nonlinear relationship between the exposure (intensity) and the silver density deposited on the . Sections 53. 4.4 and 53. 4.5, respectively. Restoration of multispectral images and space-varyingblurred images are addressed in Sections 53. 4.6 and 53. 4.7,. Tekalp. Image and Video Restoration. ” 2000 CRC Press LLC. <http://www.engnetbase.com>. ImageandVideoRestoration A.MuratTekalp UniversityofRochester 53. 1Introduction 53. 2Modeling Intra-FrameObservationModel • MultispectralObserva- tionModel • MultiframeObservationModel • Regularization Models 53. 3ModelParameterEstimation BlurIdentification • EstimationofRegularizationParameters • EstimationoftheNoiseVariance 53. 4Intra-FrameRestoration BasicRegularizedRestorationMethods • RestorationofIm- agesRecordedbyNonlinearSensors • RestorationofImages DegradedbyRandomBlurs • AdaptiveRestorationforRing- ingReduction • BlindRestoration(Deconvolution) • Restora- tionofMultispectralImages • RestorationofSpace-Varying BlurredImages 53. 5MultiframeRestorationandSuperresolution MultiframeRestoration • Superresolution • Superresolution withSpace-VaryingRestoration 53. 6Conclusion References 53. 1