FUNDAMENTAL CONCEPTS FOR MEMORYLESS SOURCES
7.1 THE SOURCE CODING PROBLEM
Rate distortion theory is the fundamental theory ofdata compression. It estab lishes the theoreticalminimumaveragenumberofbinarydigitsper sourcesymbol
(or per unit time), i.e., the rate, required to represent asource so that it can be reconstructed tosatisfya givenfidelitycriterion,onewithin thealloweddistortion.
Althoughthe foundationswerelaidby Shannonin 1948,itwasnotuntil 1959that
Shannon fully developed this theory when he established the fundamental theorems for the rate distortion function of a source with respect to a fidelity criterionwhich endowthisfunctionwithitsoperationalsignificance.Initially,rate distortion theory did not receive as much attention as the betterknown channel coding theorytreated inChaps. 2through6.Ultimately,however,interestgrewin expandingthistheoryandintheinsightsitaffords intodata compressionpractice.
Let us now re-examine the general basicblock diagramofacommunication systemdepictedin Fig. 7.1.As alwayswe assumethatwe have nocontroloverthe source, channel, and user.1 We are free to construct only the encoders and de coders. In Chap. 1 we determined the minimum number ofbinary symbols per sourcesymbolsuchthatthe originalsource sequence can beperfectlyreconstructed byobservingthebinarysequence.There wefoundthatShannonsnoiselesscoding
1 Inearlierchapterswereferred tothe user as the destination.Toemphasizethe activeroleof the user ofinformationindeterminingthefidelitymeasure,wenowcallthefinaldestination point theuser.
385
386 SOURCECODING FOR DIGITAL COMMUNICATION
Figure7.1 Communicationsystem model.
theoremgave operationalsignificance to theentropyfunction ofasource. In this chapterwegeneralize thetheoryof noiselesssource codinginChap. 1bydefininga distortion measure and examining the problem of representing source symbols within a givenfidelity criterion.Weshallexaminethe tradeoffbetweentherateof information needed to represent the sourcesymbols and thefidelity with which sourcesymbols can be reconstructed from this information.
Chapters 2 through 6 weredevoted to thechannel codingproblem where we
restrictedourattention toonlythe part of theblockdiagramofFig. 7.1 consisting of the channel encoder, channel, and channel decoder. In these chapters, we showed that channel encoders and decoders can be found which ensurean arbi trarily small error probability for messages transmitted through the channel encoder, channel,andchannel decoder aslongasthe messagerateislessthanthe channelcapacity. For the development ofrate distortion theory, we assumethat ideal channel encoders and decoders are employed so that the link between the source encoderandsourcedecoderisnoiseless asshownin Fig. 7.2.2Thisrequires the assumption that the rate on this link is less than thechannel capacity.
The assumption that source and channel encoders can be considered separately will bejustified on the basis that, in the limit ofarbitrarily complex overall encoders and decoders, no loss in performance results from separating source and channel coding in this way. Representing the source output by a sequence ofbinarydigitsalsoisolates the problem ofsourcerepresentation from that ofinformation transmission. From a practical viewpoint, this separation is desirable since it allows channel encoders and decoders to be designed inde pendently of the actualsourceand user. The source encoderand source decoder in effect adapt the sourceand user to the channel codingsystem.
2Thisis alsoa naturalmodelforstorage ofdatainacomputer.Inthiscase the capacity of the noiselesschannelrepresents the limitedamountofmemoryallowedper sourcesymbol.
r~ ~~i JNoiseless! 1 channel r
i i
Figure7.2 Source coding model.
We begin by defining a source alphabet #, a user alphabet V (sometimes
called the representation alphabet), a distortion measure d(u, r) for each pair of symbols in # and i\ and a statistical characterization of the source.With these definitionsandassumptions,wecan begin ourdiscussionofratedistortion theory.
For this chapter we will consider discrete-time memoryless sources that emit a symbol ubelongingtoalphabet %eachunitof time,say every Tsseconds.Herethe user alphabet i depends on theuser, althoughin manycasesitisthesameasthe
sourcealphabet.Throughoutthis chapter wewill alsoassume asingle-letter dis tortionmeasure between anysourcesymbol uand anyusersymbolv represented by d(u, r) and satisfying
d(u, v)> (7.1.1)
This is sometimesreferredto asacontext-freedistortionmeasuresinceitdoes not depend on the other terms in the sequence ofsourceand usersymbols.
Referring to Fig. 7.2 we now consider the problem ofsource encoding and decoding so as toachieve an average distortion no greater than D. Suppose we considerallpossible encoder-decoder pairs that achieve average distortion D or
less anddenote by $Dthe set ofrates required by these encoder-decoder pairs.
Byrate R, we mean the average number of nats per source symbol3 transmitted over the link between source encoder and source decoder in Fig. 7.2. We now
define theratedistortion function for agiven D as the minimumpossible rate R
necessary to achieve average distortion Dor less. Formally, we define4 R*(D) = min R
Re D
nats/source symbol (7.1.2;
Naturally thisfunctiondepends on the particularsourcestatisticsandthe distor tion measure. This directdefinition of the ratedistortion functiondoes not allow us actually to evaluate R*(D) for variousvalues of D. However, weshall seethat thisdefinition ismeaningfulfor allstationaryergodicdiscrete-timesourceswitha single-letter distortion measure, and for these cases we will showthatR*(D)can be expressed interms ofanaveragemutualinformationfunction,R(D),whichwill be derived in Sec. 7.2.
There is anotherway oflookingat thissameproblem, namelythe distortion rateviewpoint.Suppose weconsiderallsourceencoder-decoderpairs thatrequire fixed rateR and let QR betheset ofall theaverage distortions of theseencoder-
3 RecallthatR =rIn 2nats persymbolwhereristheratemeasuredinbitspersymbol.
4
Strictlyspeaking, the"minimum"hereshouldbe "infimum."
388 SOURCECODING FOR DIGITALCOMMUNICATION
decoderpairs.Then, analogouslytothepreviousdefinition,wedefine the distortion rate function as
D*(R) = min D (7.1.3)
De R
For stationary ergodic sources with single-letter distortion measures, the definitions ofR*(D) and D*(R) yieldequivalent results,theonlydifferencebeing the choiceofdependent and independentvariables.
The study ofrate distortion theory can be divided roughly into three areas.
First, for each kind ofsource and distortion measure, one must find an explicit functionR(D) andprove codingtheorems which showthatitispossible toachieve anaveragedistortionofDorlesswithan encoding anddecodingschemeofrateR
for any rate R> R(D). A converse must also be derived which shows that ifan
encoder-decoderpair has rateR< R(D), thenit isimpossibletoachieve average distortion ofD or less with this pair. These two theorems (direct and converse) establish that R*(D) =
R(D) and give operational significance to the function R(D).Thesecond area concernsthe actualdeterminationof theoptimalattainable performance, and this requires finding the form of the rate distortion function, R*(D), for various sources and distortion measures. Often when this isdifficult, tightbounds on R*(D)can beobtained.Thefinalcategoryofstudydealswiththe application of rate distortion theory to data compression practice. Developing
effective sets ofimplementation techniques for source encodingwhich produces
rates approaching R*(D\ finding meaningful measures of distortion that agree well with users needs, and finding reasonable statistical models for important sources are the threemain problemsassociated withapplication ofthistheoryto practice.
Inthischapter,wedevelopthe basictheoryformemorylesssources,beginning with block codes for discrete memoryless sources in the next section, and its
relationship tochannel coding theory inSec. 7.3.Results ontree codesand trellis
codes are presented in Sec. 7.4. All these results are extended to continuous- amplitude (discrete-time) memoryless sources in Sec. 7.5. Sections 7.6 and 7.7 treattheevaluationof theratedistortionfunctionfordiscretememorylesssources and continuous-amplitude memorylesssources, respectively. Various generaliza tionsof thetheoryarepresentedin Chap. 8, including sourceswith memory and
universal coding concepts.