FUNDAMENTAL CONCEPTS FOR MEMORYLESS SOURCES
7.2 DISCRETE MEMORYLESS SOURCES BLOCK CODES
In thissectionandthe followingtwosections we shall restrictour studyofsource coding with a fidelity criterion to the case of a discretememoryless source with alphabet ^ = {al5 a2,...,aA] and letter probabilities Q(al ), Q(a2\ ..., Q(aA).
Then in each unit of time, say Ts seconds, the source emits a symbol u ety according to these probabilities and independent of past or future outputs. The
user alphabet is denoted V {bl9 b2, ..., bB}and there is a nonnegative distor-
tion measure d(u, r) defined for each pair (w, v) in ^ x y. Since the alphabet is
finite,we may assume that there exists a finite numberd such that for all w6^
and ve i^
<
d(u, v)< d < oo (7.2.1)
Inthissection,weconsiderblock source codingwheresequencesofNsource symbolswill berepresented bysequences ofN usersymbols.Theaverageamount
ofdistortion between N source output symbols u= (ut, w2, ..., UN) and N rep
resentation symbols v=
(i-j, v2, ..., VN) is given by un, v
n) (7.2.:
Let $ = {Y!, v2, ...,VM}beasetofMrepresentationsequencesofNusersymbols each. This is called a block source code ofsize M and block length N,and each sequence in M is called a codeword. Code $ will be used to encode a source sequence u e#v by choosing thecodeword v e $ which minimizes dv(u, v). We
denote this minimum by
d(uIM} = min d
v(u, v) (7.2.3)
\e
and we define in a natural way the average distortion achieved with code $ as
where
(7.2.4)
(7.2.5) n=1
as follows from the assumption that the source is memoryless.
Each N units of time when N source symbols are collected by the source encoder, theencoderselectsacodewordaccordingtotheminimumdistortionrule (7.2.3). The index of the selected codeword is then transmitted over the link
between source encoder and source decoder.Thesourcedecoder then selectsthe codeword with this transmitted index and presents it to the user. This block source coding system is shown in Fig. 7.3. Since, for each sequence ofN source symbols, one ofM indices is transmitted over the noiseless channel between the encoder and decoder (which can be represented by a distinct binary sequence whose length is the smallest integer greater than orequal to log M)the required
Figure7.3 Blocksourcecodingsystem.
390 SOURCE CODINGFORDIGITAL COMMUNICATION
rate5 is R = (In M)/N nats per source symbol. In the following wewill refer to code ^ as a block code ofblock length N and rate R.
For a given fidelitycriterionD,weare interestedin determininghowsmalla rate .Rcan be achieved when d(&)< D. Unfortunately, foranygiven code38, the averagedistortiond($)isgenerallydifficult toevaluate. Indeed, theevaluationof d($) is analogous to the evaluation of error probabilities for specific codes in channel coding. Just aswe did in channel coding, we nowuse ensembleaverage codingargumentsto getaroundthis difficultyand show howwell theaboveblock source coding system can perform. Thus we proceed to prove codingtheorems that establish the theoretically minimumpossible rate Rfora givendistortionD.
Let us first introduce an arbitrary conditional probability distribution {P(v|u):vei^, u eW}.
6 For sequences u eWN and v e i^N, we assume condi tional independence in this distribution so that
Pw(v|u)= flP(^k) (7.2.6)
n=l
Corresponding marginal probabilities are thus given by
= II P(.) (7.2.7)
n=l where
P(v)= ZP(v\u)Q(u)
u
Similarly, applying Bayes rule, we have the backward conditional probabilities
= ne(.K) (7.2.8)
n=1
where
We attach no physical significance to the conditional probabilities {P(v|u):v e 1f, u e U\ but merely use them as a convenient tool for deriving bounds on theaverage distortionwhen using acode ^ofsizeM andblock length N.
5 Misusuallytakentobeapowerof2;however, evenifthisisnotthecase,wemaycombinethe
transmission of several indices intoonelargerchannelcodeword andthusapproachRas closely as desired.
6Weshalldenoteallprobability distributionanddensity functions associated with sourcecoding bycapitalletters.
Recall from (7.2.4) thatthe averagedistortion achieved usingcode M is
rfW = I&r(u)rf(u|^) (7.2.4) u
Since
ijyv|u)=i
V
we can also write this as
4) = I X 2/yv|u)d( |M) (7.2.9)
U V
Here v e VN is not a codeword but only a dummy variable ofsummation. We
now split the summation over u and v into two disjoint regions bydefining the indicator function
/ \f\ J I \ J
Since (1 -
O) + O = 1, we have
d() = Z Zftv(u)p]v(vlu)d(uWt1 ~ <&(u, v;4
U V
+ Z Z 6]v(u)^N(v|u)d(u|^)a>(u, v; ^) (7.2.11)
U V
Using the inequality, which results from definition (7.2.10) d(u| )[l -
O(u, v;^)] <
4v(u, v) (7.2.12)
in the firstsummation and using the inequality, which followsfrom (7.2.1) d(u\&) = min dN(u, v) < d (7.2.13)
in the second summation in (7.2.11), we obtain the bound
Z Z &v(u)p N(v|)4vK v)+ d Z Z GN()^N(VI"Wu, v; ^) (7.2.14)
The first term in this bound simplifies to
I Z ejyvIu)^(u,v) = z Z o^(^ w 4 i ^K,
u v u v ^ n=l
= D(P) (7.2.15)
To bound thesecondterm,we need toapply an ensembleaverage argument.
In particular,weconsider an ensembleofblock codesofsize M andblock length
392 SOURCECODINGFOR DIGITAL COMMUNICATION
N where # ={YJ,v2, ...,VM} is assigned theproduct measure
Af
m=l
(7.2.16)
where PN(v)isdefinedby(7.2.7)andisthemarginaldistributioncorrespondingto the given conditional probability distribution {P(v\u): ve_i^, uetft}. Averages overthis code ensemblewillbe denoted by an upperbar ( ). The desired bound
for theensemble average ofthe second term in (7.2.14) is given by the following lemma.
Lemma 7.2.1
where
E(R; p,P)=- P R + E (p, P)
R = InM N
(7.2.17)
(7.2.18) l+p
PROOF Using the Holder inequality (see App. 3 A), we have, for any
-1 <
p <0,
-p
(7.2.19)
since it followsfrom definition (7.2.10) that <I>
1/p=<I>. Averaging this over
thecode ensemble andapplyingtheJenseninequalityoverthesame rangeof p yields
Si I^XM-I )""*" Z ^(vWu, v; &
l+p
l+p
v
-P
(7.2.20)
The second bracketed term above is simply
^\M)
= Pr
{dN(u, v) < min (^(u, v^, dN(u, v2
), ..., dN(u, \M))}
1
M + 1
(7.2.21) sincethecode $hastheproductmeasuregivenin(7.2.16)andthus, forafixed u, each of the random variables dN(u, v), dN(u, vx ), ..., dN(u, VM), which are independentand identically distributed,has thesameprobability ofbeingthe minimum. Using (7.2.21) in (7.2.20), we have
I+P
np(OCkk)
:
n=1
1+p
1+p
1+P
1+P
(7.2.22) Let us briefly examine the behavior of this bound for various parameter values. As statedin theabove lemma, thebound givenin (7.2.17)applies forallp
in the range 1 <
p < and for any choice of the conditional probability
{P(v\u) :vei^, u e W}. The expression E(R;p, P) is identical to the random codingexponentin channel coding theory introduced inSec. 3.1.The onlydiffer ence is that here the parameter p ranges between 1 and while for channel coding this parameter ranges from to 1. Also, here we can pick an arbitrary conditional probability {P(v\u)} which influencesboth P(v) and Q(u\v),while in thechannelrandom codingexponentthechannelconditional probabilityisfixed and only the distribution of the code ensemble is allowed to change. In the following lemmas, we draw upon our earlierexamination of the random coding bound for channelcoding. Here E (p, P) isa form of the Gallagerfunction first defined in (3.1.18).
394 SOURCE CODINGFOR DIGITAL COMMUNICATION
Lemma7.2.2
u r
has the following properties for - 1 <
p <0:
E (P,P) <
>,P)
>
/(P)> (7.2.24)
<p
>,o
dp*
E (0, P)=
>,P)
dp where7
/(P) = 11 Q(u)P(vI")
In
P ~^ (7.2.25)
V / L^ LI <G*\ / V I /
Pliii
is the usual average mutual information function.
PROOF This lemmaisthesame as Lemma3.2.1. ItsproofisgiveninApp. 3A.
Since we are free to choose any p in the interval 1 <
p < 0, the bound in
Lemma 7.2.1 can be minimized with respect to p or, equivalently, the negative exponent can be maximized. We first establish that the minimum always corre sponds to anegative exponent, and then show how to determine its value.
Lemma 7.2.3
max E(R;p, P) > for R > /(P) (7.2.26) -
I<P<O
PROOF It follows from the properties given in Lemma 7.2.2 and the mean
valuetheoremthat, forany6 > 0,thereexistsap intheinterval 1 < p <
such that8
7
/(P)=/(^; 1~)wasfirstdefinedinSec. 1.2.Henceforth, the conditional probability distribution
isusedas theargument becausethisisthevariableoverwhichweoptimize.
8 Weassume E(p,P)isstrictlyconvexn in p.Otherwisethisproofistrivial.
which, since (0, P) = and
;(0, P) =
/(P), implies
E (Po,P)> Po [I(P) + d] (7.2.27)
Hence
max E(R;p, P) = max [-pR + E
(p,P)]
-
i<p<o
-
i<p<o
> -
PoR + E (Po,P)
= -po[R-I(P)-S]
We can choose 6 = [R - /(P)]/2> so that
max E(R,p, P) > -
Po(
R ~ /(P)
)
> (7.2.28)
I<<O *
Analogously to the channel coding bound for fixed conditional probability distribution{P(i;
|u): v ei\ue%}, thevalue of the exponent max E(R;p,P)
-1
<P<O
isdetermined by the parametric equation
max E(Ri p, P)= -p*R + >*, P)
-
I<P<O
R = (7.2.29)
=p*
for
/(P) < R
and 1 <p* < 0. In Fig. 7.4 we sketch these relationships.
Nowletuscombinetheseresultsintoabound ontheaveragedistortionusing codes ofblock length N and rate R. We take the code ensemble average ofd()
given by (7.2.14) and bound thisbythesum of(7.2.15)and thebound inLemma
7.2.1. This results in the bound on d(J3) given by
d e~- (7.2.30)
for any 1 < p <0. Minimizing the bound with respect to p yields
D(P) + d exp -
-
l<p<0
max E(R;p9P)\\ (7.2.31)
where
max E(K;p,P)>0 for R> /(P) -
i<p<o
396 SOURCECODINGFOR DIGITALCOMMUNICATION E (P,
Slope=R
Slope= /(P)
p*R
Figure7.4 (p,P)curve.
and
At this point we are free to choose the conditional probability {P(v\u)} to mini mizethebound on d($)further. Suppose weare givenafidelitycriterionD which we wish to satisfy with theblock source encoder and decoder system ofFig. 7.3.
Let us next define the set ofconditional probabilities that satisfy the condition
D
(7.2.32)
&D=
{P(v\u):D(P)<D}
It follows that 3PD isa nonempty,closed, convex set forall
D > IQ(u)min d(u, v)= >
min (7.2.33)
sinceindefining v(u)by the relationd(u, v(u))= min d(u, v)we mayconstruct the
V
conditional distribution
-i; (7.2.34)
which belongs to 0>D and achieves the lower bound. Now we define the source reliability function
E(R, D)=max max (K; p, P) (7.2.35) -
i<p<o
and the function
R(D) = min
/(P) (7.2.36)
PeJ
We will soon show that in fact R(D) is therate distortionfunction as defined in (7.1.2), but for the moment we shall treat it only as a candidate for the rate distortion function. With these definitions we have the source coding theorem.
Theorem 7.2.1: Source coding theorem For any block length N and rate R, there exists ablock code M with average distortion d(M) satisfying
)
(7.2.37)
where
E(R, D)> for R > R(D)
PROOF Suppose P* e3?D achieves the maximization (7.2.35) in the source reliability function. Then from (7.2.31)we have
+ dQe~> (7.2.38)
where
E(R, Z))>0 for#>/(P*)
But by definition (7.2.32) of^D, we have D(P*)< D. Also since
E(R, D) > max (K; p, P) > forK > /(P)
-
I<P<O
where P can be any P eJ?D, we have
(R, D) > for R > min /(P)= R(D)
Hence
)e-JV <*-l
(7.2.39)
where E(R, D)> for R > R(D). Since this bound holds for the ensemble average overallcodes ofblock lengthN andrateR,we knowthat thereexists at least one code whosedistortion islessthan orequalto d(38\ thuscomplet ing the proof.
Example (Binarysymmetricsource, error distortion) Let# = i~={0, 1}andd(u,v)= 1-6
U1..
Also suppose Q(Q)= Q(\)=
\.Bysymmetry,the distributionPe&D that achievesboth E(R, D) and R(D)isgiven by
P(r|
U)=
[
V*U
where <D<
\ (7.2.40)
11- D v=u
398 SOURCECODINGFOR DIGITALCOMMUNICATION
Thentheparametric equations(7.2.29)become(seealsoSec. 3.4) E(R,D)=
E(R;p*,P)= -SDInD -(1 -<5D)In (1-
D)-Jf
(<5
and
R =In2-jT(d
D)
where
(7.2.41)
(7.2.42)
(7-2-43)
E(R,D)issketchedin Fig. 7.5 for <D<
\and R(D)<R<\n2whereR(D)=In2-Jf
(D).
E(R,D)
Figure7.5 Sketch ofE(R,D)forthe binary symmetricsource with errordistribution.
Theorem7.2.1 showsthat,as blocklengthincreases,wecanfindacodeofany
rate R > R(D) whose average distortion is arbitrarily close to D. A weaker but
more common form ofthis theorem is given next.
Corollary 7.2.2 Given any 6 > 0, there exists a block code ^ of rate
R <
R(D) + with average distortion d(M] < D + e.
PROOF Let R satisfy
R(D)<R<
R(D) + e
and choose N largeenough so that
In order to show that R(D) is indeed the rate distortion function, we must show that it is impossible to achieve an averagedistortion ofD or less withany source encoder-decoderpair that has rate R < R(D). To show this we firstneed twoproperties of7(P). First let{Pv (v |u):v e y\,ue 9fN}beanyarbitrarycondi tional distribution on sequences of length N. Also let P(n)(vn
\
un ) bethemarginal conditional distribution for the nth pair (rn, un
] derived from this distribution.
Defining
/(P.v) = 11 QN(U)P(V|u) In (7.2.44)
U V *JNV
and
P(n)(r\u\
/(P"")
= X I Q(u)P M(v| ) In jjLL (7.2.45)
u t r (V)
where
n=l and
)
= Z Q
u
we have the following inequalities
and
- Z/(P(n))<4
7(Pv) (7.2.47)
N n=l N
400 SOURCECODING FORDIGITAL COMMUNICATION
Inequality (7.2.46) is the statement that 7(P) is a convex u function of P. This statement is given in Lemma 1A.2 in App. 1A. Inequality (7.2.47) can be shown usingan argument analogoustotheproofofLemma 1.2.2forI(3 N\ ^N)givenin Chap. 1 (see Prob. 7.1).
Theorem 7.2.3: Converse source coding theorem For any source encoder- decoderpairitisimpossibletoachieve averagedistortionlessthan or equalto
D whenever the rate R satisfies R < R(D).
PROOF Any encoder-decoder pair defines amapping from source sequences to user sequences. For any length N, consider themapping from ^N to i^N where we let M be the number ofdistinct sequences in i^N into which the sequences of<%N are mapped. Define the conditionaldistribution
h (7.2.48)
otherwise
and letP(n)(v\u)bethe resultingmarginal conditionaldistributionon the nth terms in the sequences. Also, define the conditionaldistribution
(7.2.49)
Nowletusassumethatthemappingresultsin an averagedistortionofD
or less. Then
u
< D (7.2.50)
where u ismapped into v(u).But by definition (7.2.2)
N) = Z Z QN(U)/\(v|u)I
d(un, )
u v Pi n=1
n=l u v
n=1
(7.2.51)
where the inequality follows from (7.2.50). Hence P(v
\u) given by (7.2.49) belongs to 0*D and so
R(D) <
I(P)
I 1
"
- \
(sir)
4/(P.)
<-lnM
N
= R (7.2.52)
Weused here inequalities (7.2.46), (7.2.47),andI(PN
)
< In M(seeProb. 1.7).9
Hence,D(PN) <D implies that R(D) <R, which proves the theorem.
Note thatthisconverse source codingtheoremapplies toallsource encoder- decoderpairs and is not limited toblock coding. For any encoder-decoder pair and anysequence of length N,there issome mappingdefinedfrom-%N to VNand that isall that is required in the proof. Later in Sec. 7.3 when we consider non- block codes called trellis codes, this theorem will still be applicable.
The source codingtheorem (Theorem 7.2.1) and the converse source coding theorem (Theorem 7.2.3)together show that R(D)isthe ratedistortion function.
Hence for discrete memoryless sources we have R*(D) =
R(D) where R(D) = min
7(P) nats/source symbol
(V(v| ) In
P
(7.2.53)
#D=
p(t-1 ): 1 1 Q(u)P(v|u)d(u, v)<D
u r
Thusfor this casewe have an explicit formof the rate distortion functioninterms ofa minimization ofaverage mutual information.
The preceding sourcecodingtheorem and itsconverseestablish that therate distortionfunctionR(D)given by(7.2.53) specifies theminimumrateatwhichthe source decodermust receiveinformation aboutthesource outputs in ordertobe able to represent ittotheuser withanaveragedistortion that does not exceedD.
9 Withentropy sourcecodingdiscussed inChap. 1 it may bepossible to reducetheratebelow
(In M)/N,butneverbelowI(PN )/N.
402 SOURCECODINGFOR DIGITAL COMMUNICATION
Theorem 7.2.1 also shows that block codes can achieve distortion D with rate
R(D)inthelimitas theblocklengthNgoestoinfinity.Forablock code3doffinite
block lengthN andrate R,itisnatural toaskhowclose totheratedistortionlimit (D, R(D))wecan have (d(@\ R). Thefollowingtheorem provides abound onthe rate ofconvergence to the limit (D, R(D)).
Theorem 7.2.4 There exists a code ^ ofblock length N andrateRsuch that 0^- AlriA\ n *- A ,,-Nd2(N)/2C /7~\ CA\
<
cL\y3) L) < a ^/.Z.j4j
when
<
<5(N)
- R -
R(D) <\C
where C = 2 + 16[lnA]2 isa constant such that for all P
A-c. - ls , s .
PROOF From(7.2.30)we knowthat, foreachpinthe interval 1 <
p < and
fortheconditionalprobability {P(v|u):ve Y*,u e ^U\ thereexistsacode ^of block length N and rate R such that
d e~r (7.2.55)
Recall from (7.2.18)that
<-p
(7.2.56)
For fixed P, twice integrating E,(p, P) =d2E (p, P)/Sp2 yields
f [*; (, P) ^ dft= -pi(0, P) + (p, P)-
(0, P) (7.2.57) o o
Since (0, P) = and
;(0, P) =
/(P), we have
(p, P) =p/(P)+ [
[ ;(, P) da ^ (7.2.58)
Jo Jo
Let C be any constant upper bound to El(p, P). (See Prob. 7.3,where we show that C =2 + 16[ln A}2 is such abound for -J< p <
0.)Then
, P) >p/(P)-
f P
I C
>f\ ">r\
o
.2
da
(7.2.59)
Hence
E(R; p, P) > -pK + PI(P)
-^-C (7.2.60)
Now choose P* e J>Dsuch that /(P*) = R(D). Then
E(R;p, P*)>
-p[R -
R(D)] - -C P (7.2.61) Defining 6(N) = R-R(D), we choose
P*=-^>*- (7.2.62)
o
where S(N) is assumed small enoughto guarantee -j <p* <0. Then
(K;p*,P*)>^ (7.2.63)
^^o and putting this into (7.2.55) gives
(7.2.64)
There are many ways in which the bound on (d(M\ R) can be made to converge to (D, R(D)). For example, for some constant a >
6(N) = R- R(D) = aAT3/8 (7.2.65)
yields
/ ~2\ri/4
(7.2.66)
2C A different choice of
/In /V
(7.2.67) yields
H-
(7.2.68)
which shows that, ifR->
/?(D) as ^/(In N)/N, wecan have d($)-*Z) as AT for
any fixed y >0.
Although Theorem 7.2.4 does not yield the tightest known bounds on the convergence of (d(\ R) to (D, R(D)) (cf. Berger [1971], Gallager [1968], Pile [1968]), the bounds are easy to evaluate. It turns out that some sources called symmetricsources have block source codingschemes that can be shownto con verge much faster with block length (see Chap. 8, Sec. 8.5).
404 SOURCE CODINGFOR DIGITAL COMMUNICATION