8.5 SYMMETRIC SOURCES WITH BALANCED DISTORTION MEASURES AND FIXED COMPOSITION SEQUENCES
8.5.2 Fixed-Composition Sequences Binary Alphabet Example
There isaclose relationshipbetween symmetricsourceswithbalanceddistortions and fixed-composition source output sequences ofan arbitrary discrete source.
For sequences of fixed composition, we can prove a theorem analogous to
Theorem 8.5.1.Although thispropertyiseasilygeneralizable to arbitrary discrete alphabet sources with a bounded single-letter distortion measure (see Martin
[1976]), we demonstrate the results for the binary source alphabet and error distortion measure.
Suppose we have a source alphabet ^ = {0, 1}, a representation alphabet i^ =
{0, 1}, and error distortion measure
d(k,j)=\- dkj for /c,j= 0, 1
(8.5.22)
Foru e3SN, defineitsweightas w(u) = numberofIsin
u,anddefinethecomposi tion classes,
<?(/)= {u:ue#w, w(u)=/} /= 0, 1,2, ..., N (8.5.23)
with probabilities
-
/= 0, 1, 2, ..., N (8.5.24)
and correspondingrate distortion functions [see (7.6.62)]
R(D- Q(/)) = je~ - JT(D) (8.5.25) 0<D<min(-,l --
| /= 0, 1,2, . , N
\N Nj
518 SOURCE CODINGFOR DIGITALCOMMUNICATION
Using the Chernoff bound (see Prob. 1.5) we have for the number of se quences in %y(/), denoted \^N(l)\
:eN*(l/N)
(8.5.26)
Thismeans we can always find a code of rate
R >*
(N) SUChthatM =eNR>
\VN(I)\
whichcan uniquelyrepresenteach sequencein^N(l)andthus achieve zerodistor tion. We shall encode some composition classeswith zero distortion and others with some nonzero distortion.
Let us now pick 6 such that < 6 < In 2, pick fixed rate R in the interval
6 <R < In 2, and choose <e < 0.3 to satisfy
jr(c)<d (8.5.27)
Observe that wecan make eand 6 as small as wepleaseand stillsatisfy (8.5.27).
Let the binary distribution Q* satisfying Q*(l) <i bedefined parametrically in terms of the rate R, eand d, as follows:
= JT(fi*(l))-
JT() +6 (8.5.28)
Also let /* be the largest integer such that l*/N <
Q*(l) <i Then from Fig. 8.3
wesee that for any fixedcomposition class %y(/) where either
l/N < l*/N or 1 -
IfN < l*/N (8.5.29)
we have
~ -
*ji - * (Q*(l)) (8-5-30) and
(8.5.31)
Thusforany compositionclass<#
N(l)forwhich jjf(l/N) <
Jj?(Q*(l)),wecanfinda block code of rate R and block length Nsuch that
(8.5.32)
and from (8.5.26)
M = eNR > eN*(llN)
>
|
VN(l)
| (8.5.33)
_[*_ Q* 0.2 _/_0.4
N N
0.6 0.8 i _J^ 1.0
N
Figure83 Binary entropyrelationships.
Therefore, since there aremore representationsequencesMthan sequencesinthe
class, such a code can encode sequences from ^N(l) with zero distortion where /
satisfies (8.5.29).
For any other fixed composition class ^N(l)for which instead
/* L _ l*
N <
N <
~N
define D{ > c to satisfy
R =
(8.5.34)
(8.5.35)
520 SOURCECODINGFOR DIGITALCOMMUNICATION SuchaDlcan befoundintherangee<Dt
<
l/N.Thisisillustrated in Fig. 8.3.We
show nextthat, like ourresult forthe symmetricsource with balanceddistortion measure presented in Theorem 8.5.1, wecan find a code of rate R such that all
sequences in ^N(l) can beencoded with distortion D,or less. Firstweestablish a
lemma analogous to Lemma 8.5.1 by considering an ensemble ofblock codes
^ = {vi> v2, ..., VM} ofblock length N and rate R = (In M)/N with probability
distribution
= m=l n n=lnno (s.5.36)
where
l) (8.5.37)
and P(l}(v
|u) is the conditional probability yielding the rate distortion function
Lemma 8.5.2 Let c >0, d > and rate d< R <In 2 satisfy (8.5.27) and
(8.5.28). For a fixed composition class %>N(l) satisfying (8.5.34), D,satisfying (8.5.35),and anyu e%>N(l\overtheensembleofblockcodes withprobability distribution (8.5.36)
Pr {d(u\0)>D l\
PROOF
= Pr
[dN(u, vm)>DI; m = 1, 2, ..., M|u e N(l)}
^e-MPr{dN(u,v)<Di\ue<#N(l)}
(8.5.39)
Here the key property we employ is that Pr {dN(u, v) <Dt\
ue^N(l)} is
independent of u e^N(l\sinceonlythe composition determinestheprobabil
itydistributionof dN(u, v),whichisanormalizedsumofindependent (though not identically distributed) random variables. The generalized Chernoff bounds in App. 8A again suffice for ourpurpose. Here we have
Pr
{</>, v) < Df|uG
<*(/)}
> l - -^-w^^)- "^)]
(g.5.40)
Substituting(8.5.35)into(8.5.40)andtheresult into(8.5.39)then gives us the desired result.
It is easy to see that, for e<0.3, we have
3tf(e)> e In (c/2) so that
6 > -c In (c/2) > (see Prob. 8.12). Hence the exponent [6 + e In (c/2)] > in
(8.5.38). From this lemmafollows the desiredresult.
Theorem8.5.2 Letc> 0, d > satisfyJjf(c)< S.Forsufficiently largeinteger
N*9 for any rate R in the interval d < R < In2, and any composition class%v(/)where N > N*,thereexistsacode ^ofblock lengthN andrateR
such that
d(u|#|) <
D, for allu e <
N(l) (8.5.41)
where D, satisfies
when
and D,= otherwise. Here Q*(l)< \satisfies
PROOF For l/N [Q*(l\ 1 -
fi*(l)], Dl = as a result of(8.5.33). Now for
any l/N e[(?*(!), 1 -
fi*(l)], suppose we have a source that emits only se quencesfrom %v(0 w ^h equal probabilities.For any block code $ ofblock length N and rate R, define the indicator function
AveragingO over output sequences, we obtain
(8-5-
43)
Nextconsideran ensembleofblockcodeswhere code M = {vl5v2, ..., VM}is
chosen accordingto the probability distribution (8.5.36)and(8.5.37).Averag ing(8.5.43) over this code ensemble yields
* i ./ i ^,\
^-i ^
=
2-< \w i]\\
522 SOURCE CODINGFOR DIGITALCOMMUNICATION
where the inequality follows from Lemma 8.5.2. Using the bound
|##(/)|
<2N
, it follows that thereexistsacode ^ofblock lengthN andrate
R such that
X O(u|#,)< X O(u|#)
ue #jv(0 ue<#N(l)
^ TN _ (1 4/Ne2)expN[d+cIn (e/2)] /o r AC\
^^ c ^O.J.tJ^
Choosing N* tobeanyinteger forwhichtheboundislessthanone, itfollows as in the proofofTheorem 8.5.1 that O(u 1^) = for all u e^N(/).
Thistheorem shows that given any <6 < In 2, rate R such that (5 <R <
In 2, and < e<0.3 satisfying^f(e)< d, for any composition class %N(l) where
JV >N*, we can find a block code, &t, ofblock length N and rate fl such that
d(u\^t)= for all u eVN(l) if JP(1/N) <tf(Q*(\)\ and d(u\&i) < Dl for all
u e%v(/) if JT(//N)> *r(Q*(l)) where Q* satisfies (8.5.28) and D, > satisfies (8.5.35) (see also Fig. 8.3).
It is natural to define the composite code N
@c= \j t (8.5.46)
/=o
which has (N + l)eNRelements(eNR
foreach of theN + 1 compositeclasses) and hence rate
For the code ^c, we have
d(u\$c)<Dl ifu^(/) (8.5.48)
where wetakeDl = ifje(l/N) <
JT(Q*(1)).Weseethat,asN - oo,Rc->R,and thusby choosingN largeenough wecanmakethe rate of thecomposite codeJ? c arbitrarily close to R.
Up to this point, the results depend only on the source alphabet and are independent of the source statistics. The composite code 3# c satisfies (8.5.48) regardless of the actualsourcestatistics.Suppose, however,thatour binary source
is memoryless with probability Q(l)= q<\ and Q(0)= 1 -
q. Then the rate distortion function for this source is R(D) = jtf(q)-
J-f(D) for < D <
q. How
welldoes thecomposite code encodethissource?Theaveragedistortionusingthe composite code is
1= Z e
tfjv(J)
< z <?<!
- r D,
(i)
r ^ (8-5.49)
As N increases, (f)g(l -
q) N~l
concentrates its mass around its mean Nq. This follows from theasymptoticequipartitionproperty(McMillan[1953])whichsays
that,as block lengthincreases, almostallsource sequences tend tohavethesame composition. Thus we have (see Prob. 8.13 and Chap. 1)
lim CWO-)"-*> =*> (8-5-50) N-oc /=
where D satisfies (8.5.35) with 1=
Nq\ that is
= R(D) + 6 (8.5.51)
The code rate for J$ c then becomes Rc = R(D) + 6 +
M^l> (8.5.52)
Hence given any r\ >0, wecan find 6 small enough and N large enough so that
4#C)<D+ ^ (8.5.53)
and
Rc< R(D) + r] (8.5.54)
Thus the composite codes can encode any memoryless binary source with error distortionarbitrarilyclose to the theoreticalratedistortion limit.This is arobust source encodingschemeformemorylesssourcesinthesense that thesame compo
sition classcode isefficient (near the ratedistortionlimit) forallsuchsourcesand thecomposite code is constructed independent of actual source statistics.
Theprecedingexample ofabinaryalphabet withtheerror distortionmeasure can be generalized to arbitrary discretealphabets andarbitrary single-letter dis
tortions(seeProb.8.14).Furthergeneralizations are possiblebyconsideringfixed
finite sequences of source outputs as elements of a larger extended discrete alphabet. In this manner, the robust source coding technique can be applied to sources with memory (see Martin [1976]). The basic approach ofconsidering a singlesource asa compositeofsubsourcesandfindingcodesforeachsubsourcein constructing atotal composite code isalsoused inencoding nonergodicstation ary sources. This is referred to as universal source coding and is discussed in Sec. 8.6.
Wehavedemonstratedasimilarity between symmetricsourceswithbalanced distortions and fixed composition classes. In general with any discrete alphabet, forany fixedcomposition class,we maydefinea function R(D; Q) where Qisthe
distributiondetermined by the composition. We can show that ifR > R(D; Q) and the block length is large enough, we can find a code that will encode all sequences of the composition class to distortion D or less. Certainly if
R > max R(D; Q) (8.5.55)
Q
524 SOURCE CODINGFOR DIGITALCOMMUNICATION
then every output sequence can be encoded withdistortionD orless. Symmetric sources with balanced distortionshave the property that
R(D) = maxR(D ,Q) (8.5.56) Q
Thus the symmetric source coding theorem (Theorem 8.5.1) is actually a special caseof thecompositionclasssourcecoding theorem (Theorem8.5.2appropriately generalized to arbitrary discrete alphabets and any single-letter distortionmeas
ures). See Probs. 8.14 and 8.15 for generalizationsand further details.