8.5 SYMMETRIC SOURCES WITH BALANCED DISTORTION MEASURES AND FIXED COMPOSITION SEQUENCES
8.5.1 Symmetric Sources with Balanced Distortion Measures
A symmetric source is a discrete memoryless source with equally likely output
letters. That is,
if=
fa,02,...,*} (8.5.1)
where
Q(ak) = ~ fc=l,2,...,,4 (8.5.2)
Assuming the same number of representation letters as source letters where i~ =
{/>!, 62, ^}>
f r abalanced distortion measure, there exist nonnegative numbers {dl, d2, ..., dA} such that
{d(u, b,\ d(u, b2\ ..., d(u, bA)} =
[dl9 d29...,dA] for all u e V
and (8.5.3)
{d(al9 v\ d(a2, v), ..., d(aA, v)} = {d^ d2,...,dA
} for all v e i
514 SOURCECODING FORDIGITAL COMMUNICATION
The rate distortion function R(D)is given parametrically by
**
(7.6.69) k=l
R(DS) = sDs + In A -In
( esdk
} (7.6.70)
u=i /
where s < is the independent parameter.
Consideragaintheblock sourceencoding and decodingsystemofFig. 7.3.As we did earlier, weprove a coding theorem by consideringan ensemble ofblock codes of size M and block length N. By symmetry in thisensemble, we choose code $ ={Y!, v2, ..., VM} with uniform probability distribution, that is
MN
(8-5.4)
Here each code letter is chosen independently ofother code letters and with a uniformone-dimensional probability distribution. Furthermore,since thedistor tionmatrixisbalanced,forfixedu e <%,therandomvariabled(u, v)isindependent ofu. That is, for anyu e W
Pr {d(u, v)= dk
\u} = - k = 1,
2, ..., A (8.5.5)
VT.
This means that for any fidelity criterion D and any two source sequences
u, u eWH we have
Pr {^(u, v)> D u} = Pr
{</>, v)> D
\
u} (8.5.6)
Thisis the key property ofsymmetricsourceswith balanceddistortionmeasures which we nowexploit.
Lemma 8.5.1 Given block length N, distortion level D>Dmm, and any source output sequence u e91N, overtheensembleofcodes3$ofblock length
N and rate R> R(D)
Du<e~MFN(D)
(8.5.7)
where
o(N)-> as N->oo
and
Ffl(D)= Pr{d
fl(u,v)<D\U} (8.5.8)
is independent ofu e VN.
PROOF Let .$ =
{vt, v2, ..., VM}. Thensince code wordsareindependentand
identically distributed, according to (8.5.6) Pr {d(u\)>D\u}= Pr
Jmin dv(u, v)>
ve.3 D\u\
= Pr{dv(u, vm)>D:m=l
PrKv(u,vm)>D|u
= [1 - FX(D)]M
where the inequality follows from In x < x 1.
Next note that, for fixed u e <%N
4*M) = ^ X </(, O (8.5.10)
n=1
is a normalizedsum ofindependent identicallydistributedrandom variables.
In App. 8A weapplytheChernoffboundingtechniquetoobtainforanye>
(8.5.11)
\ ^vt /
where s satisfies
A
D-e =k-^- (8.5.12)
We assume D> Dmin and choose e > small enough so that D - e > Dmin.
This guarantees that s is finite and converges to a finite limit as e- 0. In
particular, choosing
C= J^? (8-5-13)
we have
:-lnFv(D)< -R(D) (8.5.14)
From this lemma it follows immediately that the average distortion
satisfies
D + j ^-exp.v[K-d (8.5.15)
and hence that there exists a code & for which d() also satisfies this bound.
Comparing this with Theorem 7.2.1, we see that this lemma is a stronger result
516 SOURCE CODINGFOR DIGITAL COMMUNICATION
since the second term here is decreasing ata doubleexponentialrate withblock length N, compared to the single exponential rate of Theorem 7.2.1. Another observationis thatLemma8.5.1 holds regardlessofthe source probabilitydistri bution and is true even for sources with memory.This happenssince we have a balanced distortion matrix and assume a uniform distribution on the code ensemble. Ofcourse, when the source output probability distribution isnot uni form, we cannot say that the R(D) ofthe symmetric source is the rate distortion function. It is clear, however, that the rate distortion function of the symmetric source, R(D), is an upper bound to the rate distortion functions of all other sources with thesamebalanceddistortion, since wecan always achievedistortion arbitrarily close to D with rate arbitrarily close to R(D). We consider this in greaterdetailwhen we examinetheproblemofencodingsource sequencesoffixed composition. We next prove the source coding theorem for symmetric sources with balanced distortion measures.
Theorem 8.5.1 For a symmetric source with a balanced distortion measure and any rate R where R>R(D), there exists a block code $ofsufficiently large block length N and rateR such that
d(u\@)<D for all u e WN (8.5.16)
PROOF For any code $ of block length N and rate R, define the indicator function
for u eWN. Averaging<I>over source output sequences gives
Ie*(u)<I>(u|)=
-^<l>(u|) (8.5.18)
U A U
Averaging this over the ensemble ofcodes yields
u A
= XQw(u)Pr {*( |)>D|u
(8.5.19)
where the inequality follows from Lemma 8.5.1. This means there exists at least one code ^ for which
<e
u
-expN[R-R(D)+o(N)]
or
(8.5.20) u
The bound can be made less than 1 by choosing N large enough when
R > R(D). Then we have
1 (8.5.21)
But by the definition (8.5.17), foreach u,O(u|^) can only be and 1. Hence
(8.5.21) implies that O(u|J?)= for all u e ^N, which requires d(u\M) <D
for all u.
Since (8.5.16) holds for all output sequences, we see that this theorem holds for any source distribution (QN(u): u etfSN} when R(D) is the symmetric source
rate distortion function and R > R(D). For any other source distribution, the actual rate distortion function will be less than that of the uniform distribution.