Consider a source which generates 2 symbols with probability
and
.
Now consider a sequence of 2 outputs as a single symbol
,
Assuming that consecutive outputs from the source are statically independent, show directly that
Solution
First find
Where in this case is 2. The above becomes
Hence becomes
Now, we conside Below we write
with the probability of each 2 outputs as single symbol on top
of each. Notice that since symbols are statistically independent, then
, we
obtain
Hence since
Where now, the above becomes
Now we will expand the terms labeled above as and simplify, then we will obtain (1) showing the
desired results.
Substitute the result we found for back into (2) we obtain
But the above can be written as
Compare (4) and (1), we see they are the same.
Hence
A source emits sequence of independent symbols from alphabet of symbols
with probabilities
, find the entropy of the source alphabet
Solution
Where , hence the above becomes
But , hence the above becomes
To verify, we know that must be less than or equal to
where
in this case,
hence
or
, therefore, our result above agrees with this upper limit
restriction.