Relationships with other entropies

The relationship between conditional entropy and the other types of entropy is as follows:

$$H(Y\vert X) = H(X,Y) - H(X)$$ (6.9)

$$H(Y\vert X) = H(Y) - I(X,Y)$$ (6.10)

Conditional entropy of Y given X is the joint entropy of X and Y minus the entropy of X; in other words, it is the difference between the information of X and Y, and the information brought by X. This is clearly the meaning of this type of entropy, a reduction of uncertainty. The second variant $H(Y\vert X) = H(Y) - I(X,Y)$ puts it a little differently: it is difference between the information contained in Y and the shared part of X and Y.

Contrary to the joint entropy or the mutual information, conditional entropy is not a symmetrical measure: $H(X\vert Y) \neq H(Y\vert X)$. Conditioning on a variable or the other does not give the same result. This property will be fully exploited later, by conditioning the state of a system at time $t$ given its state at time $t-1$. Of course it would not be the same as conditioning the other way around.