Method

When considering two discrete variables $x$ and $y$ at the same time, it is possible to measure the degree of uncertainty or information associated with them. It is called the joint entropy , $H(x,y)$. If $x$ and $y$ may respectively take $m_1$ and $m_2$ possible values, the joint entropy is computed as in equation 6.4:

$$H(x,y) = - \sum_{x=i}^{m_1} \sum_{y=j}^{m_2} p_{ij}(x,y) \log_{2} p_{ij}(x,y)$$ (6.4)

where $p_{ij}$ represents the probability of being classified in both category $i$ of variable $x$ and category $j$ of variable $y$.

The joint entropy varies from a theoretical 0 (or empirically $min(H(x),H(y))$) to $\log_2(m_1) + \log_2(m_2)$. The relation between the individual entropies and their joint entropy is given by equation 6.5:

$$H(x,y) \leq H(x) + H(y)$$ (6.5)

It expresses the fact that the joint entropy is always smaller then the sum of the individual entropies. The equality holds only when the two variables are independent.