Independence
Ref:
- NOTES ON PROBABILITY by Greg Lawler
- Probability_Theory by Kyle Siegrist
- The Book of Statistical Proofs
Independence of Events
Let \(A_1,\cdots, A_n\) be \(n\) arbitrary statements about random variables \(X_1,X_2, \cdots,X_n\) with possible values \(\mathcal{X}_1,\mathcal{X}_2, \cdots,\mathcal{X}_n\) ,
\(A_1,\cdots, A_n\) is called statistically independent, if \(\text{for all} \; x_i \in \mathcal{X}_i, \; i = 1, \ldots, n\): \[ p(\bigcap_{i=1}^n A_i) = \prod_{i=1}^{n} p(X_i = x_i) \]
Note: \[ p(\bigcap_{i=1}^n A_i) = p(A_1,\cdots, A_n) = p(X_1 = x_1, \cdots, X_n = x_n) \]
where \(p(X_1 = x_1, \cdots, X_n = x_n)\) are the joint probabilities of \(X_1 = x_1, \cdots, X_n = x_n\) and \(p(X_i=x_i)\) are the marginal probabilities of \(A_i: X_i=x_i\).
Independence of Random Variables
Two random variables \(X\) and \(Y\) are independent <==> the elements of the π-system generated by them are independent; that is: \[ \textrm{P}(X\le x, Y\le y) = \textrm{P}(X\le x)\textrm{P}(Y\le y) \]
Conditional Independence
As noted at the beginning of our discussion, independence of events or random variables depends on the underlying probability measure.
Thus, suppose that \(B\) is an event with positive probability. A collection of events or a collection of random variables is conditionally independent given \(B\) if the collection is independent relative to the conditional probability measure \(A \rightarrow p(A|B)\).
For example, a collection of events \(p(\bigcap_{i=1}^n A_i)\) is conditionally independent given \(B\) if for every \(i\): \[ p(\bigcap_{i=1}^n A_i | B) = \prod_{i=1}^{n} p(A_i | B) \] Note: \[ p(\bigcap_{i=1}^n A_i | B) = p(X_1 = x_1, \ldots, X_n = x_n|Y = y), \\ \prod_{i=1}^{n} p(A_i | B) = \prod_{i=1}^{n} p(X_i = x_i|Y = y), \quad \text{for all} \; x_i \in \mathcal{X}_i \quad \text{and all} \; y \in \mathcal{Y} \]
where \(X_1,X_2, \cdots,X_n\) are discrete random variables with possible values \(\mathcal{X}_1,\mathcal{X}_2, \cdots,\mathcal{X}_n\); \(Y\) is discrete random variable with possible values \(\mathcal Y\). \[ p(X_1 = x_1, \ldots, X_n = x_n|Y = y) = \prod_{i=1}^{n} p(X_i = x_i|Y = y) \quad \]