Expected Value, Variance and Covariance of a Random Variable

Posted on Edited on In Views:

Sources:

  • Wikipidia

Notation

  • The expected value of a random variable \(X\) is often denoted by \(\mathrm{E}(X)\), \(\mathrm{E}[X]\), \(\mathrm{E} X\), with E also often stylized as \(\mathbb{E}\) or \(E\), or symbolically as \(\mu_X\) or simply \(\mu\).
  • The variance of random variable \(X\) is typically designated as \(\operatorname{Var}(X)\), or sometimes as \(V(X)\) or \(\mathbb{V}(X)\), or symbolically as \(\sigma_X^2\) or simply \(\sigma^2\) (pronounced "sigma squared").
  • The covariance of of two random variables \(X\) and \(Y\) is typically designated as \(\operatorname{Cov}(X, Y)\).

Expected value

In probability theory, the expected value (also called expectation, expectancy, expectation operator, mathematical expectation, mean, expectation value, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of the possible values a random variable can take, weighted by the probability of those outcomes.

For a discrete random variable \(X\) with sample space (or alphabet) \(\mathcal {X}\), the expectation of \(X\) is then given by the weighted average \[ \mathrm{E}[X]=\sum_{i=1}^{\infty} x_i P(X=x_i) \] where \(x_i \in \mathcal{X}\).

For a continuous random variable \(X\) with probability density function given by a function \(f\), the expectation of \(X\) is then given by the integral \[ \mathrm{E}[X]=\int_{-\infty}^{\infty} x f(x) d x . \]

Linearity of expectation

The expectation is a linear operator, i.e., \[ \mathrm{E}[a X+b Y] = a \mathrm{E}[X] + b \mathrm{E}[Y] . \] This can be easily proved by LOTUS.

Variance

The variance of a random variable \(X\) is the expected value of the squared deviation from the mean of \(X\) : \[ \operatorname{Var}(X)=\mathrm{E}\left[(X-\mathrm{E}[X])^2\right] . \]

This definition encompasses random variables that are generated by processes that are discrete, continuous, neither, or mixed.

The variance can also be thought of as the covariance of a random variable with itself: \[ \operatorname{Var}(X)=\operatorname{Cov}(X, X) \]

One important property is that, \[ \begin{aligned} \operatorname{Var}(X) & =\mathrm{E}\left[(X-\mathrm{E}[X])^2\right] \\ & =\mathrm{E}\left[X^2-2 X \mathrm{E}[X]+\mathrm{E}[X]^2\right] \\ & =\mathrm{E}\left[X^2\right]-2 \mathrm{E}[X] \mathrm{E}[X]+\mathrm{E}[X]^2 \\ & =\mathrm{E}\left[X^2\right]-2 \mathrm{E}[X]^2+\mathrm{E}[X]^2 \\ & =\mathrm{E}\left[X^2\right]-\mathrm{E}[X]^2 \end{aligned} \]

Covariance

Covariance in probability theory and statistics is a measure of the joint variability of two random variables.

For two jointly distributed real-valued random variables \(X\) and \(Y\), the covariance is \[ \operatorname{Cov}(X, Y)=\mathrm{E}[(X-\mathrm{E}[X])(Y-\mathrm{E}[Y])] \] ## Expectation --> Covariance

One important property is that \[ \begin{aligned} \operatorname{Cov}(X, Y) & =\mathrm{E}[(X-\mathrm{E}[X])(Y-\mathrm{E}[Y])] \\ & = \mathrm{E}[X Y-X \mathrm{E}[Y]-\mathrm{E}[X] Y+\mathrm{E}[X] \mathrm{E}[Y]] \\ & = \mathrm{E}[X Y]-\mathrm{E}[X] \mathrm{E}[Y]-\mathrm{E}[X] \mathrm{E}[Y]+\mathrm{E}[X] \mathrm{E}[Y] \\ & = \color{salmon}{\mathrm{E}[X Y]-\mathrm{E}[X] \mathrm{E}[Y]} . \end{aligned} \]

What if two random variables are independent?

Suppose \(X, Y\) are independent, we have \[ \begin{aligned} \operatorname{Cov}(X, Y) & =\mathrm{E}[(X-\mathrm{E}[X])(Y-\mathrm{E}[Y])] \\ & = \color{salmon}{\mathrm{E}[X Y]-\mathrm{E}[X] \mathrm{E}[Y]} \\ & = \mathrm{E}[X] \mathrm{E}[Y] - \mathrm{E}[X] \mathrm{E}[Y] \\ & = 0 . \end{aligned} \]