Source:

• Random Vectors from the textbook Introduction to Probability, Statistics, and Random Processes by Hossein Pishro-Nik.
• Random Vectors and the Variance–Covariance Matrix

Random Vectors

Notation

Symbol	Type	Description
$\overrightarrow{X}$	Random vector	A vector of jointly distributed random variables $\overrightarrow{X}=[X_1,X_2,\dots ,X_p{]}^T$
$\mathrm{E}[\overrightarrow{X}]$	Vector	Expectation of the random vector $\overrightarrow{X}$
${\overrightarrow{\mu }}_{\overrightarrow{X}}$, $\overrightarrow{\mu }$	Vector	Alternative notation for the expectation $\mathrm{E}[\overrightarrow{X}]$
$\mathrm{Var}(\overrightarrow{X})$	Matrix	Variance-covariance matrix (or simply covariance matrix) of $\overrightarrow{X}$
${\mathrm{\Sigma }}_{\overrightarrow{X}}$, $\mathrm{\Sigma }$, $\mathrm{Cov}(\overrightarrow{X})$	Matrix	Alternative notations for the variance-covariance matrix
$\mathrm{Cov}(\overrightarrow{X},\overrightarrow{Y})$	Matrix	Covariance matrix between two random vectors $\overrightarrow{X}$ and $\overrightarrow{Y}$
$A\overrightarrow{X}$	Linear transformation	Transformation of the random vector $\overrightarrow{X}$ by a $k\times p$ matrix $A$
$\mathrm{Cov}(X_i,Y_j)$	Scalar	Covariance between the $i$-th component of $\overrightarrow{X}$ and the $j$-th component of $\overrightarrow{Y}$
$[\mathrm{E}[\overrightarrow{X}]]$	Column vector	Expectation of the random vector $\overrightarrow{X}$ expressed as a column matrix
$\mathrm{E}[\overrightarrow{X}{\overrightarrow{Y}}^T]$	Matrix	Matrix of expected pairwise products between components of $\overrightarrow{X}$ and $\overrightarrow{Y}$
$X_1,X_2,\dots ,X_p$	Random variables	Components of the random vector $\overrightarrow{X}$

Distinction: Variance-Covariance Matrix vs. Covariance Matrix:

The terms variance-covariance matrix and covariance matrix are often used interchangeably in many contexts, but they can have subtle distinctions depending on the scenario:

1. Variance-Covariance Matrix: Refers specifically to the covariance matrix of a single random vector.
2. Covariance Matrix: A more general term that applies to the covariance between two random vectors.

Abbreviations

Abbreviation	Description
r.v.	Random variable

Notation

• The expected value of a random vector $\overrightarrow{X}$ is often denoted by $\mathrm{E}(\overrightarrow{X})$, $\mathrm{E}[\overrightarrow{X}]$, $\mathrm{E}\overrightarrow{X}$, with E also often stylized as $\mathrm{E}$ or $E$, or symbolically as ${\overrightarrow{\mu }}_{\overrightarrow{X}}$ or simply $\overrightarrow{\mu }$.
• The variance of random vector $\overrightarrow{X}$ is typically designated as $\mathrm{Var}(\overrightarrow{X})$, or sometimes as $\mathrm{Cov}(\overrightarrow{X})$. Since the variance is a variance-covariance matrix, it's also denoted as ${\mathrm{\Sigma }}_{\overrightarrow{X}}$ or $\mathrm{\Sigma }$. The element at i-th row j-th column is ${\mathrm{\Sigma }}_{ij}$
• The covariance of two random vectors $\overrightarrow{X}$ and $\overrightarrow{Y}$ is typically designated as $\mathrm{Cov}(\overrightarrow{X},\overrightarrow{Y})$. Since the covariance is a variance-covariance matrix, it's also denoted as ${\mathrm{\Sigma }}_{(\overrightarrow{X},\overrightarrow{Y})}$.

Abbrevations

Definition

Definition: A random vector $\overrightarrow{X}$ is a vector $$\overrightarrow{X}=\left[\begin{array}{c}{X}_1\\ X_2\\ \cdot \\ \cdot \\ X_p\end{array}\right]$$ of jointly distributed random variables $X_1,\cdots ,X_p$. As is customary in linear algebra, we will write vectors as column matrices whenever convenient.

Expectation of a random vector

Definition: The expectation $E\overrightarrow{X}$ of a random vector $\overrightarrow{X}={[X_1,X_2,\dots ,X_p]}^T$ is given by $$\mathrm{E}[\overrightarrow{X}]=\left[\begin{array}{c}\mathrm{E}[X_1]\\ \mathrm{E}[X_2]\\ ⋮\\ \mathrm{E}[X_p]\end{array}\right]$$ It's also denoted as ${\overrightarrow{\mu }}_{\overrightarrow{X}}$ or $\overrightarrow{\mu }$.

Linearity of expectation

Recalling that, the expectation for random variables is a linear operation, this linearity also holds for random vectors.

The linearity properties of the expectation can be expressed compactly by stating that for any $k\times p$-matrix $A$ and any $1\times j$-matrix $B$, $$\mathrm{E}[A\overrightarrow{X}]=A\mathrm{E}[\overrightarrow{X}]\text{ and }\mathrm{E}[\overrightarrow{X}B]=\mathrm{E}[\overrightarrow{X}]B$$

Variance of a random vector

The variance of a random vector $\overrightarrow{X}$ is represented as a matrix, known as the variance-covariance matrix (often simply referred to as the covariance matrix in some literature). $$\mathrm{Var}(\overrightarrow{X})=\mathrm{Cov}(\overrightarrow{X},\overrightarrow{X})=\mathrm{E}[(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}])(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]{)}^T].$$ It's also denoted as ${\mathrm{\Sigma }}_{\overrightarrow{X}}$, $\mathrm{\Sigma }$, or $\mathrm{Cov}(\overrightarrow{X})$.

Expectation --> Variance

One important property is that, $$\begin{array}{rl}\mathrm{Var}(\overrightarrow{X})\triangleq \mathrm{Cov}(\overrightarrow{X},\overrightarrow{X})& =\mathrm{E}[(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}])(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]{)}^T]\\ & =\mathrm{E}[\overrightarrow{X}{\overrightarrow{X}}^T]-\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{X}{]}^T\end{array}.$$ The proof is easily derived from covariance operator $\mathrm{Cov}(\overrightarrow{X},\overrightarrow{Y})$.

Covariance between two random vectors

For two jointly distributed real-valued random vectors $\overrightarrow{X}$ and $\overrightarrow{Y}$, the covariance is represented as a matrix, called the covariance matrix: $$\mathrm{Cov}(\overrightarrow{X},\overrightarrow{Y})=\mathrm{E}[(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}])(\overrightarrow{Y}-\mathrm{E}[\overrightarrow{Y}]{)}^T]$$ It's also denoted as ${\mathrm{\Sigma }}_{(\overrightarrow{X},\overrightarrow{Y})}$.

The covariance matrix

For two random vectors $\overrightarrow{X}={[X_1,X_2,\dots ,X_p]}^T\in {\mathbb{R}}^p$ and $\overrightarrow{Y}={[Y_1,Y_2,\dots ,Y_q]}^T\in {\mathbb{R}}^q$, their covariance matrix is a $p\times q$ matrix defined as:

$$\mathrm{Cov}(\overrightarrow{X},\overrightarrow{Y})=\left[\begin{array}{cccc}\mathrm{Cov}(X_1,Y_1)& \mathrm{Cov}(X_1,Y_2)& \cdots & \mathrm{Cov}(X_1,Y_q)\\ \mathrm{Cov}(X_2,Y_1)& \mathrm{Cov}(X_2,Y_2)& \cdots & \mathrm{Cov}(X_2,Y_q)\\ ⋮& ⋮& \ddots & ⋮\\ \mathrm{Cov}(X_p,Y_1)& \mathrm{Cov}(X_p,Y_2)& \cdots & \mathrm{Cov}(X_p,Y_q)\end{array}\right]$$

Here: - $\mathrm{Cov}(X_i,Y_j)$ represents the covariance between the random variables $X_i($ from $\overrightarrow{X})$ and $Y_j$ (from $\overrightarrow{Y}$ ). - If $\overrightarrow{X}=\overrightarrow{Y}$, this matrix reduces to the variance-covariance matrix of $\overrightarrow{X}$, which is symmetric because $\mathrm{Cov}(X_i,Y_j)=\mathrm{Cov}(X_j,Y_i)$ by the definition of covariance for random variables.

Expectation --> Covariance

$$\begin{array}{rl}\mathrm{Cov}(\overrightarrow{X},\overrightarrow{Y})& =\mathrm{E}[(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}])(\overrightarrow{Y}-\mathrm{E}[\overrightarrow{Y}]{)}^T]\\ & =\mathrm{E}[\overrightarrow{X}{\overrightarrow{Y}}^T-\overrightarrow{X}\mathrm{E}[\overrightarrow{Y}{]}^T-\mathrm{E}[\overrightarrow{X}]{\overrightarrow{Y}}^T+\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{Y}{]}^T]\\ & =\mathrm{E}[\overrightarrow{X}{\overrightarrow{Y}}^T]-\mathrm{E}[\overrightarrow{X}\mathrm{E}[\overrightarrow{Y}{]}^T]-\mathrm{E}[\mathrm{E}[\overrightarrow{X}]{\overrightarrow{Y}}^T]+\mathrm{E}[\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{Y}{]}^T]\\ & =\mathrm{E}[\overrightarrow{X}{\overrightarrow{Y}}^T]-\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{Y}{]}^T-\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{Y}{]}^T+\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{Y}{]}^T\\ & =\mathrm{E}[\overrightarrow{X}{\overrightarrow{Y}}^T]-\mathrm{E}[\overrightarrow{X}]\mathrm{E}[\overrightarrow{Y}{]}^T\end{array}$$

Linear combinations of random variables

Consider random variables $X_1,\dots ,X_p$. We want to find the expectation and variance of a new random variable $L(X_1,\dots ,X_p)$ obtained as a linear combination of $X_1,\dots ,X_p$; that is, $$L(X_1,\dots ,X_p)=\sum _{i=1}^p{a}_i{X}_i.$$

Using vector-matrix notation we can write this in a compact way: $$L(\overrightarrow{X})={\overrightarrow{a}}^T\overrightarrow{X},$$ where ${\overrightarrow{a}}^T=[a_1,\dots ,a_p]$. Then we get: $$E[L(\overrightarrow{X})]=E\left[{\overrightarrow{a}}^T\overrightarrow{X}\right]={\overrightarrow{a}}^{T}E\overrightarrow{X},$$ and $$\begin{array}{rl}\mathrm{Var}[L(\overrightarrow{X})]& =\mathrm{E}[L(\overrightarrow{X})L(\overrightarrow{X}{)}^T]-\mathrm{E}[L(\overrightarrow{X})]\mathrm{E}[L(\overrightarrow{X}){]}^T\\ & =E\left[{\overrightarrow{a}}^T\overrightarrow{X}{\overrightarrow{X}}^T\overrightarrow{a}\right]-E\left({\overrightarrow{a}}^T\overrightarrow{X}\right){\left[E\left({\overrightarrow{a}}^T\overrightarrow{X}\right)\right]}^T\\ & ={\overrightarrow{a}}^{T}E\left[\overrightarrow{X}{\overrightarrow{X}}^T\right]\overrightarrow{a}-{\overrightarrow{a}}^{T}E\overrightarrow{X}(E\overrightarrow{X}{)}^T\overrightarrow{a}\\ & ={\overrightarrow{a}}^T(E\left[\overrightarrow{X}{\overrightarrow{X}}^T\right]-E\overrightarrow{X}(E\overrightarrow{X}{)}^T)\overrightarrow{a}\\ & ={\overrightarrow{a}}^T\mathrm{Cov}(\overrightarrow{X})\overrightarrow{a}\end{array}$$

Thus, knowing $E\overrightarrow{X}$ and $\mathrm{Cov}(\overrightarrow{X})$, we can easily find the expectation and variance of any linear combination of $X_1,\dots ,X_p$.

Collary: $\mathrm{\Sigma }$ is positive semi-definite

Corollary: If $\mathrm{\Sigma }$ is the covariance matrix of a random vector, then for any constant vector $\overrightarrow{a}$ we have $${\overrightarrow{a}}^T\mathrm{\Sigma }\overrightarrow{a}\ge 0.$$

That is, $\mathrm{\Sigma }$ satisfies the property of being a positive semi-definite (PSD) matrix.

Proof: According to the previous section, ${\overrightarrow{a}}^T\mathrm{\Sigma }\overrightarrow{a}$ is the variance of a random variable. We know that variance is always non-negative.

This suggests the question: Given a symmetric, positive semi-definite matrix, is it the covariance matrix of some random vector? The answer is yes.

#TODO

Linear transform of a random vector

Consider a random vector $\overrightarrow{X}$ with covariance matrix $\mathrm{\Sigma }$. Then, for any $k$ dimensional constant vector $\overrightarrow{c}$ and any $p\times k$-matrix $A$, the $k$ dimensional random vector $\overrightarrow{c}+A^T\overrightarrow{X}$ has mean $\overrightarrow{c}+A^{T}E\overrightarrow{X}$ and has covariance matrix $$\mathrm{Cov}(\overrightarrow{c}+A^T\overrightarrow{X})=A^T\mathrm{\Sigma }A\text{. }$$

The proof is quite simple:

Let $\overrightarrow{Y}=\overrightarrow{c}+A^T\overrightarrow{X}$, due to the linearity expectation operator, its expectation is $$\mathrm{E}[\overrightarrow{Y}]=\mathrm{E}[\overrightarrow{c}+A^T\overrightarrow{X}]=\overrightarrow{c}+A^T\mathrm{E}[\overrightarrow{X}].$$ Thus, $$\overrightarrow{Y}-\mathrm{E}[\overrightarrow{Y}]=(\overrightarrow{c}+A^T\overrightarrow{X})-(\overrightarrow{c}+A^T\mathrm{E}[\overrightarrow{X}])=A^T(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]).$$ Therefore, $$\begin{array}{rl}\mathrm{Cov}(\overrightarrow{c}+A^T\overrightarrow{X})& =\mathrm{Cov}\left(\overrightarrow{Y}\right)\\ & =\mathrm{E}[(\overrightarrow{Y}-\mathrm{E}[\overrightarrow{Y}])(\overrightarrow{Y}-\mathrm{E}[\overrightarrow{Y}]{)}^T]\\ & =\mathrm{E}[(A^T(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]))(A^T(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]){)}^T]\\ & =\mathrm{E}[A^T(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}])(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]{)}^{T}A]\\ & =A^T\mathrm{E}[(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}])(\overrightarrow{X}-\mathrm{E}[\overrightarrow{X}]{)}^T]A\\ & =A^T\mathrm{\Sigma }A\end{array}$$ Remember that $\mathrm{\Sigma }\triangleq \mathrm{Cov}\left(\overrightarrow{X}\right)$ .

What if all elements are independent?

If $X_1,X_2,\dots ,X_p$ are i.i.d. (independent identically distributed), then $\mathrm{Cov}\left({[X_1,X_2,\dots ,X_p]}^T\right)$​, or the covariance matrix $\mathrm{\Sigma }$, is a diagonal matrix with ${\sigma }^2$ on the diagonal and zeros elsewhere: $$\mathrm{\Sigma }=\left[\begin{array}{cccc}{\sigma }^2& 0& \cdots & 0\\ 0& {\sigma }^2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\sigma }^2\end{array}\right]={\sigma }^2{I}_p$$ where $I_p$ is the $p\times p$ identity matrix.

Proof:

• The diagonal elements ${\mathrm{\Sigma }}_{ii}$ represent the variance of each $X_i$ :

$${\mathrm{\Sigma }}_{ii}=\mathrm{Var}\left(X_i\right)={\sigma }^2\text{ for all }i$$

• The off-diagonal elements ${\mathrm{\Sigma }}_{ij}$ represent the covariance between different $X_i$ and $X_j$. Since $X_i$ and $X_j$ are independent, we have:

$${\mathrm{\Sigma }}_{ij}=\mathrm{Cov}(X_i,X_j)=0\text{ for }i\ne j$$