Deriving conditional multivariate normal distribution

Let random vector $\mathbf{x}$ follows a multivariate normal distribution $\mathbf{x}\sim N(\mathit{\mu },\mathbf{\Sigma })$. Consider the conditional distribution of ${\mathbf{x}}_{\mathbf{1}}|{\mathbf{x}}_{\mathbf{2}}$ where ${\mathbf{x}}_{\mathbf{1}}$ and ${\mathbf{x}}_{\mathbf{2}}$ are partitions of the random vector $\mathbf{x}$. We first create partitions as follows.

$$\mathbf{x}=\left[\begin{array}{c}{\mathbf{x}}_{\mathbf{1}}\\ {\mathbf{x}}_{\mathbf{2}}\end{array}\right]\mathit{\mu }=\left[\begin{array}{c}{\mu }_{\mathbf{1}}\\ {\mu }_{\mathbf{2}}\end{array}\right]\mathbf{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_{11}& {\mathrm{\Sigma }}_{12}\\ {\mathrm{\Sigma }}_{21}& {\mathrm{\Sigma }}_{22}\end{array}\right]$$

Then, ${\mathbf{x}}_{\mathbf{1}}|{\mathbf{x}}_{\mathbf{2}}\sim N(\tilde{\mathit{\mu }},\tilde{\mathbf{\Sigma }})$ where $\tilde{\mathit{\mu }}$ and $\tilde{\mathbf{\Sigma }}$ is given by,

$$\tilde{\mathit{\mu }}={\mu }_1+{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}({\mathbf{x}}_{\mathbf{2}}-{\mathit{\mu }}_2)$$ $$\tilde{\mathbf{\Sigma }}={\mathrm{\Sigma }}_{11}-{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}$$

Proof

It is known that conditional distribution of multivaritate normal, conditioned on a subset of variables are again multivariate normal. Thus, we can verify the above theorem by deriving the conditional mean and covariace. Assuming ${\mathrm{\Sigma }}_{22}$ is invertible, define a new variable $\mathbf{z}={\mathbf{x}}_{\mathbf{1}}+A{\mathbf{x}}_{\mathbf{2}}$, where $A=-{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}$.

First notice that $\mathbf{z}$ and ${\mathbf{x}}_{\mathbf{2}}$ has zero covariance, $$\begin{array}{rl}\text{Cov}(\mathbf{z},{\mathbf{x}}_2)& =\text{Cov}({\mathbf{x}}_1+A{\mathbf{x}}_2,{\mathbf{x}}_2)\\ & =\text{Cov}({\mathbf{x}}_1,{\mathbf{x}}_2)+\text{Cov}(A{\mathbf{x}}_2,{\mathbf{x}}_2)\\ & =\text{Cov}({\mathbf{x}}_1,{\mathbf{x}}_2)+A\text{Cov}({\mathbf{x}}_2,{\mathbf{x}}_2)\\ & ={\mathrm{\Sigma }}_{12}+A{\mathrm{\Sigma }}_{22}\\ & ={\mathrm{\Sigma }}_{12}+-{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{22}\\ & ={\mathrm{\Sigma }}_{12}-{\mathrm{\Sigma }}_{12}\\ & =0\end{array}$$

Since $\mathbf{z}$ and $\mathbf{x}$ are jointly normal and uncorrelated, they are independent by the property of mulitvariate normal. Now we can easily find the conditional mean as follows.

$$\begin{array}{rl}E({\mathbf{x}}_1|{\mathbf{x}}_2)& =E(\mathbf{z}-A{\mathbf{x}}_2|{\mathbf{x}}_2)\\ & =E(\mathbf{z}|{\mathbf{x}}_2)-AE({\mathbf{x}}_2|{\mathbf{x}}_2)\\ & =E(\mathbf{z})-A{\mathbf{x}}_2(\because {\mathbf{x}}_2\perp \perp \mathbf{z})\\ & =E({\mathbf{x}}_1+A{\mathbf{x}}_2)-A{\mathbf{x}}_2\\ & =E({\mathbf{x}}_1)+AE({\mathbf{x}}_2)-A{\mathbf{x}}_2\\ & ={\mu }_1+A{\mu }_2-A{\mathbf{x}}_2\\ & ={\mu }_1-A({\mathbf{x}}_2-{\mu }_2)\\ & ={\mu }_1+{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}({\mathbf{x}}_2-{\mu }_2)\\ & =\tilde{\mathit{\mu }}\end{array}$$

Before jumping into conditional variance. Recall the following facts for random vectors $\mathbf{x},\mathbf{y}$ and non-random matrix $A,B$,

$$\begin{array}{rl}\text{Var}(\mathbf{x}+\mathbf{y})& =\text{Var}(\mathbf{x})+\text{Var}(\mathbf{y})+\text{Cov}(\mathbf{x},\mathbf{y})+\text{Cov}(\mathbf{y},\mathbf{x})\\ \text{Cov}(A\mathbf{x},B\mathbf{y})& =A\text{Cov}(\mathbf{x},\mathbf{y})B^T\end{array}$$

Using the above fact, we can expand the conditional variance as follows.

$$\begin{array}{rl}\text{Var}({\mathbf{x}}_1|{\mathbf{x}}_2)& =\text{Var}(\mathbf{z}-A{\mathbf{x}}_2|{\mathbf{x}}_2)\\ & =\text{Var}(\mathbf{z}|{\mathbf{x}}_2)+\text{Var}(-A{\mathbf{x}}_2|{\mathbf{x}}_2)+\text{Cov}(\mathbf{z},-A{\mathbf{x}}_2|{\mathbf{x}}_2)+\text{Cov}(-A{\mathbf{x}}_2,\mathbf{z}|{\mathbf{x}}_2)\end{array}$$

Since $-A{\mathbf{x}}_2|{\mathbf{x}}_2$ is not random anymore given ${\mathbf{x}}_2$, we are only left with the first term.

$$\begin{array}{rl}& =\text{Var}(\mathbf{z}|{\mathbf{x}}_2)\\ & =\text{Var}(\mathbf{z})(\because {\mathbf{x}}_2\perp \perp \mathbf{z})\\ & =\text{Var}({\mathbf{x}}_1+A{\mathbf{x}}_2)\\ & =\text{Var}({\mathbf{x}}_1)+\text{Var}(A{\mathbf{x}}_2)+\text{Cov}({\mathbf{x}}_1,A{\mathbf{x}}_2)+\text{Cov}(A{\mathbf{x}}_2,{\mathbf{x}}_1)\\ & =\text{Var}({\mathbf{x}}_1)+A\text{Var}({\mathbf{x}}_2)A^T+\text{Cov}({\mathbf{x}}_1,{\mathbf{x}}_2)A^T+A\text{Cov}({\mathbf{x}}_2,{\mathbf{x}}_1)\\ & ={\mathrm{\Sigma }}_{11}+A{\mathrm{\Sigma }}_{22}{A}^T+{\mathrm{\Sigma }}_{12}{A}^T+A{\mathrm{\Sigma }}_{21}\\ & ={\mathrm{\Sigma }}_{11}-A{\mathrm{\Sigma }}_{22}({\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{)}^T+A{\mathrm{\Sigma }}_{21}+A{\mathrm{\Sigma }}_{21}\\ & ={\mathrm{\Sigma }}_{11}-A{\mathrm{\Sigma }}_{22}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}+2A{\mathrm{\Sigma }}_{21}\\ & ={\mathrm{\Sigma }}_{11}-A{\mathrm{\Sigma }}_{21}+2A{\mathrm{\Sigma }}_{21}\\ & ={\mathrm{\Sigma }}_{11}+A{\mathrm{\Sigma }}_{21}\\ & ={\mathrm{\Sigma }}_{11}-{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}\\ & =\tilde{\mathbf{\Sigma }}\end{array}$$

Therefore, the proof is done.

References

(https://stats.stackexchange.com/users/4856/macro), Macro. n.d. “Deriving the Conditional Distributions of a Multivariate Normal Distribution.” Cross Validated. https://stats.stackexchange.com/q/30600.