Indexof

Lite v2.0Cross Validated › Defining the Low-Rank Multivariate Normal Density | Cross Validated Guide › Last update: About

Defining the Low-Rank Multivariate Normal Density | Cross Validated Guide

The Probability Density of a Multivariate Normal with a Low-Rank Covariance

In standard Multivariate Statistics, a $k$-variate normal distribution is considered non-degenerate only if its covariance matrix $\Sigma$ is positive definite. However, in 2026, many Machine Learning models utilize low-rank approximations to speed up computation. This leads to a "degenerate" MVN that lives on a lower-dimensional subspace of $\mathbb{R}^k$.

1. The Problem: The Standard PDF Fails

The classic PDF formula for $X \sim \mathcal{N}(\mu, \Sigma)$ is:

$$f(x) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}} \exp\left(-\frac{1}{2}(x - \mu)^\top \Sigma^{-1} (x - \mu)\right)$$

If $\Sigma$ has rank $r < k$, then $|\Sigma| = 0$ and $\Sigma^{-1}$ is undefined. Mathematically, the distribution does not have a density with respect to the $k$-dimensional Lebesgue measure because all the probability mass is concentrated on an $r$-dimensional hyperplane.

2. The Solution: Generalized PDF

To define a density for a low-rank MVN, we must define it with respect to the Lebesgue measure on the $r$-dimensional subspace spanned by the eigenvectors of $\Sigma$. The generalized formula uses the Moore-Penrose Pseudo-Inverse ($\Sigma^+$) and the Pseudo-Determinant ($|\Sigma|_+$).

The Generalized Formula:

$$f(x) = \frac{1}{\sqrt{(2\pi)^r |\Sigma|_+}} \exp\left(-\frac{1}{2}(x - \mu)^\top \Sigma^+ (x - \mu)\right)$$
  • $r$: The rank of $\Sigma$.
  • $|\Sigma|_+$: The product of the non-zero eigenvalues of $\Sigma$.
  • $\Sigma^+$: The pseudo-inverse, calculated via Singular Value Decomposition (SVD).

3. Practical Implementation: Low-Rank Parameterization

In GIS and Gaussian Processes, we often parameterize $\Sigma$ as a low-rank matrix plus noise:

$$\Sigma = WW^\top + \sigma^2 I$$

Where $W$ is a $k \times r$ matrix. This "Sherman-Morrison-Woodbury" approach ensures the matrix remains invertible (due to the $\sigma^2 I$ term), even if the underlying signal is low-rank. This is the preferred method for 2026 data pipelines to avoid numerical instability.

4. Comparison: Full-Rank vs. Low-Rank (Degenerate)

Feature Full-Rank MVN Low-Rank (Degenerate) MVN
Support Entire space $\mathbb{R}^k$ Subspace of dimension $r$
Determinant Standard $|\Sigma| > 0$ Pseudo-determinant $|\Sigma|_+$
Inverse Standard $\Sigma^{-1}$ Moore-Penrose $\Sigma^+$
Density Existence Defined on $\mathbb{R}^k$ Defined on the $r$-subspace only

5. Key Constraints on Cross Validated

When discussing this on Cross Validated, remember these critical caveats:

  • Support Requirement: The density $f(x)$ is only valid if $(x - \mu)$ lies in the range of $\Sigma$. If $x$ is outside the subspace where the distribution lives, the probability is zero.
  • Mahalanobis Distance: The term $(x - \mu)^\top \Sigma^+ (x - \mu)$ is the generalized Mahalanobis distance. It correctly penalizes deviations only within the dimensions where variance exists.

Conclusion

Defining the probability density of a low-rank multivariate normal requires shifting your perspective from the full $k$-dimensional space to the active $r$-dimensional subspace. By replacing the determinant and inverse with their generalized counterparts, you can maintain statistical rigor even when your covariance matrix is singular. This technique is indispensable for high-dimensional Spatial Reconstruction where data redundancy is a feature, not a bug.

Keywords

low-rank multivariate normal density, degenerate MVN pdf formula, pseudo-determinant covariance matrix, Moore-Penrose pseudo-inverse MVN, multivariate normal singular covariance, spatial statistics low rank 2026, Cross Validated MVN tutorial, singular value decomposition covariance.

Profile: Learn how to define and compute the probability density function (PDF) of a degenerate multivariate normal distribution using pseudo-inverses and pseudo-determinants. - Indexof

About

Learn how to define and compute the probability density function (PDF) of a degenerate multivariate normal distribution using pseudo-inverses and pseudo-determinants. #cross-validated #definingthelowrankmultivariatenormaldensity


Edited by: Bilal Fellaki, Sara Hauksdottir, Liva Roed & Nnaemeka Mgbemena

Close [x]
Loading special offers...

Suggestion