The Probability Density of a Multivariate Normal with a Low-Rank Covariance
In standard Multivariate Statistics, a $k$-variate normal distribution is considered non-degenerate only if its covariance matrix $\Sigma$ is positive definite. However, in 2026, many Machine Learning models utilize low-rank approximations to speed up computation. This leads to a "degenerate" MVN that lives on a lower-dimensional subspace of $\mathbb{R}^k$.
1. The Problem: The Standard PDF Fails
The classic PDF formula for $X \sim \mathcal{N}(\mu, \Sigma)$ is:
$$f(x) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}} \exp\left(-\frac{1}{2}(x - \mu)^\top \Sigma^{-1} (x - \mu)\right)$$If $\Sigma$ has rank $r < k$, then $|\Sigma| = 0$ and $\Sigma^{-1}$ is undefined. Mathematically, the distribution does not have a density with respect to the $k$-dimensional Lebesgue measure because all the probability mass is concentrated on an $r$-dimensional hyperplane.
2. The Solution: Generalized PDF
To define a density for a low-rank MVN, we must define it with respect to the Lebesgue measure on the $r$-dimensional subspace spanned by the eigenvectors of $\Sigma$. The generalized formula uses the Moore-Penrose Pseudo-Inverse ($\Sigma^+$) and the Pseudo-Determinant ($|\Sigma|_+$).
The Generalized Formula:
$$f(x) = \frac{1}{\sqrt{(2\pi)^r |\Sigma|_+}} \exp\left(-\frac{1}{2}(x - \mu)^\top \Sigma^+ (x - \mu)\right)$$- $r$: The rank of $\Sigma$.
- $|\Sigma|_+$: The product of the non-zero eigenvalues of $\Sigma$.
- $\Sigma^+$: The pseudo-inverse, calculated via Singular Value Decomposition (SVD).
3. Practical Implementation: Low-Rank Parameterization
In GIS and Gaussian Processes, we often parameterize $\Sigma$ as a low-rank matrix plus noise:
$$\Sigma = WW^\top + \sigma^2 I$$Where $W$ is a $k \times r$ matrix. This "Sherman-Morrison-Woodbury" approach ensures the matrix remains invertible (due to the $\sigma^2 I$ term), even if the underlying signal is low-rank. This is the preferred method for 2026 data pipelines to avoid numerical instability.
4. Comparison: Full-Rank vs. Low-Rank (Degenerate)
| Feature | Full-Rank MVN | Low-Rank (Degenerate) MVN |
|---|---|---|
| Support | Entire space $\mathbb{R}^k$ | Subspace of dimension $r$ |
| Determinant | Standard $|\Sigma| > 0$ | Pseudo-determinant $|\Sigma|_+$ |
| Inverse | Standard $\Sigma^{-1}$ | Moore-Penrose $\Sigma^+$ |
| Density Existence | Defined on $\mathbb{R}^k$ | Defined on the $r$-subspace only |
5. Key Constraints on Cross Validated
When discussing this on Cross Validated, remember these critical caveats:
- Support Requirement: The density $f(x)$ is only valid if $(x - \mu)$ lies in the range of $\Sigma$. If $x$ is outside the subspace where the distribution lives, the probability is zero.
- Mahalanobis Distance: The term $(x - \mu)^\top \Sigma^+ (x - \mu)$ is the generalized Mahalanobis distance. It correctly penalizes deviations only within the dimensions where variance exists.
Conclusion
Defining the probability density of a low-rank multivariate normal requires shifting your perspective from the full $k$-dimensional space to the active $r$-dimensional subspace. By replacing the determinant and inverse with their generalized counterparts, you can maintain statistical rigor even when your covariance matrix is singular. This technique is indispensable for high-dimensional Spatial Reconstruction where data redundancy is a feature, not a bug.
Keywords
low-rank multivariate normal density, degenerate MVN pdf formula, pseudo-determinant covariance matrix, Moore-Penrose pseudo-inverse MVN, multivariate normal singular covariance, spatial statistics low rank 2026, Cross Validated MVN tutorial, singular value decomposition covariance.
