kld function

Kullback-Leibler Divergence between Centered Multivariate Distributions

Computes the Kullback-Leibler divergence between two random vectors distributed according to centered multivariate distributions:

• multivariate generalized Gaussian distribution (MGGD) with zero mean vector, using the kldggd function
• multivariate Cauchy distribution (MCD) with zero location vector, using the kldcauchy function
• multivariate $t$ distribution (MTD) with zero mean vector, using the kldstudent function

One can also use one of the kldggd, kldcauchy or kldstudent functions, depending on the probability distribution.

kld(Sigma1, Sigma2, distribution = c("mggd", "mcd", "mtd"),
           beta1 = NULL, beta2 = NULL, nu1 = NULL, nu2 = NULL, eps = 1e-06)

Arguments

• Sigma1: symmetric, positive-definite matrix. The scatter matrix of the first distribution.
• Sigma2: symmetric, positive-definite matrix. The scatter matrix of the second distribution.
• distribution: the probability distribution. It can be "mggd" (multivariate generalized Gaussian distribution) "mcd" (multivariate Cauchy) or "mtd" (multivariate $t$).
• beta1, beta2: numeric. If distribution = "mggd", the shape parameters of the first and second distributions. NULL if distribution is "mcd" or "mtd".
• nu1, nu2: numeric. If distribution = "mtd", the degrees of freedom of the first and second distributions. NULL if distribution is "mggd" or "mcd".
• eps: numeric. Precision for the computation of the Lauricella $D$-hypergeometric function if distribution is "mggd" (see kldggd) or of its partial derivative if distribution = "mcd" or distribution = "mtd" (see kldcauchy or kldstudent). Default: 1e-06.

Returns

A numeric value: the Kullback-Leibler divergence between the two distributions, with two attributes attr(, "epsilon")

(precision of the Lauricella $D$-hypergeometric function or of its partial derivative) and attr(, "k") (number of iterations).

Examples

# Generalized Gaussian distributions
beta1 <- 0.74
beta2 <- 0.55
Sigma1 <- matrix(c(0.8, 0.3, 0.2, 0.3, 0.2, 0.1, 0.2, 0.1, 0.2), nrow = 3)
Sigma2 <- matrix(c(1, 0.3, 0.2, 0.3, 0.5, 0.1, 0.2, 0.1, 0.7), nrow = 3)
# Kullback-Leibler divergence
kl12 <- kld(Sigma1, Sigma2, "mggd", beta1 = beta1, beta2 = beta2)
kl21 <- kld(Sigma2, Sigma1, "mggd", beta1 = beta2, beta2 = beta1)
print(kl12)
print(kl21)
# Distance (symmetrized Kullback-Leibler divergence)
kldist <- as.numeric(kl12) + as.numeric(kl21)
print(kldist)

# Cauchy distributions
Sigma1 <- matrix(c(1, 0.6, 0.2, 0.6, 1, 0.3, 0.2, 0.3, 1), nrow = 3)
Sigma2 <- matrix(c(1, 0.3, 0.1, 0.3, 1, 0.4, 0.1, 0.4, 1), nrow = 3)
kld(Sigma1, Sigma2, "mcd")
kld(Sigma2, Sigma1, "mcd")

Sigma1 <- matrix(c(0.5, 0, 0, 0, 0.4, 0, 0, 0, 0.3), nrow = 3)
Sigma2 <- diag(1, 3)
# Case when all eigenvalues of Sigma1 %*% solve(Sigma2) are < 1
kld(Sigma1, Sigma2, "mcd")
# Case when all eigenvalues of Sigma1 %*% solve(Sigma2) are > 1
kld(Sigma2, Sigma1, "mcd")

# Student distributions
nu1 <- 2
Sigma1 <- matrix(c(2, 1.2, 0.4, 1.2, 2, 0.6, 0.4, 0.6, 2), nrow = 3)
nu2 <- 4
Sigma2 <- matrix(c(1, 0.3, 0.1, 0.3, 1, 0.4, 0.1, 0.4, 1), nrow = 3)
# Kullback-Leibler divergence
kld(Sigma1, Sigma2, "mtd", nu1 = nu1, nu2 = nu2)
kld(Sigma2, Sigma1, "mtd", nu1 = nu2, nu2 = nu1)

References

N. Bouhlel, A. Dziri, Kullback-Leibler Divergence Between Multivariate Generalized Gaussian Distributions. IEEE Signal Processing Letters, vol. 26 no. 7, July 2019. tools:::Rd_expr_doi("10.1109/LSP.2019.2915000")

N. Bouhlel, D. Rousseau, A Generic Formula and Some Special Cases for the Kullback–Leibler Divergence between Central Multivariate Cauchy Distributions. Entropy, 24, 838, July 2022. tools:::Rd_expr_doi("10.3390/e24060838")

N. Bouhlel and D. Rousseau (2023), Exact Rényi and Kullback-Leibler Divergences Between Multivariate t-Distributions, IEEE Signal Processing Letters. tools:::Rd_expr_doi("10.1109/LSP.2023.3324594")

Author(s)

Pierre Santagostini, Nizar Bouhlel