On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid

Abstract

The Jensen–Shannon divergence is a renown bounded symmetrization of the Kullback–Leibler divergence which does not require probability densities to have matching supports. In this paper, we introduce a vector-skew generalization of the scalar $\alpha$-Jensen–Bregman divergences and derive thereof the vector-skew $\alpha$-Jensen–Shannon divergences. We prove that the vector-skew $\alpha$-Jensen–Shannon divergences are f-divergences and study the properties of these novel divergences. Finally, we report an iterative algorithm to numerically compute the Jensen–Shannon-type centroids for a set of probability densities belonging to a mixture family: This includes the case of the Jensen–Shannon centroid of a set of categorical distributions or normalized histograms.

1. Introduction

Let $(\mathcal{X},\mathcal{F},\mu )$ be a measure space [1] where $\mathcal{X}$ denotes the sample space, $\mathcal{F}$ the $\sigma$-algebra of measurable events, and $\mu$ a positive measure; for example, the measure space defined by the Lebesgue measure ${\mu }_L$ with Borel $\sigma$-algebra $\mathcal{B}\left({\mathbb{R}}^d\right)$ for $\mathcal{X}={\mathbb{R}}^d$ or the measure space defined by the counting measure ${\mu }_c$ with the power set $\sigma$-algebra $2^{\mathcal{X}}$ on a finite alphabet $\mathcal{X}$. Denote by $L_1(\mathcal{X},\mathcal{F},\mu )$ the Lebesgue space of measurable functions, ${\mathcal{P}}_1$ the subspace of positive integrable functions f such that ${\int }_{\mathcal{X}}f\left(x\right)\mathrm{d}\mu \left(x\right)=1$ and $f\left(x\right)>0$ for all $x\in \mathcal{X}$, and ${\overline{\mathcal{P}}}_1$ the subspace of non-negative integrable functions f such that ${\int }_{\mathcal{X}}f\left(x\right)\mathrm{d}\mu \left(x\right)=1$ and $f\left(x\right)\ge 0$ for all $x\in \mathcal{X}$.

We refer to the book of Deza and Deza [2] and the survey of Basseville [3] for an introduction to the many types of statistical divergences met in information sciences and their justifications. The Kullback–Leibler Divergence (KLD) $\mathrm{KL}:{\mathcal{P}}_1\times {\mathcal{P}}_1\to [0,\infty ]$ is an oriented statistical distance (commonly called the relative entropy in information theory [4]) defined between two densities p and q (i.e., the Radon–Nikodym densities of $\mu$-absolutely continuous probability measures P and Q) by

$$\mathrm{KL}(p:q):=\int plog\frac{p}{q}\mathrm{d}\mu .$$

(1)

Although $\mathrm{KL}(p:q)\ge 0$ with equality iff. $p=q$$\mu$-a. e. (Gibb’s inequality [4]), the KLD may diverge to infinity depending on the underlying densities. Since the KLD is asymmetric, several symmetrizations [5] have been proposed in the literature.

A well-grounded symmetrization of the KLD is the Jensen–Shannon Divergence [6] (JSD), also called capacitory discrimination in the literature (e.g., see [7]):

$$\begin{array}{ccc}\hfill \mathrm{JS}(p,q)& :=& \frac{1}{2}\left(\mathrm{KL}\left(p:\frac{p+q}{2}\right)+\mathrm{KL}\left(q:\frac{p+q}{2}\right)\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}=& \frac{1}{2}\int \left(plog\frac{2p}{p+q}+qlog\frac{2q}{p+q}\right)\mathrm{d}\mu =\mathrm{JS}(q,p).\hfill \end{array}$$

(3)

The Jensen–Shannon divergence can be interpreted as the total KL divergence to the average distribution $\frac{p+q}{2}$. The Jensen–Shannon divergence was historically implicitly introduced in [8] (Equation (19)) to calculate distances between random graphs. A nice feature of the Jensen–Shannon divergence is that this divergence can be applied to densities with arbitrary support (i.e., $p,q\in {\overline{\mathcal{P}}}_1$ with the convention that $0log0=0$ and $log\frac{0}{0}=0$); moreover, the JSD is always upper bounded by $log2$. Let ${\mathcal{X}}_p=\mathrm{supp}\left(p\right)$ and ${\mathcal{X}}_q=\mathrm{supp}\left(q\right)$ denote the supports of the densities p and q, respectively, where $\mathrm{supp}\left(p\right):=\{x\in \mathcal{X}:p(x)>0\}$. The JSD saturates to $log2$ whenever the supports ${\mathcal{X}}_p$ and ${\mathcal{X}}_p$ are disjoints. We can rewrite the JSD as

$$\mathrm{JS}(p,q)=h\left(\frac{p+q}{2}\right)-\frac{h\left(p\right)+h\left(q\right)}{2},$$

(4)

where $h\left(p\right)=-\int plogp\mathrm{d}\mu$ denotes Shannon’s entropy. Thus, the JSD can also be interpreted as the entropy of the average distribution minus the average of the entropies.

The square root of the JSD is a metric [9] satisfying the triangle inequality, but the square root of the JD is not a metric (nor any positive power of the Jeffreys divergence, see [10]). In fact, the JSD can be interpreted as a Hilbert metric distance, meaning that there exists some isometric embedding of $(\mathcal{X},\sqrt{\mathrm{JS}})$ into a Hilbert space [11,12]. Other principled symmetrizations of the KLD have been proposed in the literature: For example, Naghshvar et al. [13] proposed the extrinsic Jensen–Shannon divergence and demonstrated its use for variable-length coding over a discrete memoryless channel (DMC).

Another symmetrization of the KLD sometimes met in the literature [14,15,16] is the Jeffreys divergence [17,18] (JD) defined by

$$J(p,q):=\mathrm{KL}(p:q)+\mathrm{KL}(q:p)=\int (p-q)log\frac{p}{q}\mathrm{d}\mu =J(q,p).$$

(5)

However, we point out that this Jeffreys divergence lacks sound information-theoretical justifications.

For two positive but not necessarily normalized densities $\tilde{p}$ and $\tilde{q}$, we define the extended Kullback–Leibler divergence as follows:

$$\begin{array}{ccc}\hfill {\mathrm{KL}}^{+}(\tilde{p}:\tilde{q})& :=& \mathrm{KL}(\tilde{p}:\tilde{q})+\int \tilde{q}\mathrm{d}\mu -\int \tilde{p}\mathrm{d}\mu ,\hfill \end{array}$$

(6)

$$\begin{array}{cc}=& \int \left(\tilde{p}log\frac{\tilde{p}}{\tilde{q}}+\tilde{q}-\tilde{p}\right)\mathrm{d}\mu .\hfill \end{array}$$

(7)

The Jensen–Shannon divergence and the Jeffreys divergence can both be extended to positive (unnormalized) densities without changing their formula expressions:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{+}(\tilde{p},\tilde{q})& :=& \frac{1}{2}\left({\mathrm{KL}}^{+}\left(\tilde{p}:\frac{\tilde{p}+\tilde{q}}{2}\right)+{\mathrm{KL}}^{+}\left(\tilde{q}:\frac{\tilde{p}+\tilde{q}}{2}\right)\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}=& \frac{1}{2}\left(\mathrm{KL}\left(\tilde{p}:\frac{\tilde{p}+\tilde{q}}{2}\right)+\mathrm{KL}\left(\tilde{q}:\frac{\tilde{p}+\tilde{q}}{2}\right)\right)=\mathrm{JS}(\tilde{p},\tilde{q}),\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill J^{+}(\tilde{p},\tilde{q})& :=& {\mathrm{KL}}^{+}(\tilde{p}:\tilde{q})+{\mathrm{KL}}^{+}(\tilde{p}:\tilde{q})=\int (\tilde{p}-\tilde{q})log\frac{\tilde{p}}{\tilde{q}}\mathrm{d}\mu =J(\tilde{p},\tilde{q}).\hfill \end{array}$$

(10)

However, the extended ${\mathrm{JS}}^{+}$ divergence is upper-bounded by $(\frac{1}{2}log2)(\int (\tilde{p}+\tilde{q})\mathrm{d}\mu )=\frac{1}{2}(\mu \left(p\right)+\mu \left(q\right))log2$ instead of $log2$ for normalized densities (i.e., when $\mu \left(p\right)+\mu \left(q\right)=2$).

Let ${\left(pq\right)}_{\alpha }\left(x\right):=(1-\alpha )p\left(x\right)+\alpha q\left(x\right)$ denote the statistical weighted mixture with component densities p and q for $\alpha \in [0,1]$. The asymmetric $\alpha$-skew Jensen–Shannon divergence can be defined for a scalar parameter $\alpha \in (0,1)$ by considering the weighted mixture ${\left(pq\right)}_{\alpha }$ as follows:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_a^{\alpha }(p:q)& :=& (1-\alpha )\mathrm{KL}(p:{\left(pq\right)}_{\alpha })+\alpha \mathrm{KL}(q:{\left(pq\right)}_{\alpha }),\hfill \end{array}$$

(11)

$$\begin{array}{cc}=& (1-\alpha )\int plog\frac{p}{{\left(pq\right)}_{\alpha }}\mathrm{d}\mu +\alpha \int qlog\frac{q}{{\left(pq\right)}_{\alpha }}\mathrm{d}\mu .\hfill \end{array}$$

(12)

Let us introduce the α-skew K-divergence [6,19] $K_{\alpha }(p:q)$ by:

$$K_{\alpha }\left(p:q\right):=\mathrm{KL}\left(p:(1-\alpha )p+\alpha q\right)=\mathrm{KL}\left(p:{\left(pq\right)}_{\alpha }\right).$$

(13)

Then, both the Jensen–Shannon divergence and the Jeffreys divergence can be rewritten [20] using $K_{\alpha }$ as follows:

$$\begin{array}{ccc}\hfill \mathrm{JS}\left(p,q\right)& =& \frac{1}{2}\left(K_{\frac{1}{2}}\left(p:q\right)+K_{\frac{1}{2}}\left(q:p\right)\right),\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill J\left(p,q\right)& =& K_1(p:q)+K_1(q:p),\hfill \end{array}$$

(15)

since ${\left(pq\right)}_1=q$, $\mathrm{KL}(p:q)=K_1(p:q)$ and ${\left(pq\right)}_{\frac{1}{2}}={\left(qp\right)}_{\frac{1}{2}}$.

We can thus define the symmetric α-skew Jensen–Shannon divergence [20] for $\alpha \in (0,1)$ as follows:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{\alpha }(p,q)& :=& \frac{1}{2}{K}_{\alpha }(p:q)+\frac{1}{2}{K}_{\alpha }(q:p)={\mathrm{JS}}^{\alpha }(q,p).\hfill \end{array}$$

(16)

The ordinary Jensen–Shannon divergence is recovered for $\alpha =\frac{1}{2}$.

In general, skewing divergences (e.g., using the divergence $K_{\alpha }$ instead of the KLD) have been experimentally shown to perform better in applications like in some natural language processing (NLP) tasks [21].

The α-Jensen–Shannon divergences are Csiszár f-divergences [22,23,24]. An f-divergence is defined for a convex function f, strictly convex at 1, and satisfies $f\left(1\right)=0$ as:

$$I_f(p:q)=\int q\left(x\right)f\left(\frac{p\left(x\right)}{q\left(x\right)}\right)\mathrm{d}x\ge f\left(1\right)=0.$$

(17)

We can always symmetrize f-divergences by taking the conjugate convex function $f^{*}\left(x\right)=xf\left(\frac{1}{x}\right)$ (related to the perspective function): $I_{f+f^{*}}(p,q)$ is a symmetric divergence. The f-divergences are convex statistical distances which are provably the only separable invariant divergences in information geometry [25], except for binary alphabets $\mathcal{X}$ (see [26]).

The Jeffreys divergence is an f-divergence for the generator $f\left(x\right)=(x-1)logx$, and the $\alpha$-Jensen–Shannon divergences are f-divergences for the generator family $f_{\alpha }\left(x\right)=-log((1-\alpha )+\alpha x)-xlog((1-\alpha )+\frac{\alpha }{x})$. The f-divergences are upper-bounded by $f\left(0\right)+f^{*}\left(0\right)$. Thus, the f-divergences are finite when $f\left(0\right)+f^{*}\left(0\right)<\infty$.

The main contributions of this paper are summarized as follows:

• First, we generalize the Jensen–Bregman divergence by skewing a weighted separable Jensen–Bregman divergence with a k-dimensional vector $\alpha \in {[0,1]}^k$ in Section 2. This yields a generalization of the symmetric skew $\alpha$-Jensen–Shannon divergences to a vector-skew parameter. This extension retains the key properties for being upper-bounded and for application to densities with potentially different supports. The proposed generalization also allows one to grasp a better understanding of the “mechanism” of the Jensen–Shannon divergence itself. We also show how to directly obtain the weighted vector-skew Jensen–Shannon divergence from the decomposition of the KLD as the difference of the cross-entropy minus the entropy (i.e., KLD as the relative entropy).
• Second, we prove that weighted vector-skew Jensen–Shannon divergences are f-divergences (Theorem 1), and show how to build families of symmetric Jensen–Shannon-type divergences which can be controlled by a vector of parameters in Section 2.3, generalizing the work of [20] from scalar skewing to vector skewing. This may prove useful in applications by providing additional tuning parameters (which can be set, for example, by using cross-validation techniques).
• Third, we consider the calculation of the Jensen–Shannon centroids in Section 3 for densities belonging to mixture families. Mixture families include the family of categorical distributions and the family of statistical mixtures sharing the same prescribed components. Mixture families are well-studied manifolds in information geometry [25]. We show how to compute the Jensen–Shannon centroid using a concave–convex numerical iterative optimization procedure [27]. The experimental results graphically compare the Jeffreys centroid with the Jensen–Shannon centroid for grey-valued image histograms.

2. Extending the Jensen–Shannon Divergence

2.1. Vector-Skew Jensen–Bregman Divergences and Jensen Diversities

Recall our notational shortcut: ${\left(ab\right)}_{\alpha }:=(1-\alpha )a+\alpha b$. For a k-dimensional vector $\alpha \in {[0,1]}^k$, a weight vector w belonging to the $(k-1)$-dimensional open simplex ${\Delta }_k$, and a scalar $\gamma \in (0,1)$, let us define the following vector skew α-Jensen–Bregman divergence ($\alpha$-JBD) following [28]:

$${\mathrm{JB}}_F^{\alpha ,\gamma ,w}({\theta }_1:{\theta }_2):={\displaystyle \sum _{i=1}^k}{w}_i{B}_F\left({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}:{\left({\theta }_1{\theta }_2\right)}_{\gamma }\right)\ge 0,$$

(18)

where $B_F$ is the Bregman divergence [29] induced by a strictly convex and smooth generator F:

$$B_F({\theta }_1:{\theta }_2):=F\left({\theta }_1\right)-F\left({\theta }_2\right)-\langle {\theta }_1-{\theta }_2,\nabla F\left({\theta }_2\right)\rangle ,$$

(19)

with $\langle ·,·\rangle$ denoting the Euclidean inner product $\langle x,y\rangle =x^{\top }y$ (dot product). Expanding the Bregman divergence formulas in the expression of the $\alpha$-JBD and using the fact that

$${\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}-{\left({\theta }_1{\theta }_2\right)}_{\gamma }=(\gamma -{\alpha }_i)({\theta }_1-{\theta }_2),$$

(20)

we get the following expression:

$${\mathrm{JB}}_F^{\alpha ,\gamma ,w}({\theta }_1:{\theta }_2)=\left({\displaystyle \sum _{i=1}^k}{w}_{i}F\left({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}\right)\right)-F\left({\left({\theta }_1{\theta }_2\right)}_{\gamma }\right)-〈{\displaystyle \sum _{i=1}^k}{w}_i(\gamma -{\alpha }_i)({\theta }_1-{\theta }_2),\nabla F\left({\left({\theta }_1{\theta }_2\right)}_{\gamma }\right)〉.$$

(21)

The inner product term of Equation (21) vanishes when

$$\gamma ={\displaystyle \sum _{i=1}^k}{w}_i{\alpha }_i:=\overline{\alpha }.$$

(22)

Thus, when $\gamma =\overline{\alpha }$ (assuming at least two distinct components in $\alpha$ so that $\gamma \in (0,1)$), we get the simplified formula for the vector-skew $\alpha$-JBD:

$$\begin{array}{|c|}\hline {\mathrm{JB}}_F^{\alpha ,w}({\theta }_1:{\theta }_2)=\left({\displaystyle \sum _{i=1}^k}{w}_{i}F\left({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}\right)\right)-F\left({\left({\theta }_1{\theta }_2\right)}_{\overline{\alpha }}\right).\\ \hline\end{array}$$

(23)

This vector-skew Jensen–Bregman divergence is always finite and amounts to a Jensen diversity [30] $J_F$ induced by Jensen’s inequality gap:

$${\mathrm{JB}}_F^{\alpha ,w}({\theta }_1:{\theta }_2)=J_F({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_1},\dots ,{\left({\theta }_1{\theta }_2\right)}_{{\alpha }_k};w_1,\dots ,w_k):={\displaystyle \sum _{i=1}^k}{w}_{i}F\left({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}\right)-F\left({\left({\theta }_1{\theta }_2\right)}_{\overline{\alpha }}\right)\ge 0.$$

(24)

The Jensen diversity is a quantity which arises as a generalization of the cluster variance when clustering with Bregman divergences instead of the ordinary squared Euclidean distance; see [29,30] for details. In the context of Bregman clustering, the Jensen diversity has been called the Bregman information [29] and motivated by rate distortion theory: Bregman information measures the minimum expected loss when encoding a set of points using a single point when the loss is measured using a Bregman divergence. In general, a k-point measure is called a diversity measure (for $k>2$), while a distance/divergence is the special case of a 2-point measure.

Conversely, in 1D, we may start from Jensen’s inequality for a strictly convex function F:

$${\displaystyle \sum _{i=1}^k}{w}_{i}F\left({\theta }_i\right)\ge F\left({\displaystyle \sum _{i=1}^k}{w}_i{\theta }_i\right).$$

(25)

Let us notationally write $\left[k\right]:=\{1,\dots ,k\}$, and define ${\theta }_m:={min}_{i\in \left[k\right]}{\left\{{\theta }_i\right\}}_i$ and ${\theta }_M:={max}_{i\in \left[k\right]}{\left\{{\theta }_i\right\}}_i>{\theta }_m$ (i.e., assuming at least two distinct values). We have the barycenter $\overline{\theta }={\sum }_i{w}_i{\theta }_i=:{\left({\theta }_m{\theta }_M\right)}_{\gamma }$ which can be interpreted as the linear interpolation of the extremal values for some $\gamma \in (0,1)$. Let us write ${\theta }_i={\left({\theta }_m{\theta }_M\right)}_{{\alpha }_i}$ for $i\in \left[k\right]$ and proper values of the ${\alpha }_i$s. Then, it comes that

$$\begin{array}{ccc}\hfill \overline{\theta }& =& \sum _i{w}_i{\theta }_i,\hfill \end{array}$$

(26)

$$\begin{array}{cc}=& \sum _i{w}_i{\left({\theta }_m{\theta }_M\right)}_{{\alpha }_i},\hfill \end{array}$$

(27)

$$\begin{array}{cc}=& \sum _i{w}_i((1-{\alpha }_i){\theta }_m+{\alpha }_i{\theta }_M),\hfill \end{array}$$

(28)

$$\begin{array}{cc}=& \left(1-\sum _i{w}_i{\alpha }_i\right){\theta }_m+\sum _i{\alpha }_i{w}_i{\theta }_M,\hfill \end{array}$$

(29)

$$\begin{array}{cc}=& {\left({\theta }_m{\theta }_M\right)}_{{\sum }_i{w}_i{\alpha }_i}={\left({\theta }_m{\theta }_M\right)}_{\gamma },\hfill \end{array}$$

(30)

so that $\gamma ={\sum }_i{w}_i{\alpha }_i=\overline{\alpha }$.

2.2. Vector-Skew Jensen–Shannon Divergences

Let $f\left(x\right)=xlogx-x$ be a strictly smooth convex function on $(0,\infty )$. Then, the Bregman divergence induced by this univariate generator is

$$B_f(p:q)=plog\frac{p}{q}+q-p={\mathrm{kl}}_{+}(p:q),$$

(31)

the extended scalar Kullback–Leibler divergence.

We extend the scalar-skew Jensen–Shannon divergence as follows: ${\mathrm{JS}}^{\alpha ,w}(p:q):={\mathrm{JB}}_{-h}^{\alpha ,\overline{\alpha },w}(p:q)$ for h, the Shannon’s entropy [4] (a strictly concave function [4]).

Definition 1

(Weighted vector-skew $(\alpha ,w)$-Jensen–Shannon divergence). For a vector $\alpha \in {[0,1]}^k$ and a unit positive weight vector $w\in {\Delta }_k$, the $(\alpha ,w)$-Jensen–Shannon divergence between two densities $p,q\in {\overline{\mathcal{P}}}_1$ is defined by:

$$\begin{array}{|c|}\hline {\mathrm{JS}}^{\alpha ,w}(p:q):={\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\overline{\alpha }})=h\left({\left(pq\right)}_{\overline{\alpha }}\right)-{\displaystyle \sum _{i=1}^k}{w}_{i}h\left({\left(pq\right)}_{{\alpha }_i}\right),\\ \hline\end{array}$$

with $\overline{\alpha }={\sum }_{i=1}^k{w}_i{\alpha }_i$, where $h\left(p\right)=-\int p\left(x\right)logp\left(x\right)\mathrm{d}\mu \left(x\right)$ denotes the Shannon entropy [4] (i.e., $-h$ is strictly convex).

This definition generalizes the ordinary JSD; we recover the ordinary Jensen–Shannon divergence when $k=2$, ${\alpha }_1=0$, ${\alpha }_2=1$, and $w_1=w_2=\frac{1}{2}$ with $\overline{\alpha }=\frac{1}{2}$: $\mathrm{JS}(p,q)={\mathrm{JS}}^{(0,1),(\frac{1}{2},\frac{1}{2})}(p:q)$.

Let ${\mathrm{KL}}_{\alpha ,\beta }(p:q):=\mathrm{KL}({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta })$. Then, we have ${\mathrm{KL}}_{\alpha ,\beta }(q:p)={\mathrm{KL}}_{1-\alpha ,1-\beta }(p:q)$. Using this $(\alpha ,\beta )$-KLD, we have the following identity:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{\alpha ,w}(p:q)& =& {\displaystyle \sum _{i=1}^k}{w}_i{\mathrm{KL}}_{{\alpha }_i,\overline{\alpha }}(p:q),\hfill \end{array}$$

(32)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i{\mathrm{KL}}_{1-{\alpha }_i,1-\overline{\alpha }}(q:p)={\mathrm{JS}}^{1_k-\alpha ,w}(q:p),\hfill \end{array}$$

(33)

since ${\sum }_{i=1}^k{w}_i(1-{\alpha }_i)=\overline{1_k-\alpha }=1-\overline{\alpha }$, where $1_k=(1,\dots ,1)$ is a k-dimensional vector of ones.

A very interesting property is that the vector-skew Jensen–Shannon divergences are f-divergences [22].

Theorem 1.

The vector-skew Jensen–Shannon divergences ${\mathrm{JS}}^{\alpha ,w}(p:q)$ are f-divergences for the generator $f_{\alpha ,w}\left(u\right)={\sum }_{i=1}^k{w}_i({\alpha }_{i}u+(1-{\alpha }_i))log\frac{(1-{\alpha }_i)+{\alpha }_{i}u}{(1-\overline{\alpha })+\overline{\alpha }u}$ with $\overline{\alpha }={\sum }_{i=1}^k{w}_i{\alpha }_i$.

Proof. 

First, let us observe that the positively weighted sum of f-divergences is an f-divergence: ${\sum }_{i=1}^k{w}_i{I}_{f_i}(p:q)=I_f(p:q)$ for the generator $f\left(u\right)={\sum }_{i=1}^k{w}_i{f}_i\left(u\right)$.

Now, let us express the divergence ${\mathrm{KL}}_{\alpha ,\beta }(p:q)$ as an f-divergence:

$${\mathrm{KL}}_{\alpha ,\beta }(p:q)=I_{f_{\alpha ,\beta }}(p:q),$$

(34)

with generator

$$f_{\alpha ,\beta }\left(u\right)=(\alpha u+1-\alpha )log\frac{(1-\alpha )+\alpha u}{(1-\beta )+\beta u}.$$

(35)

Thus, it follows that

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{\alpha ,w}(p:q)& =& {\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\overline{\alpha }}),\hfill \end{array}$$

(36)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i{I}_{f_{{\alpha }_i,\overline{\alpha }}}(p:q),\hfill \end{array}$$

(37)

$$\begin{array}{cc}=& I_{{\sum }_{i=1}^k{w}_i{f}_{{\alpha }_i,\overline{\alpha }}}(p:q).\hfill \end{array}$$

(38)

Therefore, the vector-skew Jensen–Shannon divergence is an f-divergence for the following generator:

$$\begin{array}{|c|}\hline f_{\alpha ,w}\left(u\right)={\displaystyle \sum _{i=1}^k}{w}_i({\alpha }_{i}u+(1-{\alpha }_i))log\frac{(1-{\alpha }_i)+{\alpha }_{i}u}{(1-\overline{\alpha })+\overline{\alpha }u},\\ \hline\end{array}$$

(39)

where $\overline{\alpha }={\sum }_{i=1}^k{w}_i{\alpha }_i$.

When $\alpha =(0,1)$ and $w=(\frac{1}{2},\frac{1}{2})$, we recover the f-divergence generator for the JSD:

$$\begin{array}{ccc}\hfill f_{\mathrm{JS}}\left(u\right)& =& \frac{1}{2}log\frac{1}{\frac{1}{2}+\frac{1}{2}u}+\frac{1}{2}ulog\frac{u}{\frac{1}{2}+\frac{1}{2}u},\hfill \end{array}$$

(40)

$$\begin{array}{cc}=& \frac{1}{2}\left(log\frac{2}{1+u}+ulog\frac{2u}{1+u}\right).\hfill \end{array}$$

(41)

Observe that $f_{\alpha ,w}^{*}\left(u\right)=u{f}_{\alpha ,w}(1/u)=f_{1-\alpha ,w}\left(u\right)$, where $1-\alpha :=(1-{\alpha }_1,\dots ,1-{\alpha }_k)$.

We also refer the reader to Theorem 4.1 of [31], which defines skew f-divergences from any f-divergence.  □

Remark 1.

Since the vector-skew Jensen divergence is an f-divergence, we easily obtain Fano and Pinsker inequalities following [32], or reverse Pinsker inequalities following [33,34] (i.e., upper bounds for the vector-skew Jensen divergences using the total variation metric distance), data processing inequalities using [35], etc.

Next, we show that ${\mathrm{KL}}_{\alpha ,\beta }$ (and ${\mathrm{JS}}^{\alpha ,w}$) are separable convex divergences. Since the f-divergences are separable convex, the ${\mathrm{KL}}_{\alpha ,\beta }$ divergences and the ${\mathrm{JS}}^{\alpha ,w}$ divergences are separable convex. For the sake of completeness, we report a simplex explicit proof below.

Theorem 2

(Separable convexity). The divergence ${\mathrm{KL}}_{\alpha ,\beta }(p:q)$ is strictly separable convex for $\alpha \ne \beta$ and $x\in {\mathcal{X}}_p\cap {\mathcal{X}}_q$.

Proof. 

Let us calculate the second partial derivative of ${\mathrm{KL}}_{\alpha ,\beta }(x:y)$ with respect to x, and show that it is strictly positive:

$$\frac{{\partial }^2}{\partial x^2}{\mathrm{KL}}_{\alpha ,\beta }(x:y)=\frac{{(\beta -\alpha )}^2{y}^2}{{\left(xy\right)}_{\alpha }{\left(xy\right)}_{\beta }^2}>0,$$

(42)

for $x,y>0$. Thus, ${\mathrm{KL}}_{\alpha ,\beta }$ is strictly convex on the left argument. Similarly, since ${\mathrm{KL}}_{\alpha ,\beta }(y:x)={\mathrm{KL}}_{1-\alpha ,1-\beta }(x:y)$, we deduce that ${\mathrm{KL}}_{\alpha ,\beta }$ is strictly convex on the right argument. Therefore, the divergence ${\mathrm{KL}}_{\alpha ,\beta }$ is separable convex.  □

It follows that the divergence ${\mathrm{JS}}^{\alpha ,w}(p:q)$ is strictly separable convex, since it is a convex combination of weighted ${\mathrm{KL}}_{{\alpha }_i,\overline{\alpha }}$ divergences.

Another way to derive the vector-skew JSD is to decompose the KLD as the difference of the cross-entropy $h^{\times }$ minus the entropy h (i.e., KLD is also called the relative entropy):

$$\mathrm{KL}(p:q)=h^{\times }(p:q)-h\left(p\right),$$

(43)

where $h^{\times }(p:q):=-\int plogq\mathrm{d}\mu$ and $h\left(p\right):=h^{\times }(p:p)$ (self cross-entropy). Since ${\alpha }_1{h}^{\times }(p_1:q)+{\alpha }_2{h}^{\times }(p_2:q)=h^{\times }({\alpha }_1{p}_1+{\alpha }_2{p}_2:q)$ (for ${\alpha }_2=1-{\alpha }_1$), it follows that

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{\alpha ,w}(p:q)& :=& {\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\gamma }),\hfill \end{array}$$

(44)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i\left(h^{\times }({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\gamma })-h\left({\left(pq\right)}_{{\alpha }_i}\right)\right),\hfill \end{array}$$

(45)

$$\begin{array}{cc}=& h^{\times }\left({\displaystyle \sum _{i=1}^k}{w}_i{\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\gamma }\right)-{\displaystyle \sum _{i=1}^k}{w}_{i}h\left({\left(pq\right)}_{{\alpha }_i}\right).\hfill \end{array}$$

(46)

Here, the “trick” is to choose $\gamma =\overline{\alpha }$ in order to “convert” the cross-entropy into an entropy: $h^{\times }({\sum }_{i=1}^k{w}_i{\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\gamma })=h\left({\left(pq\right)}_{\overline{\alpha }}\right)$ when $\gamma =\overline{\alpha }$. Then, we end up with

$$\begin{array}{|c|}\hline {\mathrm{JS}}^{\alpha ,w}(p:q)=h\left({\left(pq\right)}_{\overline{\alpha }}\right)-{\displaystyle \sum _{i=1}^k}{w}_{i}h\left({\left(pq\right)}_{{\alpha }_i}\right).\\ \hline\end{array}$$

(47)

When $\alpha =({\alpha }_1,{\alpha }_2)$ with ${\alpha }_1=0$ and ${\alpha }_2=0$ and $w=(w_1,w_2)=(\frac{1}{2},\frac{1}{2})$, we have $\overline{\alpha }=\frac{1}{2}$, and we recover the Jensen–Shannon divergence:

$$\mathrm{JS}(p:q)=h\left(\frac{p+q}{2}\right)-\frac{h\left(p\right)+h\left(q\right)}{2}.$$

(48)

Notice that Equation (13) is the usual definition of the Jensen–Shannon divergence, while Equation (48) is the reduced formula of the JSD, which can be interpreted as a Jensen gap for Shannon entropy, hence its name: The Jensen–Shannon divergence.

Moreover, if we consider the cross-entropy/entropy extended to positive densities $\tilde{p}$ and $\tilde{q}$:

$$h_{+}^{\times }(\tilde{p}:\tilde{q})=-\int (\tilde{p}log\tilde{q}+\tilde{q})\mathrm{d}\mu ,h_{+}\left(\tilde{p}\right)=h_{+}^{\times }(\tilde{p}:\tilde{p})=-\int (\tilde{p}log\tilde{p}+\tilde{p})\mathrm{d}\mu ,$$

(49)

we get:

$${\mathrm{JS}}_{+}^{\alpha ,w}(\tilde{p}:\tilde{q})={\displaystyle \sum _{i=1}^k}{w}_i{\mathrm{KL}}_{+}({\left(\tilde{p}\tilde{q}\right)}_{{\alpha }_i}:{\left(\tilde{p}\tilde{q}\right)}_{\gamma })=h_{+}\left({\left(\tilde{p}\tilde{q}\right)}_{\overline{\alpha }}\right)-{\displaystyle \sum _{i=1}^k}{w}_i{h}_{+}\left({\left(\tilde{p}\tilde{q}\right)}_{{\alpha }_i}\right).$$

(50)

Next, we shall prove that our generalization of the skew Jensen–Shannon divergence to vector-skewing is always bounded. We first start by a lemma bounding the KLD between two mixtures sharing the same components:

Lemma 1

(KLD between two w-mixtures). For $\alpha \in [0,1]$ and $\beta \in (0,1)$, we have:

$${\mathrm{KL}}_{\alpha ,\beta }(p:q)=\mathrm{KL}\left({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta }\right)\le logmax\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}.$$

Proof. 

For $p\left(x\right),q\left(x\right)>0$, we have

$$\frac{(1-\alpha )p\left(x\right)+\alpha q\left(x\right)}{(1-\beta )p\left(x\right)+\beta q\left(x\right)}\le max\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}.$$

(51)

Indeed, by considering the two cases $\alpha \ge \beta$ (or equivalently, $1-\alpha \le 1-\beta$) and $\alpha \le \beta$ (or equivalently, $1-\alpha \ge 1-\beta$), we check that $(1-\alpha )p\left(x\right)\le max\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}(1-\beta )p\left(x\right)$ and $\alpha q\left(x\right)\le max\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}\beta q\left(x\right)$. Thus, we have $\frac{(1-\alpha )p\left(x\right)+\alpha q\left(x\right)}{(1-\beta )p\left(x\right)+\beta q\left(x\right)}\le max\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}$. Therefore, it follows that:

$$\mathrm{KL}\left({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta }\right)\le \int {\left(pq\right)}_{\alpha }logmax\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}\mathrm{d}\mu =logmax\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}.$$

(52)

Notice that we can interpret $logmax\left\{\frac{1-\alpha }{1-\beta },\frac{\alpha }{\beta }\right\}=max\{log\frac{1-\alpha }{1-\beta },log\frac{\alpha }{\beta }\}$ as the ∞-Rényi divergence [36,37] between the following two two-point distributions: $(\alpha ,1-\alpha )$ and $(\beta ,1-\beta )$. See Theorem 6 of [36].

A weaker upper bound is $\mathrm{KL}({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta })\le log\frac{1}{\beta (1-\beta )}$. Indeed, let us form a partition of the sample space $\mathcal{X}$ into two dominance regions:

• $R_p:=\{x\in \mathcal{X}:q\left(x\right)\le p\left(x\right)\}$ and
• $R_q:=\{x\in \mathcal{X}:q\left(x\right)>p\left(x\right)\}$.

We have ${\left(pq\right)}_{\alpha }\left(x\right)=(1-\alpha )p\left(x\right)+\alpha q\left(x\right)\le p\left(x\right)$ for $x\in R_p$ and ${\left(pq\right)}_{\alpha }\left(x\right)\le q\left(x\right)$ for $x\in R_q$. It follows that

$$\mathrm{KL}\left({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta }\right)\le {\int }_{R_p}{\left(pq\right)}_{\alpha }\left(x\right)log\frac{p\left(x\right)}{(1-\beta )p\left(x\right)}\mathrm{d}\mu \left(x\right)+{\int }_{R_q}{\left(pq\right)}_{\alpha }\left(x\right)log\frac{q\left(x\right)}{\beta q\left(x\right)}\mathrm{d}\mu \left(x\right).$$

That is, $\mathrm{KL}({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta })\le -log(1-\beta )-log\beta =log\frac{1}{\beta (1-\beta )}$. Notice that we allow $\alpha \in \{0,1\}$ but not $\beta$ to take the extreme values (i.e., $\beta \in (0,1)$).  □

In fact, it is known that for both $\alpha ,\beta \in (0,1)$, $\mathrm{KL}\left({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\beta }\right)$ amount to compute a Bregman divergence for the Shannon negentropy generator, since $\{{\left(pq\right)}_{\gamma }:\gamma \in (0,1)\}$ defines a mixture family [38] of order 1 in information geometry. Hence, it is always finite, as Bregman divergences are always finite (but not necessarily bounded).

By using the fact that

$${\mathrm{JS}}^{\alpha ,w}(p:q)={\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}\left({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\overline{\alpha }}\right),$$

(53)

we conclude that the vector-skew Jensen–Shannon divergence is upper-bounded:

Lemma 2

(Bounded $(w,\alpha )$-Jensen–Shannon divergence). ${\mathrm{JS}}^{\alpha ,w}$ is bounded by $log\frac{1}{\overline{\alpha }(1-\overline{\alpha })}$ where $\overline{\alpha }={\sum }_{i=1}^k{w}_i{\alpha }_i\in (0,1)$.

Proof. 

We have ${\mathrm{JS}}^{\alpha ,w}(p:q)={\sum }_i{w}_i\mathrm{KL}\left({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\overline{\alpha }}\right)$. Since $0\le \mathrm{KL}\left({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{\overline{\alpha }}\right)\le log\frac{1}{\overline{\alpha }(1-\overline{\alpha })}$, it follows that we have

$$0\le {\mathrm{JS}}^{\alpha ,w}(p:q)\le log\frac{1}{\overline{\alpha }(1-\overline{\alpha })}.$$

Notice that we also have

$${\mathrm{JS}}^{\alpha ,w}(p:q)\le \sum _i{w}_{i}logmax\left\{\frac{1-{\alpha }_i}{1-\overline{\alpha }},\frac{{\alpha }_i}{\overline{\alpha }}\right\}.$$

 □

The vector-skew Jensen–Shannon divergence is symmetric if and only if for each index $i\in \left[k\right]$ there exists a matching index $\sigma \left(i\right)$ such that ${\alpha }_{\sigma \left(i\right)}=1-{\alpha }_i$ and $w_{\sigma \left(i\right)}=w_i$.

For example, we may define the symmetric scalar α-skew Jensen–Shannon divergence as

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_s^{\alpha }(p,q)& =& \frac{1}{2}\mathrm{KL}({\left(pq\right)}_{\alpha }:{\left(pq\right)}_{\frac{1}{2}})+\frac{1}{2}\mathrm{KL}({\left(pq\right)}_{1-\alpha }:{\left(pq\right)}_{\frac{1}{2}}),\hfill \end{array}$$

(54)

$$\begin{array}{cc}=& \frac{1}{2}\int {\left(pq\right)}_{\alpha }log\frac{{\left(pq\right)}_{\alpha }}{{\left(pq\right)}_{\frac{1}{2}}}\mathrm{d}\mu +\frac{1}{2}\int {\left(pq\right)}_{1-\alpha }log\frac{{\left(pq\right)}_{1-\alpha }}{{\left(pq\right)}_{\frac{1}{2}}}\mathrm{d}\mu ,\hfill \end{array}$$

(55)

$$\begin{array}{cc}=& \frac{1}{2}\int {\left(qp\right)}_{1-\alpha }log\frac{{\left(qp\right)}_{1-\alpha }}{{\left(qp\right)}_{\frac{1}{2}}}\mathrm{d}\mu ++\frac{1}{2}\int {\left(qp\right)}_{\alpha }log\frac{{\left(qp\right)}_{\alpha }}{{\left(qp\right)}_{\frac{1}{2}}}\mathrm{d}\mu ,\hfill \end{array}$$

(56)

$$\begin{array}{cc}=& h\left({\left(pq\right)}_{\frac{1}{2}}\right)-\frac{h\left({\left(pq\right)}_{\alpha }\right)+h\left({\left(pq\right)}_{1-\alpha }\right)}{2},\hfill \end{array}$$

(57)

$$\begin{array}{cc}=:& {\mathrm{JS}}_s^{\alpha }(q,p),\hfill \end{array}$$

(58)

since it holds that ${\left(ab\right)}_c={\left(ba\right)}_{1-c}$ for any $a,b,c\in \mathbb{R}$. Note that ${\mathrm{JS}}_s^{\alpha }(p,q)\ne {\mathrm{JS}}^{\alpha }(p,q)$.

Remark 2.

We can always symmetrize a vector-skew Jensen–Shannon divergence by doubling the dimension of the skewing vector. Let $\alpha =({\alpha }_1,\cdots ,{\alpha }_k)$ and w be the vector parameters of an asymmetric vector-skew JSD, and consider ${\alpha }^{\prime }=(1-{\alpha }_1,\dots ,1-{\alpha }_k)$ and w to be the parameters of ${\mathrm{JS}}^{{\alpha }^{\prime },w}$. Then, ${\mathrm{JS}}^{(\alpha ,{\alpha }^{\prime }),(\frac{w}{2},\frac{w}{2})}$ is a symmetric skew-vector JSD:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{(\alpha ,{\alpha }^{\prime }),(\frac{w}{2},\frac{w}{2})}(p:q)& :=& \frac{1}{2}{\mathrm{JS}}^{\alpha ,w}(p:q)+\frac{1}{2}{\mathrm{JS}}^{{\alpha }^{\prime },w}(p:q),\hfill \end{array}$$

(59)

$$\begin{array}{cc}=& \frac{1}{2}{\mathrm{JS}}^{\alpha ,w}(p:q)+\frac{1}{2}{\mathrm{JS}}^{\alpha ,w}(q:p)={\mathrm{JS}}^{(\alpha ,{\alpha }^{\prime }),(\frac{w}{2},\frac{w}{2})}(q:p).\hfill \end{array}$$

(60)

Since the vector-skew Jensen–Shannon divergence is an f-divergence for the generator $f_{\alpha ,w}$ (Theorem 1), we can take generator $f_{w,\alpha }^s\left(u\right)=\frac{f_{w,\alpha }\left(u\right)+f_{w,\alpha }^{*}\left(u\right)}{2}$ to define the symmetrized f-divergence, where $f_{w,\alpha }^{*}\left(u\right)=u{f}_{w,\alpha }\left(\frac{1}{u}\right)$ denotes the convex conjugate function. When $f_{\alpha ,w}$ yields a symmetric f-divergence $I_{f_{\alpha ,w}}$, we can apply the generic upper bound of f-divergences (i.e., $I_f\le f\left(0\right)+f^{*}\left(0\right)$) to get the upper bound on the symmetric vector-skew Jensen–Shannon divergences:

$$\begin{array}{ccc}\hfill I_{f_{\alpha ,w}}(p:q)& \le & f_{\alpha ,w}\left(0\right)+f_{\alpha ,w}^{*}\left(0\right),\hfill \end{array}$$

(61)

$$\begin{array}{cc}\le & {\displaystyle \sum _{i=1}^k}{w}_i\left((1-{\alpha }_i)log\frac{1-{\alpha }_i}{1-\overline{\alpha }}+{\alpha }_{i}log\frac{{\alpha }_i}{\overline{\alpha }}\right),\hfill \end{array}$$

(62)

since

$$\begin{array}{ccc}\hfill f_{\alpha ,w}^{*}\left(u\right)& =& u{f}_{\alpha ,w}\left(\frac{1}{u}\right),\hfill \end{array}$$

(63)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i((1-{\alpha }_i)u+{\alpha }_i)log\frac{(1-{\alpha }_i)u+{\alpha }_i}{(1-\overline{\alpha })u+\overline{\alpha }}.\hfill \end{array}$$

(64)

For example, consider the ordinary Jensen–Shannon divergence with $w=\left(\frac{1}{2},\frac{1}{2}\right)$ and $\alpha =(0,1)$. Then, we find $\mathrm{JS}(p:w)=I_{f_{(0,1),\left(\frac{1}{2},\frac{1}{2}\right)}}(p:q)\le \frac{1}{2}log2+\frac{1}{2}log2=log2$, the usual upper bound of the JSD.

As a side note, let us notice that our notation ${\left(pq\right)}_{\alpha }$ allows one to compactly write the following property:

Property 1.

We have $q={\left(qq\right)}_{\lambda }$ for any $\lambda \in [0,1]$, and ${\left({\left(p_1{p}_2\right)}_{\lambda }{\left(q_1{q}_2\right)}_{\lambda }\right)}_{\alpha }={\left({\left(p_1{q}_1\right)}_{\alpha }{\left(p_2{q}_2\right)}_{\alpha }\right)}_{\lambda }$ for any $\alpha ,\lambda \in [0,1]$.

Proof. 

Clearly, $q=(1-\lambda )q+\lambda q=:\left({\left(qq\right)}_{\lambda }\right)$ for any $\lambda \in [0,1]$. Now, we have

$$\begin{array}{ccc}\hfill {\left({\left(p_1{p}_2\right)}_{\lambda }{\left(q_1{q}_2\right)}_{\lambda }\right)}_{\alpha }& =& (1-\alpha ){\left(p_1{p}_2\right)}_{\lambda }+\alpha {\left(q_1{q}_2\right)}_{\lambda },\hfill \end{array}$$

(65)

$$\begin{array}{cc}=& (1-\alpha )((1-\lambda )p_1+\lambda p_2)+\alpha ((1-\lambda )q_1+\lambda q_2),\hfill \end{array}$$

(66)

$$\begin{array}{cc}=& (1-\lambda )((1-\alpha )p_1+\alpha q_1)+\lambda ((1-\alpha )p_2+\alpha q_2),\hfill \end{array}$$

(67)

$$\begin{array}{cc}=& (1-\lambda ){\left(p_1{q}_1\right)}_{\alpha }+\lambda {\left(p_2{q}_2\right)}_{\alpha },\hfill \end{array}$$

(68)

$$\begin{array}{cc}=& {\left({\left(p_1{q}_1\right)}_{\alpha }{\left(p_2{q}_2\right)}_{\alpha }\right)}_{\lambda }.\hfill \end{array}$$

(69)

 □

2.3. Building Symmetric Families of Vector-Skewed Jensen–Shannon Divergences

We can build infinitely many vector-skew Jensen–Shannon divergences. For example, consider $\alpha =\left(0,1,\frac{1}{3}\right)$ and $w=\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$. Then, $\overline{\alpha }=\frac{1}{3}+\frac{1}{9}=\frac{4}{9}$, and

$${\mathrm{JS}}^{\alpha ,w}(p:q)=h\left({\left(pq\right)}_{\frac{4}{9}}\right)-\frac{h\left(p\right)+h\left(q\right)+h\left({\left(pq\right)}_{\frac{1}{3}}\right)}{3}\ne {\mathrm{JS}}^{\alpha ,w}(q:p).$$

(70)

Interestingly, we can also build infinitely many families of symmetric vector-skew Jensen–Shannon divergences. For example, consider these two examples that illustrate the construction process:

• Consider $k=2$. Let $(w,1-w)$ denote the weight vector, and $\alpha =({\alpha }_1,{\alpha }_2)$ the skewing vector. We have $\overline{\alpha }=w{\alpha }_1+(1-w){\alpha }_2={\alpha }_2+w({\alpha }_1-{\alpha }_2)$. The vector-skew JSD is symmetric iff. $w=1-w=\frac{1}{2}$ (with $\overline{\alpha }=\frac{{\alpha }_1+{\alpha }_2}{2}$) and ${\alpha }_2=1-{\alpha }_1$. In that case, we have $\overline{\alpha }=\frac{1}{2}$, and we obtain the following family of symmetric Jensen–Shannon divergences:

  $$\begin{array}{ccc}\hfill {\mathrm{JS}}^{(\alpha ,1-\alpha ),(\frac{1}{2},\frac{1}{2})}(p,q)& =& h\left({\left(pq\right)}_{\frac{1}{2}}\right)-\frac{h\left({\left(pq\right)}_{\alpha }\right)+h\left({\left(pq\right)}_{1-\alpha }\right)}{2},\hfill \end{array}$$

  (71)

  $$\begin{array}{cc}=& h\left({\left(pq\right)}_{\frac{1}{2}}\right)-\frac{h\left({\left(pq\right)}_{\alpha }\right)+h\left({\left(qp\right)}_{\alpha }\right)}{2}={\mathrm{JS}}^{(\alpha ,1-\alpha ),(\frac{1}{2},\frac{1}{2})}(q,p).\hfill \end{array}$$

  (72)
• Consider $k=4$, weight vector $w=\left(\frac{1}{3},\frac{1}{3},\frac{1}{6},\frac{1}{6}\right)$, and skewing vector $\alpha =({\alpha }_1,1-{\alpha }_1,{\alpha }_2,1-{\alpha }_2)$ for ${\alpha }_1,{\alpha }_2\in (0,1)$. Then, $\overline{\alpha }=\frac{1}{2}$, and we get the following family of symmetric vector-skew JSDs:

  $$\begin{array}{ccc}\hfill {\mathrm{JS}}^{({\alpha }_1,{\alpha }_2)}(p,q)& =& h\left({\left(pq\right)}_{\frac{1}{2}}\right)-\frac{2h\left({\left(pq\right)}_{{\alpha }_1}\right)+2h\left({\left(pq\right)}_{1-{\alpha }_1}\right)+h\left({\left(pq\right)}_{{\alpha }_2}\right)+h\left({\left(pq\right)}_{1-{\alpha }_2}\right)}{6},\hfill \end{array}$$

  (73)

  $$\begin{array}{cc}=& h\left({\left(pq\right)}_{\frac{1}{2}}\right)-\frac{2h\left({\left(pq\right)}_{{\alpha }_1}\right)+2h\left({\left(qp\right)}_{{\alpha }_1}\right)+h\left({\left(pq\right)}_{{\alpha }_2}\right)+h\left({\left(qp\right)}_{{\alpha }_2}\right)}{6},\hfill \end{array}$$

  (74)

  $$\begin{array}{cc}=& {\mathrm{JS}}^{({\alpha }_1,{\alpha }_2)}(q,p).\hfill \end{array}$$

  (75)
• We can similarly carry on the construction of such symmetric JSDs by increasing the dimensionality of the skewing vector.

In fact, we can define

$${\mathrm{JS}}_s^{\alpha ,w}(p,q):=h\left({\left(pq\right)}_{\frac{1}{2}}\right)-{\displaystyle \sum _{i=1}^k}{w}_i\frac{h\left({\left(pq\right)}_{{\alpha }_i}\right)+h\left({\left(pq\right)}_{1-{\alpha }_i}\right)}{2}={\displaystyle \sum _{i=1}^k}{w}_i{\mathrm{JS}}_s^{{\alpha }_i}(p,q),$$

(76)

with

$${\mathrm{JS}}_s^{\alpha }(p,q):=h\left({\left(pq\right)}_{\frac{1}{2}}\right)-\frac{h\left({\left(pq\right)}_{\alpha }\right)+h\left({\left(pq\right)}_{1-\alpha }\right)}{2}.$$

(77)

3. Jensen–Shannon Centroids on Mixture Families

3.1. Mixture Families and Jensen–Shannon Divergences

Consider a mixture family in information geometry [25]. That is, let us give a prescribed set of $D+1$ linearly independent probability densities $p_0\left(x\right),\dots ,p_D\left(x\right)$ defined on the sample space $\mathcal{X}$. A mixture family $\mathcal{M}$ of order D consists of all strictly convex combinations of these component densities:

$$\mathcal{M}:=\left\{m(x;\theta ):={\displaystyle \sum _{i=1}^D}{\theta }^i{p}_i\left(x\right)+\left(1-{\displaystyle \sum _{i=1}^D}{\theta }^i\right)p_0\left(x\right):{\theta }^i>0,{\displaystyle \sum _{i=1}^D}{\theta }^i<1\right\}.$$

(78)

For example, the family of categorical distributions (sometimes called “multinouilli” distributions) is a mixture family [25]:

$$\mathcal{M}=\left\{m_{\theta }\left(x\right)={\displaystyle \sum _{i=1}^D}{\theta }_i\delta (x-x_i)+\left(1-{\displaystyle \sum _{i=1}^D}{\theta }_i\right)\delta (x-x_0)\right\},$$

(79)

where $\delta \left(x\right)$ is the Dirac distribution (i.e., $\delta \left(x\right)=1$ for $x=0$ and $\delta \left(x\right)=0$ for $x\ne 0$). Note that the mixture family of categorical distributions can also be interpreted as an exponential family.

Notice that the linearly independent assumption on probability densities is to ensure to have an identifiable model: $\theta \leftrightarrow m(x;\theta )$.

The KL divergence between two densities of a mixture family $\mathcal{M}$ amounts to a Bregman divergence for the Shannon negentropy generator $F\left(\theta \right)=-h\left(m_{\theta }\right)$ (see [38]):

$$\begin{array}{c}\hfill \mathrm{KL}(m_{{\theta }_1}:m_{{\theta }_2})=B_F({\theta }_1:{\theta }_2)=B_{-h\left(m_{\theta }\right)}({\theta }_1:{\theta }_2).\end{array}$$

(80)

On a mixture manifold $\mathcal{M}$, the mixture density $(1-\alpha )m_{{\theta }_1}+\alpha m_{{\theta }_2}$ of two mixtures $m_{{\theta }_1}$ and $m_{{\theta }_2}$ of $\mathcal{M}$ also belongs to $\mathcal{M}$:

$$(1-\alpha )m_{{\theta }_1}+\alpha m_{{\theta }_2}=m_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}\in \mathcal{M},$$

(81)

where we extend the notation ${\left({\theta }_1{\theta }_2\right)}_{\alpha }:=(1-\alpha ){\theta }_1+\alpha {\theta }_2$ to vectors ${\theta }_1$ and ${\theta }_2$: ${\left({\theta }_1{\theta }_2\right)}_{\alpha }^i={\left({\theta }_1^i{\theta }_2^i\right)}_{\alpha }$.

Thus, the vector-skew JSD amounts to a vector-skew Jensen diversity for the Shannon negentropy convex function $F\left(\theta \right)=-h\left(m_{\theta }\right)$:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{\alpha ,w}(m_{{\theta }_1}:m_{{\theta }_2})& =& {\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}\left({\left(m_{{\theta }_1}{m}_{{\theta }_2}\right)}_{{\alpha }_i}:{\left(m_{{\theta }_1}{m}_{{\theta }_2}\right)}_{\overline{\alpha }}\right),\hfill \end{array}$$

(82)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}\left(m_{{\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}}:m_{{\left({\theta }_1{\theta }_2\right)}_{\overline{\alpha }}}\right),\hfill \end{array}$$

(83)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i{B}_F\left({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}:{\left({\theta }_1{\theta }_2\right)}_{\overline{\alpha }}\right),\hfill \end{array}$$

(84)

$$\begin{array}{cc}=& {\mathrm{JB}}_F^{\alpha ,\overline{\alpha },w}({\theta }_1:{\theta }_2),\hfill \end{array}$$

(85)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_{i}F\left({\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}\right)-F\left({\left({\theta }_1{\theta }_2\right)}_{\overline{\alpha }}\right),\hfill \end{array}$$

(86)

$$\begin{array}{cc}=& h\left(m_{{\left({\theta }_1{\theta }_2\right)}_{\overline{\alpha }}}\right)-{\displaystyle \sum _{i=1}^k}{w}_{i}h\left(m_{{\left({\theta }_1{\theta }_2\right)}_{{\alpha }_i}}\right).\hfill \end{array}$$

(87)

3.2. Jensen–Shannon Centroids

Given a set of n mixture densities $m_{{\theta }_1},\dots ,m_{{\theta }_n}$ of $\mathcal{M}$, we seek to calculate the skew-vector Jensen–Shannon centroid (or barycenter for non-uniform weights) defined as $m_{{\theta }^{*}}$, where ${\theta }^{*}$ is the minimizer of the following objective function (or loss function):

$$L\left(\theta \right):={\displaystyle \sum _{j=1}^n}{\omega }_j{\mathrm{JS}}^{\alpha ,w}(m_{{\theta }_k}:m_{\theta }),$$

(88)

where $\omega \in {\Delta }_n$ is the weight vector of densities (uniform weight for the centroid and non-uniform weight for a barycenter). This definition of the skew-vector Jensen–Shannon centroid is a generalization of the Fréchet mean (the Fréchet mean may not be unique, as it is the case on the sphere for two antipodal points for which their Fréchet means with respect to the geodesic metric distance form a great circle) [39] to non-metric spaces. Since the divergence ${\mathrm{JS}}^{\alpha ,w}$ is strictly separable convex, it follows that the Jensen–Shannon-type centroids are unique when they exist.

Plugging Equation (82) into Equation (88), we get that the calculation of the Jensen–Shannon centroid amounts to the following minimization problem:

$$L\left(\theta \right)={\displaystyle \sum _{j=1}^n}{\omega }_j\left({\displaystyle \sum _{i=1}^k}{w}_{i}F\left({\left({\theta }_j\theta \right)}_{{\alpha }_i}\right)-F\left({\left({\theta }_j\theta \right)}_{\overline{\alpha }}\right)\right).$$

(89)

This optimization is a Difference of Convex (DC) programming optimization, for which we can use the ConCave–Convex procedure [27,40] (CCCP). Indeed, let us define the following two convex functions:

$$\begin{array}{ccc}\hfill A\left(\theta \right)& =& {\displaystyle \sum _{j=1}^n}{\displaystyle \sum _{i=1}^k}{\omega }_j{w}_{i}F\left({\left({\theta }_j\theta \right)}_{{\alpha }_i}\right),\hfill \end{array}$$

(90)

$$\begin{array}{ccc}\hfill B\left(\theta \right)& =& {\displaystyle \sum _{j=1}^n}{\omega }_{j}F\left({\left({\theta }_j\theta \right)}_{\overline{\alpha }}\right).\hfill \end{array}$$

(91)

Both functions $A\left(\theta \right)$ and $B\left(\theta \right)$ are convex since F is convex. Then, the minimization problem of Equation (89) to solve can be rewritten as:

$$\underset{\theta }{min}A\left(\theta \right)-B\left(\theta \right).$$

(92)

This is a DC programming optimization problem which can be solved iteratively by initializing $\theta$ to an arbitrary value ${\theta }^{\left(0\right)}$ (say, the centroid of the ${\theta }_i$s), and then by updating the parameter at step t using the CCCP [27] as follows:

$${\theta }^{(t+1)}={(\nabla B)}^{-1}(\nabla A\left({\theta }^{\left(t\right)}\right)).$$

(93)

Compared to a gradient descent local optimization, there is no required step size (also called “learning” rate) in CCCP.

We have $\nabla A\left(\theta \right)={\sum }_{j=1}^n{\sum }_{i=1}^k{\omega }_j{w}_i{\alpha }_i\nabla F\left({\left({\theta }_j\theta \right)}_{{\alpha }_i}\right)$ and $\nabla B\left(\theta \right)={\sum }_{j=1}^n{\omega }_j\overline{\alpha }\nabla F\left({\left({\theta }_j\theta \right)}_{\overline{\alpha }}\right)$.

The CCCP converges to a local optimum ${\theta }^{*}$ where the support hyperplanes of the function graphs of A and B at ${\theta }^{*}$ are parallel to each other, as depicted in Figure 1. The set of stationary points is $\{\theta :\nabla A(\theta )=\nabla B(\theta \left)\right\}$. In practice, the delicate step is to invert $\nabla B$. Next, we show how to implement this algorithm for the Jensen–Shannon centroid of a set of categorical distributions (i.e., normalized histograms with all non-empty bins).

3.2.1. Jensen–Shannon Centroids of Categorical Distributions

To illustrate the method, let us consider the mixture family of categorical distributions [25]:

$$\mathcal{M}=\left\{m_{\theta }\left(x\right)={\displaystyle \sum _{i=1}^D}{\theta }_i\delta (x-x_i)+\left(1-{\displaystyle \sum _{i=1}^D}{\theta }_i\right)\delta (x-x_0)\right\}.$$

(94)

The Shannon negentropy is

$$F\left(\theta \right)=-h\left(m_{\theta }\right)={\displaystyle \sum _{i=1}^D}{\theta }_{i}log{\theta }_i+\left(1-{\displaystyle \sum _{i=1}^D}{\theta }_i\right)log\left(1-{\displaystyle \sum _{i=1}^D}{\theta }_i\right).$$

(95)

We have the partial derivatives

$$\nabla F\left(\theta \right)={\left[\frac{\partial }{\partial {\theta }_i}\right]}_i,\frac{\partial }{\partial {\theta }_i}F\left(\theta \right)=log\frac{{\theta }_i}{1-{\sum }_{j=1}^D{\theta }_j}.$$

(96)

Inverting the gradient $\nabla F$ requires us to solve the equation $\nabla F\left(\theta \right)=\eta$ so that we get $\theta ={(\nabla F)}^{-1}\left(\eta \right)$. We find that

$$\nabla F^{*}\left(\eta \right)={(\nabla F)}^{-1}\left(\eta \right)=\frac{1}{1+{\sum }_{j=1}^{D}exp\left({\eta }_j\right)}{[exp\left({\eta }_i\right)]}_i,{\theta }_i={(\nabla F^{-1}\left(\eta \right))}_i=\frac{exp\left({\eta }_i\right)}{1+{\sum }_{j=1}^{D}exp\left({\eta }_j\right)},\forall i\in \left[D\right].$$

(97)

Table 1. Two views of the family of categorical distributions with d choices: An exponential family or a mixture family of order $D=d-1$. Note that the Bregman divergence associated to the exponential family view corresponds to the reverse Kullback–Leibler (KL) divergence, while the Bregman divergence associated to the mixture family view corresponds to the KL divergence.

	Exponential Family	Mixture Family
pdf	$p_{\theta }\left(x\right)={\prod }_{i=1}^d{p}_i^{t_i\left(x\right)},p_i=Pr(x=e_i),t_i\left(x\right)\in \{0,1\},{\sum }_{i=1}^d{t}_i\left(x\right)=1$	$m_{\theta }\left(x\right)={\sum }_{i=1}^d{p}_i{\delta }_{e_i}\left(x\right)$
primal $\theta$	${\theta }_i=log\frac{p_i}{p_d}$	${\theta }_i=p_i$
$F\left(\theta \right)$	$log(1+{\sum }_{i=1}^{D}exp\left({\theta }_i\right))$	${\theta }_{i}log{\theta }_i+(1-{\sum }_{i=1}^D{\theta }_i)log(1-{\sum }_{i=1}^D{\theta }_i)$
dual $\eta =\nabla F\left(\theta \right)$	$\frac{e^{{\theta }_i}}{1+{\sum }_{j=1}^{D}exp\left({\theta }_j\right)}$	$log\frac{{\theta }_i}{1-{\sum }_{j=1}^D{\theta }_j}$
primal $\theta =\nabla F^{*}\left(\eta \right)$	$log\frac{{\eta }_i}{1-{\sum }_{j=1}^D{\eta }_j}$	$\frac{e^{{\theta }_i}}{1+{\sum }_{j=1}^{D}exp\left({\theta }_j\right)}$
$F^{*}\left(\eta \right)$	${\sum }_{i=1}^D{\eta }_{i}log{\eta }_i+(1-{\sum }_{j=1}^D{\eta }_j)log(1-{\sum }_{j=1}^D{\eta }_j)$	$log(1+{\sum }_{i=1}^{D}exp\left({\eta }_i\right))$
Bregman divergence	$B_F(\theta :{\theta }^{\prime })={\mathrm{KL}}^{*}(p_{\theta }:p_{{\theta }^{\prime }})$	$B_F(\theta :{\theta }^{\prime })=\mathrm{KL}(m_{\theta }:m_{{\theta }^{\prime }})$
	$=\mathrm{KL}(p_{{\theta }^{\prime }}:p_{\theta })$

summarizes the dual view of the family of categorical distributions, either interpreted as an exponential family or as a mixture family.

We have $\mathrm{JS}(p_1,p_2)=J_F({\theta }_1,{\theta }_2)$ for $p_1=m_{{\theta }_1}$ and $p_2=m_{{\theta }_2}$, where

$$J_F({\theta }_1:{\theta }_2)=\frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}-F\left(\frac{{\theta }_1+{\theta }_2}{2}\right),$$

(98)

is the Jensen divergence [40]. Thus, to compute the Jensen–Shannon centroid of a set of n densities $p_1,\dots ,p_n$ of a mixture family (with $p_i=m_{{\theta }_i}$), we need to solve the following optimization problem for a density $p=m_{\theta }$:

$$\begin{array}{c}\underset{p}{min}\sum _i\mathrm{JS}(p_i,p),\hfill \end{array}$$

(99)

$$\begin{array}{c}\underset{\theta }{min}\sum _i{J}_F({\theta }_i,\theta ),\hfill \end{array}$$

(100)

$$\begin{array}{c}\underset{\theta }{min}\sum _i\frac{F\left({\theta }_i\right)+F\left(\theta \right)}{2}-F\left(\frac{{\theta }_i+\theta }{2}\right),\hfill \end{array}$$

(101)

$$\begin{array}{c}\equiv \underset{\theta }{min}\frac{1}{2}F\left(\theta \right)-\frac{1}{n}\sum _{i}F\left(\frac{{\theta }_i+\theta }{2}\right):=E\left(\theta \right).\hfill \end{array}$$

(102)

The CCCP algorithm for the Jensen–Shannon centroid proceeds by initializing ${\theta }^{\left(0\right)}=\frac{1}{n}{\sum }_i{\theta }_i$ (center of mass of the natural parameters), and iteratively updates as follows:

$${\theta }^{(t+1)}={(\nabla F)}^{-1}\left(\frac{1}{n}\sum _i\nabla F\left(\frac{{\theta }_i+{\theta }^{\left(t\right)}}{2}\right)\right).$$

(103)

We iterate until the absolute difference $|E\left({\theta }^{\left(t\right)}\right)-E\left({\theta }^{(t+1)}\right)|$ between two successive ${\theta }^{\left(t\right)}$ and ${\theta }^{(t+1)}$ goes below a prescribed threshold value. The convergence of the CCCP algorithm is linear [41] to a local minimum that is a fixed point of the equation

$$\theta =M_H\left(\frac{{\theta }_1+\theta }{2},\dots ,\frac{{\theta }_n+\theta }{2}\right),$$

(104)

where $M_H(v_1,\dots ,v_n):=H^{-1}\left({\sum }_{i=1}^{n}H\left(v_i\right)\right)$ is a vector generalization of the formula of the quasi-arithmetic means [30,40] obtained for the generator $H=\nabla F$. Algorithm 1 summarizes the method for approximating the Jensen–Shannon centroid of a given set of categorical distributions (given a prescribed number of iterations). In the pseudo-code, we used the notation ${}^{(t+1)}\theta$ instead of ${\theta }^{(t+1)}$ in order to highlight the conversion procedures of the natural parameters to/from the mixture weight parameters by using superscript notations for coordinates.

Algorithm 1: The CCCP algorithm for computing the Jensen–Shannon centroid of a set of categorical distributions.

Figure 2 displays the results of the calculations of the Jeffreys centroid [18] and the Jensen–Shannon centroid for two normalized histograms obtained from grey-valued images of Lena and Barbara. Figure 3 show the Jeffreys centroid and the Jensen–Shannon centroid for the Barbara image and its negative image. Figure 4 demonstrates that the Jensen–Shannon centroid is well defined even if the input histograms do not have coinciding supports. Notice that on the parts of the support where only one distribution is defined, the JS centroid is a scaled copy of that defined distribution.

3.2.2. Special Cases

Let us now consider two special cases:

• For the special case of $D=1$, the categorical family is the Bernoulli family, and we have $F\left(\theta \right)=\theta log\theta +(1-\theta )log(1-\theta )$ (binary negentropy), $F^{\prime }\left(\theta \right)=log\frac{\theta }{1-\theta }$ (and $F^{\prime \prime }\left(\theta \right)=\frac{1}{\theta (1-\theta )}>0$) and ${\left(F^{\prime }\right)}^{-1}\left(\eta \right)=\frac{e^{\eta }}{1+e^{\eta }}$. The CCCP update rule to compute the binary Jensen–Shannon centroid becomes

  $${\theta }^{(t+1)}={\left(F^{\prime }\right)}^{-1}\left(\sum _i{w}_i{F}^{\prime }\left(\frac{{\theta }^{\left(t\right)}+{\theta }_i}{2}\right)\right).$$

  (105)
• Since the skew-vector Jensen–Shannon divergence formula holds for positive densities:

  $$\begin{array}{ccc}\hfill {{\mathrm{JS}}^{+}}^{\alpha ,w}(\tilde{p}:\tilde{q})& =& {\displaystyle \sum _{i=1}^k}{w}_i{\mathrm{KL}}^{+}({\left(\tilde{p}\tilde{q}\right)}_{{\alpha }_i}:\left({\left(\tilde{p}\tilde{q}\right)}_{\overline{\alpha }}\right),\hfill \end{array}$$

  (106)

  $$\begin{array}{cc}=& {\displaystyle \sum _{i=1}^k}{w}_i\left(\mathrm{KL}({\left(\tilde{p}\tilde{q}\right)}_{{\alpha }_i}:\left({\left(\tilde{p}\tilde{q}\right)}_{\overline{\alpha }}\right)+\int {\left(\tilde{p}\tilde{q}\right)}_{\overline{\alpha }}\mathrm{d}\mu -\underset{=\int {\left(\tilde{p}\tilde{q}\right)}_{\overline{\alpha }}\mathrm{d}\mu }{\underbrace{{\sum }_{i=1}^k{w}_i\int {\left(\tilde{p}\tilde{q}\right)}_{{\alpha }_i}\mathrm{d}\mu }}\right),\hfill \end{array}$$

  (107)

  $$\begin{array}{cc}=& {\mathrm{JS}}^{\alpha ,w}(\tilde{p}:\tilde{q}),\hfill \end{array}$$

  (108)

  we can relax the computation of the Jensen–Shannon centroid by considering 1D separable minimization problems. We then normalize the positive JS centroids to get an approximation of the probability JS centroids. This approach was also considered when dealing with the Jeffreys’ centroid [18]. In 1D, we have $F\left(\theta \right)=\theta log\theta -\theta$, $F^{\prime }\left(\theta \right)=log\theta$ and ${\left(F^{\prime }\right)}^{-1}\left(\eta \right)=e^{\eta }$.

In general, calculating the negentropy for a mixture family with continuous densities sharing the same support is not tractable because of the log-sum term of the differential entropy. However, the following remark emphasizes an extension of the mixture family of categorical distributions:

3.2.3. Some Remarks and Properties

Remark 3.

Consider a mixture family $m\left(\theta \right)={\sum }_{i=1}^D{\theta }_i{p}_i\left(x\right)+\left(1-{\sum }_{i=1}^D{\theta }_i\right)p_0\left(x\right)$ (for a parameter θ belonging to the D-dimensional standard simplex) of probability densities $p_0\left(x\right),\dots ,p_D\left(x\right)$ defined respectively on the supports ${\mathcal{X}}_0,{\mathcal{X}}_1,\dots ,{\mathcal{X}}_D$. Let ${\theta }_0:=1-{\sum }_{i=1}^D{\theta }_i$. Assume that the support ${\mathcal{X}}_i$s of the $p_i$s are mutually non-intersecting(${\mathcal{X}}_i\cap {\mathcal{X}}_j=\varnothing$ for all $i\ne j$ implying that the $D+1$ densities are linearly independent) so that $m_{\theta }\left(x\right)={\theta }_i{p}_i\left(x\right)$ for all $x\in {\mathcal{X}}_i$, and let $\mathcal{X}={\cup }_i{\mathcal{X}}_i$. Consider Shannon negative entropy $F\left(\theta \right)=-h\left(m_{\theta }\right)$ as a strictly convex function. Then, we have

$$\begin{array}{ccc}\hfill F\left(\theta \right)& =& -h\left(m_{\theta }\right)={\int }_{\mathcal{X}}{m}_{\theta }\left(x\right)log{m}_{\theta }\left(x\right),\hfill \end{array}$$

(109)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=0}^D}{\theta }_i{\int }_{{\mathcal{X}}_i}{p}_i\left(x\right)log\left({\theta }_i{p}_i\left(x\right)\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(110)

$$\begin{array}{cc}=& {\displaystyle \sum _{i=0}^D}{\theta }_{i}log{\theta }_i-{\displaystyle \sum _{i=0}^D}{\theta }_{i}h\left(p_i\right).\hfill \end{array}$$

(111)

Note that the term ${\sum }_i{\theta }_{i}h\left(p_i\right)$ is affine in θ, and Bregman divergences are defined up to affine terms so that the Bregman generator F is equivalent to the Bregman generator of the family of categorical distributions. This example generalizes the ordinary mixture family of categorical distributions where the $p_i$s are distinct Dirac distributions. Note that when the support of the component distributions are not pairwise disjoint, the (neg)entropy may not be analytic [42] (e.g., mixture of the convex weighting of two prescribed distinct Gaussian distributions). This contrasts with the fact that the cumulant function of an exponential family is always real-analytic [43]. Observe that the term ${\sum }_i{\theta }_{i}h\left(p_i\right)$ can be interpreted as a conditional entropy: ${\sum }_i{\theta }_{i}h\left(p_i\right)=h\left(X\right|\Theta )$ where $\mathrm{Pr}(\Theta =i)={\theta }_i$ and $\mathrm{Pr}(X\in S|\Theta =i)={\int }_S{p}_i\left(x\right)\mathrm{d}\mu \left(x\right)$.

Notice that we can truncate an exponential family [25] to get a (potentially non-regular [44]) exponential family for defining the $p_i$s on mutually non-intersecting domains ${\mathcal{X}}_i$s. The entropy of a natural exponential family $\{e(x:\theta )=exp(x^{\top }\theta -\psi \left(\theta \right)):\theta \in \Theta \}$ with cumulant function $\psi \left(\theta \right)$ and natural parameter space Θ is $-{\psi }^{*}\left(\eta \right)$, where $\eta =\nabla \psi \left(\theta \right)$, and ${\psi }^{*}$ is the Legendre convex conjugate [45]: $h\left(e(x:\theta )\right)=-{\psi }^{*}(\nabla \psi \left(\theta \right))$.

In general, the entropy and cross-entropy between densities of a mixture family (whether the distributions have disjoint supports or not) can be calculated in closed-form.

Property 2.

The entropy of a density belonging to a mixture family $\mathcal{M}$ is $h\left(m_{\theta }\right)=-F\left(\theta \right)$, and the cross-entropy between two mixture densities $m_{{\theta }_1}$ and $m_{{\theta }_2}$ is $h^{\times }(m_{{\theta }_1}:m_{{\theta }_2})=-F\left({\theta }_2\right)-{({\theta }_1-{\theta }_2)}^{\top }{\eta }_2=F^{*}\left({\eta }_2\right)-{\theta }_1^{\top }{\eta }_2$.

Proof. 

Let us write the KLD as the difference between the cross-entropy minus the entropy [4]:

$$\begin{array}{ccc}\hfill \mathrm{KL}(m_{{\theta }_1}:m_{{\theta }_2})& =& h^{\times }(m_{{\theta }_1}:m_{{\theta }_2})-h\left(m_{{\theta }_1}\right),\hfill \end{array}$$

(112)

$$\begin{array}{cc}=& B_F({\theta }_1:{\theta }_2),\hfill \end{array}$$

(113)

$$\begin{array}{cc}=& F\left({\theta }_1\right)-F\left({\theta }_2\right)-{({\theta }_1-{\theta }_2)}^{\top }\nabla F\left({\theta }_2\right).\hfill \end{array}$$

(114)

Following [45], we deduce that $h\left(m_{\theta }\right)=-F\left(\theta \right)+c$ and $h^{\times }(m_{{\theta }_1}:m_{{\theta }_2})=-F\left({\theta }_2\right)-{({\theta }_1-{\theta }_2)}^{\top }{\eta }_2-c$ for a constant c. Since $F\left(\theta \right)=-h\left(m_{\theta }\right)$ by definition, it follows that $c=0$ and that $h^{\times }(m_{{\theta }_1}:m_{{\theta }_2})=-F\left({\theta }_2\right)-{({\theta }_1-{\theta }_2)}^{\top }{\eta }_2=F^{*}\left({\eta }_2\right)-{\theta }_1^{\top }{\eta }_2$ where $\eta =\nabla F\left(\theta \right)$.  □

Thus, we can numerically compute the Jensen–Shannon centroids (or barycenters) of a set of densities belonging to a mixture family. This includes the case of categorical distributions and the case of Gaussian Mixture Models (GMMs) with prescribed Gaussian components [38] (although in this case, the negentropy needs to be stochastically approximated using Monte Carlo techniques [46]). When the densities do not belong to a mixture family (say, the Gaussian family, which is an exponential family [25]), we face the problem that the mixture of two densities does not belong to the family anymore. One way to tackle this problem is to project the mixture onto the Gaussian family. This corresponds to an m-projection (mixture projection) which can be interpreted as a Maximum Entropy projection of the mixture [25,47]).

Notice that we can perform fast k-means clustering without centroid calculations using a generalization of the k-means++ probabilistic initialization [48,49]. See [50] for details of the generalized k-means++ probabilistic initialization defined according to an arbitrary divergence.

Finally, let us notice some decompositions of the Jensen–Shannon divergence and the skew Jensen divergences.

Remark 4.

We have the following decomposition for the Jensen–Shannon divergence:

$$\begin{array}{ccc}\hfill \mathrm{JS}(p_1,p_2)& =& h\left(\frac{p_1+p_2}{2}\right)-\frac{h\left(p_1\right)+h\left(p_2\right)}{2},\hfill \end{array}$$

(115)

$$\begin{array}{cc}=& h_{\mathrm{JS}}^{\times }(p_1:p_2)-h_{\mathrm{JS}}\left(p_2\right)\ge 0,\hfill \end{array}$$

(116)

where

$$h_{\mathrm{JS}}^{\times }(p_1:p_2)=h\left(\frac{p_1+p_2}{2}\right)-\frac{1}{2}h\left(p_1\right),$$

(117)

and $h_{\mathrm{JS}}\left(p_2\right)=h_{\mathrm{JS}}^{\times }(p_2:p_2)=h\left(p_2\right)-\frac{1}{2}h\left(p_2\right)=\frac{1}{2}h\left(p_2\right)$. This decomposition bears some similarity with the KLD decomposition viewed as the cross-entropy minus the entropy (with the cross-entropy always upper-bounding the entropy).

Similarly, the α-skew Jensen divergence

$$J_F^{\alpha }({\theta }_1:{\theta }_2):={\left(F\left({\theta }_1\right)F\left({\theta }_2\right)\right)}_{\alpha }-F\left({\left({\theta }_1{\theta }_2\right)}_{\alpha }\right),\alpha \in (0,1)$$

(118)

can be decomposed as the sum of the information $I_F^{\alpha }\left({\theta }_1\right)=(1-\alpha )F\left({\theta }_1\right)$ minus the cross-information $C_F^{\alpha }({\theta }_1:{\theta }_2):=F\left({\left({\theta }_1{\theta }_2\right)}_{\alpha }\right)-\alpha F\left({\theta }_2\right)$:

$$J_F^{\alpha }({\theta }_1:{\theta }_2)=I_F^{\alpha }\left({\theta }_1\right)-C_F^{\alpha }({\theta }_1:{\theta }_2)\ge 0.$$

(119)

Notice that the information $I_F^{\alpha }\left({\theta }_1\right)$ is the self cross-information: $I_F^{\alpha }\left({\theta }_1\right)=C_F^{\alpha }({\theta }_1:{\theta }_1)=(1-\alpha )F\left({\theta }_1\right)$. Recall that the convex information is the negentropy where the entropy is concave. For the Jensen–Shannon divergence on the mixture family of categorical distributions, the convex generator $F\left(\theta \right)=-h\left(m_{\theta }\right)={\sum }_{i=1}^D{\theta }^{i}log{\theta }^i$ is the Shannon negentropy.

Finally, let us briefly mention the Jensen–Shannon diversity [30] which extends the Jensen–Shannon divergence to a weighted set of densities as follows:

$$\mathrm{JS}(p_1,\dots ,p_k;w_1,\dots ,w_k):={\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}(p_i:\overline{p}),$$

(120)

where $\overline{p}={\sum }_{i=1}^k{w}_i{p}_i$. The Jensen–Shannon diversity plays the role of the variance of a cluster with respect to the KLD. Indeed, let us state the compensation identity [51]: For any q, we have

$${\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}(p_i:q)={\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}(p_i:\overline{p})+\mathrm{KL}(\overline{p}:q).$$

(121)

Thus, the cluster center defined as the minimizer of ${\sum }_{i=1}^k{w}_i\mathrm{KL}(p_i:q)$ is the centroid $\overline{p}$, and

$${\displaystyle \sum _{i=1}^k}{w}_i\mathrm{KL}(p_i:\overline{p})=\mathrm{JS}(p_1,\dots ,p_k;w_1,\dots ,w_k).$$

(122)

4. Conclusions and Discussion

The Jensen–Shannon divergence [6] is a renown symmetrization of the Kullback–Leibler oriented divergence that enjoys the following three essential properties:

• It is always bounded,
• it applies to densities with potentially different supports, and
• it extends to unnormalized densities while enjoying the same formula expression.

This JSD plays an important role in machine learning and in deep learning for studying Generative Adversarial Networks (GANs) [52]. Traditionally, the JSD has been skewed with a scalar parameter [19,53] $\alpha \in (0,1)$. In practice, it has been experimentally demonstrated that skewing divergences may significantly improve the performance of some tasks (e.g., [21,54]).

In general, we can symmetrize the KLD $\mathrm{KL}(p:q)$ by taking an abstract mean (we require a symmetric mean $M(x,y)=M(y,x)$ with the in-betweenness property: $min\{x,y\}\le M(x,y)\le max\{x,y\}$) M between the two orientations $\mathrm{KL}(p:q)$ and $\mathrm{KL}(q:p)$:

$${\mathrm{KL}}_M(p,q):=M(\mathrm{KL}(p:q),\mathrm{KL}(q:p)).$$

(123)

We recover the Jeffreys divergence by taking the arithmetic mean twice (i.e., $J(p,q)=2A\left(\mathrm{KL}\right(p:q),\mathrm{KL}(q:p\left)\right)$ where $A(x,y)=\frac{x+y}{2}$), and the resistor average divergence [55] by taking the harmonic mean (i.e., $R_{\mathrm{KL}}(p,q)=H(\mathrm{KL}(p:q),\mathrm{KL}(q:p))=\frac{2\mathrm{KL}(p:q)\mathrm{KL}(q:p)}{\mathrm{KL}(p:q)+\mathrm{KL}(q:p)}$ where $H(x,y)=\frac{2}{\frac{1}{x}+\frac{1}{y}}$). When we take the limit of Hölder power means, we get the following extremal symmetrizations of the KLD:

$$\begin{array}{ccc}\hfill {\mathrm{KL}}^{\min }(p:q)& =& min\{\mathrm{KL}(p:q),\mathrm{KL}(q:p)\}={\mathrm{KL}}^{\min }(q:p),\hfill \end{array}$$

(124)

$$\begin{array}{ccc}\hfill {\mathrm{KL}}^{\max }(p:q)& =& max\{\mathrm{KL}(p:q),\mathrm{KL}(q:p)\}={\mathrm{KL}}^{\max }(q:p).\hfill \end{array}$$

(125)

In this work, we showed how to vector-skew the JSD while preserving the above three properties. These new families of weighted vector-skew Jensen–Shannon divergences may allow one to fine-tune the dissimilarity in applications by replacing the skewing scalar parameter of the JSD by a vector parameter (informally, adding some “knobs” for tuning a divergence). We then considered computing the Jensen–Shannon centroids of a set of densities belonging to a mixture family [25] by using the convex–concave procedure [27].

In general, we can vector-skew any arbitrary divergence D by using two k-dimensional vectors $\alpha \in {[0,1]}^k$ and $\beta \in {[0,1]}^k$ (with $\alpha \ne \beta$) by building a weighted separable divergence as follows:

$$D^{\alpha ,\beta ,w}(p:q):={\displaystyle \sum _{i=1}^k}{w}_{i}D\left({\left(pq\right)}_{{\alpha }_i}:{\left(pq\right)}_{{\beta }_i}\right)=D^{1_k-\alpha ,1_k-\beta ,w}(q:p),\alpha \ne \beta .$$

(126)

This bi-vector-skew divergence unifies the Jeffreys divergence with the Jensen–Shannon $\alpha$-skew divergence by setting the following parameters:

$$\begin{array}{ccc}\hfill {\mathrm{KL}}^{(0,1),(1,0),(1,1)}(p:q)& =& \mathrm{KL}(p:q)+\mathrm{KL}(q:p)=J(p,q),\hfill \end{array}$$

(127)

$$\begin{array}{ccc}\hfill {\mathrm{KL}}^{(0,\alpha ),(1,1-\alpha ),(\frac{1}{2},\frac{1}{2})}(p:q)& =& \frac{1}{2}\mathrm{KL}(p:{\left(pq\right)}_{\alpha })+\frac{1}{2}\mathrm{KL}(q:{\left(pq\right)}_{\alpha }).\hfill \end{array}$$

(128)

We have shown in this paper that interesting properties may occur when the skewing vector $\beta$ is purposely correlated to the skewing vector $\alpha$: Namely, for the bi-vector-skew Bregman divergences with $\beta =(\overline{\alpha },\dots ,\overline{\alpha })$ and $\overline{\alpha }={\sum }_i{w}_i{\alpha }_i$, we obtain an equivalent Jensen diversity for the Jensen–Bregman divergence, and, as a byproduct, a vector-skew generalization of the Jensen–Shannon divergence.