Kernel-independent component analysis

In statistics, kernel-independent component analysis (kernel ICA) is an efficient algorithm for independent component analysis which estimates source components by optimizing a generalized variance contrast function, which is based on representations in a reproducing kernel Hilbert space.[1][2] Those contrast functions use the notion of mutual information as a measure of statistical independence.

Main idea

Kernel ICA is based on the idea that correlations between two random variables can be represented in a reproducing kernel Hilbert space (RKHS), denoted by ${\displaystyle {\mathcal {F}}}$, associated with a feature map ${\displaystyle L_{x}:{\mathcal {F}}\mapsto \mathbb {R} }$ defined for a fixed ${\displaystyle x\in \mathbb {R} }$. The ${\displaystyle {\mathcal {F}}}$-correlation between two random variables ${\displaystyle X}$ and ${\displaystyle Y}$ is defined as

${\displaystyle \rho _{\mathcal {F}}(X,Y)=\max _{f,g\in {\mathcal {F}}}\operatorname {corr} (\langle L_{X},f\rangle ,\langle L_{Y},g\rangle )}$

where the functions ${\displaystyle f,g:\mathbb {R} \to \mathbb {R} }$ range over ${\displaystyle {\mathcal {F}}}$ and

${\displaystyle \operatorname {corr} (\langle L_{X},f\rangle ,\langle L_{Y},g\rangle ):={\frac {\operatorname {cov} (f(X),g(Y))}{\operatorname {var} (f(X))^{1/2}\operatorname {var} (g(Y))^{1/2}}}}$

for fixed ${\displaystyle f,g\in {\mathcal {F}}}$.[1] Note that the reproducing property implies that ${\displaystyle f(x)=\langle L_{x},f\rangle }$ for fixed ${\displaystyle x\in \mathbb {R} }$ and ${\displaystyle f\in {\mathcal {F}}}$.[3] It follows then that the ${\displaystyle {\mathcal {F}}}$-correlation between two independent random variables is zero.

This notion of ${\displaystyle {\mathcal {F}}}$-correlations is used for defining contrast functions that are optimized in the Kernel ICA algorithm. Specifically, if ${\displaystyle \mathbf {X}$ :=(x_{ij})\in \mathbb {R} ^{n\times m}} is a prewhitened data matrix, that is, the sample mean of each column is zero and the sample covariance of the rows is the ${\displaystyle m\times m}$ dimensional identity matrix, Kernel ICA estimates a ${\displaystyle m\times m}$ dimensional orthogonal matrix ${\displaystyle \mathbf {A} }$ so as to minimize finite-sample ${\displaystyle {\mathcal {F}}}$-correlations between the columns of ${\displaystyle \mathbf {S}$ :=\mathbf {X} \mathbf {A} ^{\prime }} .

References

1. Bach, Francis R.; Jordan, Michael I. (2003). "Kernel independent component analysis" (PDF). The Journal of Machine Learning Research. 3: 1–48. doi:10.1162/153244303768966085.
2. Bach, Francis R.; Jordan, Michael I. (2003). Kernel independent component analysis (PDF). IEEE International Conference on Acoustics, Speech, and Signal Processing. 4. pp. IV-876-9. doi:10.1109/icassp.2003.1202783. ISBN 978-0-7803-7663-2.
3. Saitoh, Saburou (1988). Theory of Reproducing Kernels and Its Applications. Longman. ISBN 978-0582035645.