Sparse Coding

Source:

1. Yubei Chen's talk on unsupervised learning.

Most of the figures in this article are sourced from Yubei Chen's course, EEC289A, at UC Davis.

Intuition of sparse coding

Back in 1972, Horace Basil Barlow proposed the Efficient coding principle:

The sensory system is organized to achieve as complete a representation of the sensory stimulus as possible with the minimum number of active neurons.

The property of representing a sensory signal with the fewest possible neurons is known as the sparsity of neural activity.

In neuroscience, it has been observed that signals post-LGN (Lateral Geniculate Nucleus) experience a dimensional increase of hundreds or thousands of times, and their activity¹ becomes sparse.

Barlow 1981

Process of sparse coding

sparse coding

Sparse coding aims to represent each input data point, denoted as $\overrightarrow{x}$, through a transformation involving a set of features $\mathrm{\Phi }$:

$$\begin{array}{}\text{(1)}& \overrightarrow{x}=\mathrm{\Phi }\overrightarrow{\alpha }+\overrightarrow{n}.\end{array}$$ In this framework, the features $\mathrm{\Phi }$ are common and reusable across different instances.

Meanwhile, it is essential that the composition (or activity) of $\mathrm{\Phi }$ MUST be sparse. This implies that $\overrightarrow{x}$ should be constituted by the fewest number of features possible. Consequently, in $\text{(1)}$, the coefficients $\overrightarrow{\alpha }$ should have only a minimal number of non-zero entries.

---

1. Here, the activity of signals means the activity of the neurons reacting to the signals.↩︎