Week 7: Learning

Synaptic plasticity, Hebbian and statistical learning

Biological basis of learning

Hippocampus: first observations of synaptic plasticity

Long Term Potentiation: Long-term increase in synaptic strength. Observed as an increase in EPSP size over time.

Long Term Depression: Long-term decrease in synaptic strength, observed as decrease of EPSP.

Q: Can you speculate what might be happening to the post-synaptic cell that might cause an increase or decrease in EPSP amplitude?

• An increase/decrease in the number of neurotransmitter-gated channels.
• An increase/decrease in the number of leak channels.
• Both of these.
• Neither of these

Explanation: Anything that works to change the cell’s rest conductance (changing number of leak channels) will change its excitability. Similarly, anything that changes the amount of current that can flow into the post synaptic cell upon binding with neurotransmitter (changing neurotransmitter-gated channels) will change its excitability.

Hebb’s rule

Donald Hebb suggested the Hebbian learning rule before the discovery of LTP/D:

If neuron A takes part in the firing of neuron B, then the synapse from A to B is strengthened.

Let the input vector $\textbf{u}$, connected to output unit (scalar) $v$, through the weights vector $\textbf{w}$.

Assume that the network dynamics is fast enough to be ignored. Then, the output is determined by the steady state output:

Then, the continuous Hebbian learning rule can be expressed as the following rate equation:

In english, the change in connection strength is proportional to the product of input and output, which product will be low if either or both units don’t fire, and will be high if and only if both units fire.

The delta-time version Hebbian learning rule

leads to the following discrete time implementation:

where $\epsilon$ is the learning rate.

Average effect of Hebb’s rule

With several input vectors $\textbf{u}$, the Hebbian rule changes the weights by multiplying the input correlation matrix $Q$ to them:

The input correlation matrix is $Q = \langle\textbf{u}\textbf{u}^ T\rangle_\textbf{u}$.

Stability of the Hebbian learning rule

Is $w$ converging? Looking at $\frac{\textrm{d} \lVert \textbf{w} \rVert ^ 2}{\textrm{d}t}$ should indicate whether it explodes.

For the continuous Hebbian learning rule,

which means that $\textbf{w}$ grows without bounds.

The covariance rule

LTD is modeled to happen when there is no or low ouput even though there is an input. But Hebb’s rule doesn’t model that.

The covariance rule generalizes the Hebbian learning rule to also work for LTD. In the continuous expression of the Hebbian learning rule, $\tau_w\frac{\textrm{d}\textbf{w}}{\textrm{d}t} = \textbf{u}v$, we replace the output vector $v$ by its variation from average $v-\langle v\rangle$:

Q: If v (output of post-synaptic neuron) is large relative to the average of v over time in this equation, what does that imply about w?

• w will increase in magnitude and the cell will undergo LTP
• w will decrease in magnitude and the cell will undergo LTP
• w will increase in magnitude and the cell will undergo LTD
• w will decrease in magnitude and the cell will undergo LTD

Q: Conversely, if v is small (approaches 0) relative to the average of v over time, what does that imply about w?

• w will increase in magnitude and the cell will undergo LTP
• w will decrease in magnitude and the cell will undergo LTP
• w will increase in magnitude and the cell will undergo LTD
• w will decrease in magnitude and the cell will undergo LTD

We note that $v-\langle v\rangle$ is positive for high outputs, which amounts to Hebbian learning, and negative for low outputs, which makes the change of weights negative, leading to LTD.

Average effect of the covariance rule

With several input vectors $\textbf{u}$, the covariance rule changes the weights by multiplying the input covariance matrix $C$ to them:

The covariance matrix of the inputs is $C = \left(\langle\textbf{u}\textbf{u}^ T\rangle-\langle\textbf{u}\textbf{u}^ T\rangle\right)$

Stability of the covariance rule

For the covariance learning rule,

so $\textbf{w}$ grows without bounds too.

Unstable learning rules can be stabilized by normalizing $w$ at each update, ensuring that $\lVert \textbf{w} \rVert =1$ at all times. But that prevents small weights and LTD.

Oja’s learning rule

For $\alpha > 0$,

Stability of Oja’s rule

Looking at the derivative of $\lVert \textbf{w} \rVert ^ 2$,

The norm of $\textbf{w}$ will converge when $\frac{\textrm{d} \lVert \textbf{w} \rVert }{\textrm{d}t} = 0$, which corresponds to $\lVert \textbf{w} \rVert ^ 2 = \frac{1}{\alpha}$

So Oja’s rule is stable, and $\lVert \textbf{w} \rVert$ converges to $\frac{1}{\sqrt{\alpha}}$.

What do Hebbian, Covariance and Oja’s learning rules do mathematically?

The averaged Hebbian learning rule $\tau_w\frac{\textrm{d}\textbf{w}}{\textrm{d}t} = Q\textbf{w}$ gives us an equation in terms of $\textbf{w}(t)$. To solve that equation, we use eigenvectors (!).

Let $\textbf{e}_i$ the eigenvectors of the correlation matrix $Q$. $Q$ is orthonormal, so we can write $\textbf{w}(t)$ as $\sum_i c_i(t)\textbf{e}_i$. Substituting it in the expression of the averaged Hebbian rule:

with the solution

Using this solution in $\textbf{w}(t)=\sum_i c_i(t)\textbf{e}_i$, we get

$w$ is a linear combination of the eigenvectors of $Q$, with terms that are exponentially dependent on the eigenvalues of $Q$. In consequence, at the limit in time, the largest eigenvalue dominates the expression, and we have $\textbf{w}(t)\propto \textbf{e}_1$.

Similarly, for Oja’s rule, $\lim_{t \to \infty}\textbf{w}(t) = \frac{\textbf{e}_1}{\sqrt{\alpha}}$.

This means that those rules are equivalent to Principal Component Analysis.

Principal Component Analysis

For data with zero mean, the Hebb rule orients the weights vector in the direction of the maximum variance of the dataset (direction of the first eigenvector). If does not do that if the mean is not $\\textbf{0}$.

Without constraint on the mean, the covariance learning rule will orient the weights vector in the direction of the maximum variance.

This means that Hebbian learning and its derivatives perform PCA on the dataset, leading to the linear dimensionality reduction that maximizes the variance of the projected data.

When applied to clustered data (for instance, two clouds of points in a 2D input space), the covariance rule will still align $w$ with the overall maximum variance, which is less interesting for clustered data.

Networks of neurons performing unsupervised learning are able to take better advantage of such data.

Introduction to Unsupervised Learning

With clustered data, the covariance rule doesn’t find the most interesting feature of the input data, which are the two clusters.

A feedforward network that learns 2 clusters

We have two-dimensional data input through vector $\textbf{u}$, connected to output units $v_A$ and $v_B$ via the weight vectors $\textbf{w}_A$ and $\textbf{w}_B$. The output $v_i$ of unit $i$ is

Neuron A (B) can then represent cluster A (B) by having $\textbf{w}_A$ ($\textbf{w}_B$) be equal to the center (mean) of cluster A (B).

Q: Would neuron A or B be more active with an input that is closer (in Euclidian distance) to $\textbf{w}_B$?

• A
• B

For a particular input, the most active neuron will be the one closest to the input:

Indeed, assuming that the input and weight vectors have been normalized, minimizing $\lVert \textbf{u} - \textbf{w}_i \rVert ^ 2$ is equivalent to maximizing $v_i$.

Updating the weights

Given a new input vector $\textbf{u}_t$ (t is not time but a chronological index of inputs from 1 to t),

• Find the new input’s cluster
• Update the weight of that cluster by setting the weight vector to the running average of the inputs of that cluster.

How to compute that running average? We want

So, in the code, the modification of the weights vector has to be

where $\epsilon$ can be set as $\frac{1}{t}$ to compute the running average, or a small positive value.

Q: (1/t) may seem like a logical value for epsilon, but can you speculate why having a positive constant for epsilon might be more beneficial?

• It allows u to remain larger than w
• It allows the algorithm to update w indefinitely.
• (1/t) is actually the best value for epsilon.
• None of these.

Competitive learning

The competitive learning algorithm goes as follows:

• Select the most active neuron (let it be A), with weights closest to the new input (for instance using a WTA type of network)
• update the weights: $\Delta\textbf{w}_A = \epsilon\left(\textbf{u}_t - \textbf{w}_A\right)$

The weights vector is shifted from its current location in the direction of the new input by an amount of $\epsilon$.

Example

We have 3 output neurons with randomly assigned initial weight values $\textbf{w}_1$, $\textbf{w}_2$ and $\textbf{w}_3$. Inputs come one at a time, and one neuron’s weights is adapted for each input. In the end, a partitioning in 3 cluster has emerged, and each weight is close to the centroid of its cluster.

Relation to ANN algorithms

Competitive learning is closely related to Kohonen maps (SOM). Kohonen maps, however, update the weight vectors of other neurons in the neighborhood of the winning neuron too.

in this update, neighborhood is taken in a topological sense, as a network-intrinsic neighborhood relationship has been defined prior to learning. Typically, neurons form a topological two or three dimensional grid. The initial weights of neurons are typically aligned to their location in the topological map.

In biological neural systems, cortical maps can have a similar organization, with neural preferences for one input feature neatly arranged in the 2-D topological map of neighboring neurons. For instance, V1 orientation preference maps are made of neighboring neurons that prefer similar orientations.

Unsupervised learning

The input $\textbf{u}$ is generated by a set of hidden causes $\textbf{v}$.

A generative model $G$ defines the transformation $v \to \textbf{u}$. The goal is to learn a good model of the data generation process.

There is a prior joint probability $p[\textbf{v}; G]$, and a likelihood function $p[\textbf{u} \mid \textbf{v},G]$. $G$ is determined by a set of unknown parameters that we would like to learn.

The two sub-problems of unsupervised learning are:

• problem of recognition: estimate the causes $\textbf{v}$ for any observed input $\textbf{u}$
  • compute the posterior probability $p[\textbf{v} \mid \textbf{u};G]$
• use that information to learn the parameters $G$

Example

Back to our example of 2 clusters of points A and B in a 2d plane.

We assume a model of two gaussian generators $\langle\mu_A, \sigma_A\rangle$ and $\langle\mu_B, \sigma_B\rangle$. It’s a mixture of Gaussians model:

where $\gamma_v$ is the prior probability $p[v;G]$.

The problem of recognition (finding the posterior $p[\textbf{v} \mid \textbf{u};G]$) here is similar to the first step of competitive learning. The first step of competitive learning was to assign an incoming data point to a cluster. Here, we compute the posterior probability of the data point belonging to cluster A or B: $p[$ “data is A given that its coordinates are $\textbf{u}$ and that cluster A has distributions parameters $G_A$ “ $]$ ($p[v=A \mid \textbf{u}, \mu_A, \sigma_A, \gamma_A]$) and $p[$ “data is B given that its coordinates are $\textbf{u}$ and that cluster B has distributions parameters $G_B$ “ $]$ ($p[v=B \mid \textbf{u}, \mu_B, \sigma_B, \gamma_B]$), and find that $p[v=A \mid \textbf{u}, \mu_A, \sigma_A, \gamma_A] > p[v=B \mid \textbf{u}, \mu_B, \sigma_B, \gamma_B]$.

The second step of unsupervised learning is to update the parameters of $G$, using $p[\textbf{v} \mid \textbf{u};G]$. That corresponds to the second step of competitive learning, that consist in updating the weight vector to reflect the new cluster center. But here, in unsupervised learning, we do that update using the Expectation-Maximization algorithm.

The EM algorithm for unsupervised learning

The algorithm consists in iterating two steps until convergence.

The Expectation step

In the E step, we compute the posterior distribution of $v$ (of clusters A and B in our example) for each $\textbf{u}$.

Using Bayes’ rule,

As compared to competitive learning, this is a softer competition than WTA.

The Maximization step

In the M step, we chance the set of parameters $G$ using the results of the E stem.

The mean is changed as follows:

the variance:

and the prior probability:

In comparison, competitive learning adapted the mean, but not the variance or prior probability of the clusters.

The EM algorithm assumes that all datapoints are available at the time of learning (batch learning), whereas real-time learning is possible in competitive learning (online learning).

Sparse Coding and Predictive Coding

Principal Component Analysis

Even on large input spaces, the eigenvectors of the input covariance matrix can be linearly combined to approximate input exemplars.

For instance, given a set of b&w pictures of faces of N pixels, Turk and Pentland (1991) computed the eigenvectors of the input covariance matrix, and subsequently expressed each face as a linear combination of these “eigenfaces”: $\textbf{u} = \sum_{i=1}^ N\textbf{e}_iv_i$

If we restrict the reconstruction to the use of the first $M \lt N$ principal eigenvectors (associated with the $M$) largest eigenvalues of the covariance matrix, then a face is

This model can be used for lossy image compression.

However, an eigenvector analysis (or, equivalently, a PCA) is not good for local components extraction (edges, parts, etc…).

Linear model of natural images

We now split up the input sample (natural image of $N$ pixels for instance) in a number of weighted basis features (f.e. localized oriented edges). The image is the weighted sum of $M$ (this time possibly larger than $N$) basis vectors, plus noise:

where $G$ is the $M\times N$ matrix of column basis vectors $\textbf{g}_i$, and $\textbf{v}$ is the row vector of coefficients $v_i$ ($M$ elements).

We need to learn $G$ and $\textbf{v}$.

Generative model of natural images

To specify a generative model for natural images, we need to specify a prior probability distribution for natural images $p[\textbf{v}]$, and a likelihood function $p[\textbf{u} \mid \textbf{v},G]$.

Likelihood function for the generative model of natural images, assuming white noise

In our linear model $\textbf{u} = G\textbf{v} + \textbf{noise}$, if the noise vector is assumed to be a Gaussian ($\to$ no correlation across the components of the noise vector) with zero mean, then the likelihood function is also Gaussian, with a mean $G\textbf{v}$ and an identity covariance:

The proportionality to the exponential above makes that the log likelihood is

where, in the quadratic term, $\lVert textbf{u}-G\textbf{v} \rVert$ can be identified as the difference between the input image and its reconstruction.

Q: Based on the equation log p[u \mid v;G] = -(1/2) \lVert u-Gv \rVert ^2 + C, can you see what effect minimizing the squared reconstruction error has on the likelihood function?

• Minimizes it.
• Maximizes it.
• Stabilizes it.
• None of these.

Prior distribution for the generative model of natural images, assuming that causes $v_i$ are independent

If we can assume that the causes $v_i$ are independent (we typically can’t for natural images, but let’s start like that), then the prior probability for $\textbf{v}$ is equal to the product of the individual prior probabilities of its components:

In log terms:

How to find the individual priors $p[v_i; G]$?

As we assume that these $v_i$ represent very specific components of the image (matching the sparseness of biological neural representations), then for any input, we only want a few $v_i$ to be active. In consequence, we expect $p[v_i]$ to have a leptokurtic (super Gaussian) distribution (peak at 0, long tail, e.g. exponential $p[v_i]=e^ {-v_i}$, Cauchy $p[v_i]=\frac{1}{\pi \left(1+v_i^ 2\right)}$, …).

We represent each $p[v_i]$ in the form of an exponential: $p[v_i] = e^ {g(v_i)}$. For instance, $g(v_i) = - \lvert v_i \rvert$ in an exponential, $g(v_i) = -\pi\log(1+v_i^ 2)$ for a Chauchy pdf, etc…

Then, we have

Bayesian approach to finding $\textbf{v}$ and learning $G$ in the generative model of natural stimuli

Bayesian = maximize the posterior probability of causes:

This is equivalent to maximizing the log-posterior probability of causes:

We note that $-\frac{1}{2} \lVert \textbf{u}-G\textbf{v} \rVert ^ 2$ represents the reconstruction error, which we want to minimize, and $\sum_i g(v_i)$ is the sparseness constraint, that we try to maximize.

Maximization algorithm

We note the similarity with the EM algorithm.

Repeat those two steps:

• Maximize $F$ w.r.t. $\textbf{v}$ (keep $G$ fixed) (lie the E step in EM)
• Maximize $F$ w.r.t. $G$ (keep $\textbf{v}$ fixed to the value found in the previous step) (like the M step in EM)

Maximizing $F$ w.r.t. $\textbf{v}$ with gradient ascent

To maximize $F$ w.r.t. $\textbf{v}$, we look at $\frac{\textrm{d}F}{\textrm{d}\textbf{v}}$. We change $\textbf{v}$ proportionately to the slope $\frac{\textrm{d}F}{\textrm{d}\textbf{v}}$. Here,

where $\tau_v$ is a time constant $\left(\textbf{u}-G\textbf{v}\right)$ is $\textbf{u}$ minus its reconstruction, so it’s the error, and $g'(\textbf{v})$ is the sparseness constraint.

This equation $\tau_v \frac{\textrm{d}\textbf{v}}{\textrm{d}t} = G^ T \left(\textbf{u}-G\textbf{v}\right) + g'(\textbf{v})$ has the form of a recurrent network’s firing rate kinetic equation!

Recurrent network implementation of sparse coding

In $\tau_v \frac{\textrm{d}\textbf{v}}{\textrm{d}t} = G^ T \left(\textbf{u}-G\textbf{v}\right) + g'(\textbf{v})$,

• $G^ T \textbf{u}$ is the total input to the output layer $\textbf{v}$: it is the input $\textbf{u}$ after weighting through the feedforward weight matrix $G^ T$
• $-G^ T G\textbf{v}$ is the recurrent (intra-layer) input, with $-G^ T G$ the recurrent weights
• There is a feedback connection from $\textbf{v}$ to $\textbf{u}$ with weights $G$. It computes $G\textbf{v}$, which is the mean of the generative distribution, and the prediction.
• Hence, $\textbf{u}-G\textbf{v}$, the error, is further computed at the input layer and propagated to the output layer through the feedforward weights $G^ T$. With that, the output layer corrects its estimate $\textbf{v}$.
• This prediction-correction cycle is iterated until convergence is achieved ($\textbf{v}$ stable for any given $\textbf{u}$)

Learning $G$ with gradient ascent

Like for $\textbf{v}$, we maximize $F$ w.r.t. $G$ by changing $(G$ proportionately to the slope $\frac{\textrm{d}F}{\textrm{d}G}$:

Taking $\tau_G \gt \tau_\textbf{v}$ guatantees that $\textbf{v}$ converges faster than $G$, making it possible to use $\textbf{v}$ to update $G$.

This learning rule, $\tau_G \frac{\textrm{d}G}{\textrm{d}t} = \left(\textbf{u}-G\textbf{v}\right) \textbf{v}^ T$, is very similar to Oja’s learning rule. However, it’s not learning the eigenvectors thanks to the sparseness criterion.

Take a guess just for fun. Based on what we have discussed regarding the visual system so far, especially our characterization of receptive field structure in early visual processing, what kind of basis vectors would you predict are learned for natural image patches?

• Eigenvectors of natural image patches
• Objects in the environment
• Complex shapes
• Oriented bars

Feeding the network with patches from natural images, the basis vectors (columns $\textbf{g}_i$ of $G$) learnt resemble the oriented receptor fields of V1. As an interpretive model, this indicates that the brain creates an efficient sparse representation of natural images through these RF.

Predictive coding networks

The sparse coding network is an instance of predictive coding network that uses feedback connections to transmit predictions about the input, and feedforward connections to convey a prediction error signal. The predictive estimator (output layer) maintains an estimate of the (hidden) causes of the input (vector $\textbf{v}$).

Predictive coding models can have, in addition to feedforward and feedback weights, a set of recurrent weights that learn time-varying input correlations, for instance in moving images. The internal representation (recurrent weights) is allowed to vary over time to model the dynamics of the inputs. Another possible component of predictive coding networks is a sensory error gain on the input layer, allowing to model such effects as visual attention.

The visual cortices present a puzzle: the connections between cortical areas in the visual streams are almost always bidirectional. Why the feedback connections?

Predictive coding networks models of the visual cortex give an explanation. They suggest that the feedback connections convey predictions of activity from higher to lower cortical areas, and that feedforward connections convey the error signals (activity minus prediction). Contextual effects, surround suppression, etc can be explained by hierarchical predictive coding models trained on natural images.