Harmonium (Smolensky, 1986)

This circuit implements the Harmonium model proposed in (Smolensky, 1986) [1]. Specifically, this model is unsupervised and can be used to process sensory pattern (row) vector(s) x to infer internal latent states. This class offers, beyond settling and update routines through Contrastive Divergence (Hinton 1999) [2], a block Gibbs sampling function to generate a chain of synthesized patterns.

The Harmonium is technically defined by two NGC graphs. The first is the positive phase (“wake” phase) graph depicted graphically below:

while second is the negative phase (“sleep” phase) graph depicted graphically below:

class ngclearn.museum.harmonium.Harmonium(args)

Structure for constructing the Harmonium model proposed in:

Hinton, Geoffrey E. “Training products of experts by maximizing contrastive likelihood.” Technical Report, Gatsby computational neuroscience unit (1999).

Node Name Structure:

z1 -(z1-z0)-> z0

z0 -(z0-z1)-> z1

Note: z1-z0 = (z0-z1)^T (transpose-tied synapses)

Another important reference for designing stable Harmoniums is here:

Hinton, Geoffrey E. “A practical guide to training restricted Boltzmann machines.” Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012. 599-619.

Note: if you set the samp_fx to the “identity”, you force the Harmonium to

to work as a mean-field Harmonium/Botlzmann machine

Parameters

args – a Config dictionary containing necessary meta-parameters for the Harmonium

DEFINITION NOTE:

args should contain values for the following:

* batch_size - the fixed batch-size to be fed into this model

* z_dim - # of latent variables in layer z1

* x_dim - # of latent variables in layer z0 (or sensory x)

* seed - number to control determinism of weight initialization

* wght_sd - standard deviation of Gaussian initialization of weights

* K - # of steps to take when conducting Contrastive Divergence

* act_fx - activation function for layer z1 (Default: sigmoid)

* out_fx - activation function for layer z0 (prediction of z0) (Default: sigmoid)

* samp_fx - sampling function for layer z1 (Default = bernoulli)

sample(K, x_sample=None, batch_size=1)

Samples the underlying harmonium to generate a chain of patterns from a block Gibbs sampling process.

Parameters

• K – number of steps to run the Gibbs sampler

• x_sample – inital condition for the sampler (Default = None), if None, this will generate an initial sample of size (batch_size, z1_dim) where z1_dim is the dimensionality of the latent state.

• batch_size – if x_sample is None, then this dictates how many samples in parallel to create per step of running the Gibbs sampler

settle(x, calc_update=True)

Run an iterative settling process to find latent states given clamped input and output variables.

Parameters

• x – sensory input to reconstruct/predict

• calc_update – if True, computes synaptic updates @ end of settling process for both NGC system and inference co-model (Default = True)

Returns

x_hat (predicted x)

calc_updates(avg_update=True, decay_rate=- 1.0)

Calculate adjustments to parameters under this given model and its current internal state values

Returns

delta, a list of synaptic updates (that follow order of pos_phase.theta)

clear()

Clears the states/values of the stateful nodes in this NGC system

References:
[1] Smolensky, P. “Information Processing in Dynamical Systems: Foundations of Harmony Theory.” Parallel distributed processing: explorations in the microstructure of cognition 1 (1986).
[2] Hinton, Geoffrey E. “Training products of experts by maximizing contrastive likelihood.” Technical Report, Gatsby computational neuroscience unit (1999).