ngclearn.utils package

Subpackages

• ngclearn.utils.density package
  • Submodules
  • ngclearn.utils.density.gmm module
    • GMM
      • GMM.fit()
      • GMM.sample()
      • GMM.update()
  • Module contents
• ngclearn.utils.diffeq package
  • Submodules
  • ngclearn.utils.diffeq.ode_utils module
    • get_integrator_code()
    • step_euler()
    • step_heun()
    • step_rk2()
  • Module contents
• ngclearn.utils.optim package
  • Submodules
  • ngclearn.utils.optim.adam module
    • Adam
      • Adam.update()
    • step_update()
  • ngclearn.utils.optim.opt module
    • Opt
      • Opt.update()
  • ngclearn.utils.optim.sgd module
    • SGD
      • SGD.update()
    • step_update()
  • Module contents
• ngclearn.utils.viz package
  • Submodules
  • ngclearn.utils.viz.dim_reduce module
    • extract_pca_latents()
    • extract_tsne_latents()
    • plot_latents()
  • ngclearn.utils.viz.raster module
    • create_overlay_raster_plot()
    • create_raster_plot()
  • ngclearn.utils.viz.spike_plot module
    • plot_spiking_neuron()
  • ngclearn.utils.viz.synapse_plot module
    • make_video()
    • visualize()
    • visualize_frame()
    • visualize_gif()
    • visualize_labels()
  • Module contents

Submodules

ngclearn.utils.data_loader module

Data functions and utilies for data loading.

class ngclearn.utils.data_loader.DataLoader(design_matrices, batch_size, disable_shuffle=False, ensure_equal_batches=True, key=None)

Bases: object

A data loader object, meant to allow sampling w/o replacement of one or more named design matrices. Note that this object is iterable (and implements an __iter__() method).

Parameters:

• design_matrices – list of named data design matrices - [(“name”, matrix), …]

• batch_size – number of samples to place inside a mini-batch

• disable_shuffle – if True, turns off sample shuffling (thus no sampling w/o replacement)

• ensure_equal_batches – if True, ensures sampled batches are equal in size (Default = True). Note that this means the very last batch, if it’s not the same size as the rest, will reuse random samples from previously seen batches (yielding a batch with a mix of vectors sampled with and without replacement).

• key – PRNG key to control determinism of any underlying random values associated with this synaptic cable

ngclearn.utils.io_utils module

File and OS input/output (reading/writing) utilities.

ngclearn.utils.io_utils.deserialize(fname)

Deserialization (loading) routine

Parameters:

fname – file name from disk to deserialize

Returns:

deserialized object from disk

ngclearn.utils.io_utils.makedir(directory)

Creates a folder/directory on disk

Parameters:

directory – string name of directory/folder to create on disk

ngclearn.utils.io_utils.makedirs(directories)

Creates a set of folders/directories on disk

Parameters:

directories – array of string names of directories/folders to create on disk

ngclearn.utils.io_utils.serialize(fname, object)

Serialization (saving) routine

Parameters:

• fname – file name to serialize to on disk

• object – generic object to serialize

ngclearn.utils.metric_utils module

ngclearn.utils.metric_utils.measure_ACC(mu, y)

Calculates the accuracy (ACC) given a matrix of predictions and matrix of targets.

Parameters:

• mu – prediction (design) matrix

• y – target / ground-truth (design) matrix

Returns:

scalar accuracy score

ngclearn.utils.metric_utils.measure_BCE(p, x, offset=1e-07, preserve_batch=False)

Calculates the negative Bernoulli log likelihood or binary cross entropy (BCE). Note: If batch is preserved, this returns a column vector where each row is the BCE(p, x) for that row’s datapoint.

Parameters:

• p – predicted probabilities of shape; (N x D matrix)

• x – target binary values (data) of shape; (N x D matrix)

• offset – factor to control for numerical stability (Default: 1e-7)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.metric_utils.measure_CatNLL(p, x, offset=1e-07, preserve_batch=False)

Measures the negative Categorical log likelihood (Cat.NLL). Note: If batch is preserved, this returns a column vector where each row is the Cat.NLL(p, x) for that row’s datapoint.

Parameters:

• p – predicted probabilities; (N x C matrix, where C is number of categories)

• x – true one-hot encoded targets; (N x C matrix, where C is number of categories)

• offset – factor to control for numerical stability (Default: 1e-7)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.metric_utils.measure_KLD(p_xHat, p_x, preserve_batch=False)

Measures the (raw) Kullback-Leibler divergence (KLD), assuming that the two input arguments contain valid probability distributions (in each row, if they are matrices). Note: If batch is preserved, this returns a column vector where each row is the KLD(x_pred, x_true) for that row’s datapoint.

Formula:

KLD(p_xHat, p_x) = (1/N) [ sum_i(p_x * jnp.log(p_x)) - sum_i(p_x * jnp.log(p_xHat)) ]

where sum_i implies summing across dimensions of vector-space of p_x

Parameters:

• p_xHat – predicted probabilities; (N x C matrix, where C is number of categories)

• p_x – ground true probabilities; (N x C matrix, where C is number of categories)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.metric_utils.measure_MSE(mu, x, preserve_batch=False)

Measures mean squared error (MSE), or the negative Gaussian log likelihood with variance of 1.0. Note: If batch is preserved, this returns a column vector where each row is the MSE(mu, x) for that row’s datapoint.

Parameters:

• mu – predicted values (mean); (N x D matrix)

• x – target values (data); (N x D matrix)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.metric_utils.measure_sparsity(codes, tolerance=0.0)

Calculates the sparsity (ratio) of an input matrix, assuming each row within it is a non-negative vector.

Parameters:

• codes – matrix (shape: N x D) of non-negative codes to measure sparsity of (per row)

• tolerance – lowest number to consider as “empty”/non-existent (Default: 0.)

Returns:

N x 1)

Return type:

sparsity measurements per code (output shape

ngclearn.utils.model_utils module

ngclearn.utils.model_utils.binarize(data, threshold=0.5)

Converts the vector data to its binary equivalent

Parameters:

• data – the data to binarize (real-valued)

• threshold – the cut-off point for 0, i.e., if threshold = 0.5, then any number/value inside of data < 0.5 is set to 0, otherwise, it is set to 1.0

Returns:

the binarized equivalent of “data”

ngclearn.utils.model_utils.clamp_max(x, max_val)

Clamps values in data x that exceed a maximum value to that value.

Parameters:

• x – data to upper-bound clamp

• max_val – maximum value threshold

Returns:

x with maximum clamped values

ngclearn.utils.model_utils.clamp_min(x, min_val)

Clamps values in data x that exceed a minimum value to that value.

Parameters:

• x – data to lower-bound clamp

• min_val – minimum value threshold

Returns:

x with minimum clamped values

ngclearn.utils.model_utils.create_function(fun_name, args=None)

Activation function creation routine.

Parameters:

fun_name – string name of activation function to produce (Currently supports: “tanh”, “relu”, “lrelu”, “identity”)

Returns:

function fx, first derivative of function (w.r.t. input) dfx

ngclearn.utils.model_utils.d_heaviside(x)

ngclearn.utils.model_utils.d_identity(x)

Derivative of the identity function.

Parameters:

x – input (tensor) value

Returns:

output (tensor) derivative value (with respect to input argument)

ngclearn.utils.model_utils.d_lrelu(x)

Derivative of the leaky linear rectifier.

Parameters:

x – input (tensor) value

Returns:

output (tensor) derivative value (with respect to input argument)

ngclearn.utils.model_utils.d_relu(x)

Derivative of the linear rectifier.

Parameters:

x – input (tensor) value

Returns:

output (tensor) derivative value (with respect to input argument)

ngclearn.utils.model_utils.d_relu6(x)

Derivative of the bounded leaky linear rectifier (upper bounded at 6).

Parameters:

x – input (tensor) value

Returns:

output (tensor) derivative value (with respect to input argument)

ngclearn.utils.model_utils.d_sigmoid(x)

ngclearn.utils.model_utils.d_softplus(x)

Derivative of the softplus function.

Parameters:

x – input (tensor) value

Returns:

output (tensor) derivative value (with respect to input argument)

ngclearn.utils.model_utils.d_tanh(x)

Derivative of the hyperbolic tangent function.

Parameters:

x – input (tensor) value

Returns:

output (tensor) derivative value (with respect to input argument)

ngclearn.utils.model_utils.d_threshold(x, thr=1.0)

ngclearn.utils.model_utils.drop_out(dkey, input, rate=0.0)

Applies a drop-out transform to an input matrix.

Parameters:

• dkey – Jax randomness key for this operator

• input – data to apply random/drop-out mask to

• rate – probability of a dimension being dropped

Returns:

output as well as binary mask

ngclearn.utils.model_utils.heaviside(x)

ngclearn.utils.model_utils.identity(x)

The identity function: x = f(x).

Parameters:

x – input (tensor) value

Returns:

output (tensor) value

ngclearn.utils.model_utils.initialize_params(dkey, initKernel, shape)

Creates the intiial condition values for a parameter tensor.

Parameters:

• dkey – PRNG key to control determinism of this routine

• initKernel –

  triplet/3-tuple with 1st element as a string calling the name of initialization scheme to use

  Note:

  Currently supported kernel schemes include: (“hollow”, off_diagonal_scale, ~ignored~); (“eye”, diagonal_scale, ~ignored~); (“uniform”, min_val, max_val); (“gaussian”, mu, sigma) OR (“normal”, mu, sigma); (“constant”, magnitude, ~ignored~)

• shape – tuple containing the dimensions/shape of the tensor to initialize

Returns:

output (tensor) value

ngclearn.utils.model_utils.inverse_logistic(x, clip_bound=0.03)

The inverse logistic link - logit function.

Parameters:

• x – data to transform via inverse logistic function

• clip_bound – pre-processing lower/upper bounds to enforce on data before applying inverse logistic

Returns:

x transformed via inverse logistic function

ngclearn.utils.model_utils.inverse_tanh(x)

The inverse hyperbolic tangent.

Parameters:

• x – data to transform via inverse hyperbolic tangent

• clip_bound – pre-processing lower/upper bounds to enforce on data before applying inverse hyperbolic tangent

Returns:

x transformed via inverse hyperbolic tangent

ngclearn.utils.model_utils.lrelu(x)

The leaky linear rectifier: max(0, x) if x >= 0, 0.01 * x if x < 0 = f(x).

Parameters:

x – input (tensor) value

Returns:

output (tensor) value

ngclearn.utils.model_utils.measure_ACC(mu, y)

Calculates the accuracy (ACC) given a matrix of predictions and matrix of targets.

Parameters:

• mu – prediction (design) matrix

• y – target / ground-truth (design) matrix

Returns:

scalar accuracy score

ngclearn.utils.model_utils.measure_BCE(p, x, offset=1e-07, preserve_batch=False)

Calculates the negative Bernoulli log likelihood or binary cross entropy (BCE). Note: If batch is preserved, this returns a column vector where each row is the BCE(p, x) for that row’s datapoint.

Parameters:

• p – predicted probabilities of shape; (N x D matrix)

• x – target binary values (data) of shape; (N x D matrix)

• offset – factor to control for numerical stability (Default: 1e-7)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.model_utils.measure_CatNLL(p, x, offset=1e-07, preserve_batch=False)

Measures the negative Categorical log likelihood (Cat.NLL). Note: If batch is preserved, this returns a column vector where each row is the Cat.NLL(p, x) for that row’s datapoint.

Parameters:

• p – predicted probabilities; (N x C matrix, where C is number of categories)

• x – true one-hot encoded targets; (N x C matrix, where C is number of categories)

• offset – factor to control for numerical stability (Default: 1e-7)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.model_utils.measure_KLD(p_xHat, p_x, preserve_batch=False)

Measures the (raw) Kullback-Leibler divergence (KLD), assuming that the two input arguments contain valid probability distributions (in each row, if they are matrices). Note: If batch is preserved, this returns a column vector where each row is the KLD(x_pred, x_true) for that row’s datapoint.

Formula:

KLD(p_xHat, p_x) = (1/N) [ sum_i(p_x * jnp.log(p_x)) - sum_i(p_x * jnp.log(p_xHat)) ]

where sum_i implies summing across dimensions of vector-space of p_x

Parameters:

• p_xHat – predicted probabilities; (N x C matrix, where C is number of categories)

• p_x – ground true probabilities; (N x C matrix, where C is number of categories)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.model_utils.measure_MSE(mu, x, preserve_batch=False)

Measures mean squared error (MSE), or the negative Gaussian log likelihood with variance of 1.0. Note: If batch is preserved, this returns a column vector where each row is the MSE(mu, x) for that row’s datapoint.

Parameters:

• mu – predicted values (mean); (N x D matrix)

• x – target values (data); (N x D matrix)

• preserve_batch – if True, will return one score per sample in batch (Default: False), otherwise, returns scalar mean score

Returns:

an (N x 1) column vector (if preserve_batch=True) OR (1,1) scalar otherwise

ngclearn.utils.model_utils.normalize_matrix(M, wnorm, order=1, axis=0)

Normalizes the values in matrix to have a particular norm across each vector span.

Parameters:

• M – (2D) matrix to normalize

• wnorm – target norm for each

• order – order of norm to use in normalization (Default: 1); note that ord=1 results in the L1-norm, ord=2 results in the L2-norm

• axis – 0 (apply to column vectors), 1 (apply to row vectors)

Returns:

a normalized value matrix

ngclearn.utils.model_utils.one_hot(P)

Converts a matrix of probabilities to a corresponding binary one-hot matrix (each row is a one-hot encoding).

Parameters:

P – a probability matrix where each row corresponds to a particular data probability vector

Returns:

the one-hot encoding (matrix) of probabilities in P

ngclearn.utils.model_utils.pull_equations(controller)

Extracts the dynamics string of this controller (model/system).

Parameters:

controller – model/system to extract dynamics equation(s) from

Returns:

string containing this model/system’s dynamics equation(s)

ngclearn.utils.model_utils.relu(x)

The linear rectifier: max(0, x) = f(x).

Parameters:

x – input (tensor) value

Returns:

output (tensor) value

ngclearn.utils.model_utils.relu6(x)

The linear rectifier upper bounded at the value of 6: min(max(0, x), 6.).

Parameters:

x – input (tensor) value

Returns:

output (tensor) value

ngclearn.utils.model_utils.sigmoid(x)

ngclearn.utils.model_utils.softmax(x, tau=0.0)

Softmax function with overflow control built in directly. Contains optional temperature parameter to control sharpness (tau > 1 softens probs, < 1 sharpens –> 0 yields point-mass).

Parameters:

• x – a (N x D) input argument (pre-activity) to the softmax operator

• tau – probability sharpening/softening factor

Returns:

a (N x D) probability distribution output block

ngclearn.utils.model_utils.softplus(x)

The softplus elementwise function.

Parameters:

x – input (tensor) value

Returns:

output (tensor) value

ngclearn.utils.model_utils.tanh(x)

The hyperbolic tangent function.

Parameters:

x – input (tensor) value

Returns:

output (tensor) value

ngclearn.utils.model_utils.threshold(x, thr=1.0)

ngclearn.utils.model_utils.threshold_cauchy(x, lmbda)

A Cauchy distributional threshold routine applied to each dimension of input

Parameters:

• x – data to apply threshold function over

• lmbda – scalar to control strength/influence of Cauchy thresholding

Returns:

thresholded x

ngclearn.utils.model_utils.threshold_soft(x, lmbda)

A soft threshold routine applied to each dimension of input

Parameters:

• x – data to apply threshold function over

• lmbda – scalar to control strength/influence of thresholding

Returns:

thresholded x

ngclearn.utils.patch_utils module

ngclearn.utils.patch_utils.generate_patch_set(x_batch_, patch_size=(8, 8), max_patches=50, center=True)

Generates a set of patches from an array/list of image arrays (via random sampling with replacement). This uses scikit-learn’s patch creation function to generate a set of (px x py) patches. Note: this routine also subtracts each patch’s mean from itself.

Parameters:

• imgs – the array of image arrays to sample from

• patch_size – a 2-tuple of the form (pH = patch height, pW = patch width)

• max_patches – maximum number of patches to extract/generate from source images

• center – centers each patch by subtracting the patch mean (per-patch)

Returns:

an array (D x (pH * pW)), where each row is a flattened patch sample

ngclearn.utils.surrogate_fx module

A builder sub-package for spike emission functions that are mapped to surrogate (derivative) functions; these function builders are useful if differentiation through the discrete spike emission steps in spiking neuronal cells is required (e.g., cases of surrogate backprop, broadcast feedback alignment schemes, etc.). Calling the builder estimator functions below returns the following routines: 1) a spike emission routine spike_fx; 2) the surrogate function used to approximate spike emission surr_fx; 3) the corresponding surrogate derivative routine d_spike_fx.

ngclearn.utils.surrogate_fx.arctan_estimator(get_surr_fx=False)

The arctan surrogate gradient estimator for binary spike emission.

E(x) = (1/pi) arctan(pi * x)

dE(x)/dx = (1/pi) (1/(1 + (pi * x)^2))

where x = v (membrane potential/voltage)

Returns:

( spike_fx(x, thr), d_spike_fx(x, thr) ) OR ( spike_fx(x, thr), surr_fx(x, thr, args), d_spike_fx(x, thr, args) )

ngclearn.utils.surrogate_fx.secant_lif_estimator(get_surr_fx=False)

Surrogate function for computing derivative of (binary) spike function with respect to the input electrical current/drive to a leaky integrate-and-fire (LIF) neuron. (Note this is only useful for LIF neuronal dynamics.)

spike_fx(x) ~ E(x) = sech(x) = 1/cosh(x), cosh(x) = (e^x + e^(-x))/2

dE(x)/dx = (c1 c2) * sech^2(c2 * x) for x > 0 and 0 for x <= 0

where x = j (electrical current)

Reference:

Samadi, Arash, Timothy P. Lillicrap, and Douglas B. Tweed. “Deep learning with

dynamic spiking neurons and fixed feedback weights.” Neural computation 29.3

(2017): 578-602.

Parameters:

get_surr_fx – if True, makes this function also return the surrogate function that the derivative corresponds to

Returns:

( spike_fx(x, thr), d_spike_fx(x, thr) ) OR ( spike_fx(x, thr), surr_fx(x, thr, args), d_spike_fx(x, thr, args) )

ngclearn.utils.surrogate_fx.straight_through_estimator(get_surr_fx=False)

The straight-through estimator (STE) applied to binary spike emission (the Heaviside function).

Bengio, Yoshua, Nicholas Léonard, and Aaron Courville. “Estimating or

propagating gradients through stochastic neurons for conditional

computation.” arXiv preprint arXiv:1308.3432 (2013).

Returns:

( spike_fx(x, thr), d_spike_fx(x, thr) ) OR ( spike_fx(x, thr), surr_fx(x, thr, args), d_spike_fx(x, thr, args) )

ngclearn.utils.surrogate_fx.triangular_estimator(get_surr_fx=False)

The triangular surrogate gradient estimator for binary spike emission.

Returns:

( spike_fx(x, thr), d_spike_fx(x, thr) ) OR ( spike_fx(x, thr), surr_fx(x, thr, args), d_spike_fx(x, thr, args) )

Module contents