Researchers Build a Search Engine That Writes Its Own AI Learning Rules
New method automatically discovers symbolic, layer-local learning rules as sparse polynomials over tensor variables. Uses a Penrose-diagram-inspired dictionary of monomials and regresses coefficients to minimize a global loss.
Takeaways
- New method automatically discovers symbolic, layer-local learning rules as sparse polynomials over tensor variables.
- Uses a Penrose-diagram-inspired dictionary of monomials and regresses coefficients to minimize a global loss.
- Tested on classification, reconstruction, and self-supervised tasks with small MLPs on Sliced MNIST and MNIST .
- Beats frozen-first-layer PCA and ICA baselines, but does not fully match end-to-end backpropagation.
- Discovered rules are sparse and interpretable, unlike MLP-based meta-learned update rules.
- Targets neuromorphic, energy-efficient training and biologically plausible plasticity research; accepted to NeurIPS.
Backpropagation trains a neural network by sending error gradients from its output through every preceding layer. A new preprint proposes an automated search for layer-local update rules that can reduce a network-wide objective without requiring that backward chain during deployment. The search produces sparse polynomial equations that researchers can read and implement directly.
During ordinary backpropagation, each layer’s update depends on downstream computations and stored activations from the forward pass. That dependency creates communication and memory costs on conventional accelerators, complicates neuromorphic hardware, and differs from prevailing models of synaptic adaptation in biological brains.
A layer without an answer key
A local learning rule updates a layer’s weights from quantities available at that layer, such as its inputs, outputs, weights, and any target signal supplied to it. The central difficulty is credit assignment: an early layer must choose an update that improves the network’s final result without receiving the exact gradient of the final error.
- Hardware: Local updates could reduce the movement of gradients and activations across processors or physical circuits.
- Neuroscience: Layer-local signals more closely resemble models in which synapses adapt from nearby activity.
- Optimization: Existing local rules generally trail end-to-end backpropagation on deep-network benchmarks.
Methods including Oja’s rule, Equilibrium Propagation, Target Propagation, Forward-Forward, PEPITA, and predictive coding each specify a particular update mechanism. The new work treats the update equation as an object to be discovered from a structured search space.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
