Problem

Artificial neural networks, despite their capabilities, demand significant energy. This has spurred interest in neuromorphic computing, which trains physical systems to act as learning machines. However, training these systems, particularly with methods like Equilibrium Propagation (EP), often relies on theoretical models with dense, all-to-all connectivity. Such architectures are challenging to realize experimentally due to practical constraints like limited connectivity. This research addresses the gap by exploring the effectiveness of EP when applied to more realistic, sparsely connected network architectures.

Method

The study utilizes the XY model, a representative nonlinear system amenable to Equilibrium Propagation (EP) training. The researchers focused on two main architectural types: locally connected lattices and stacked lattices featuring local inter-layer connections. They systematically evaluated the performance and trainability of these networks using standard benchmark tasks. Specifically, the XOR task was used to assess nonlinear computation abilities, the Iris dataset explored the impact of system size, and the full-size MNIST dataset was employed to investigate the role of architectural choices within hidden layers, including weight sharing, channel count, and intra-layer connections. EP, which extracts approximate parameter gradients from energy-based models, served as the core training methodology.

Results

The findings indicate that sparse networks, characterized by only local connections, can achieve computational performance comparable to that of densely connected networks. During training on the XOR task, the evolution of spatially distributed responses and coupling strengths was observed. A key result from comparisons with all-to-all and dense-layer architectures is that locally connected lattices effectively reduce the number of necessary couplings while maintaining the network's functional capacity. Furthermore, the study explored the impact of architectural decisions like weight sharing, the number of channels, and intra-layer connections on network performance using the MNIST dataset.

Implications

The results carry significant implications for the design and practical implementation of neuromorphic systems. By demonstrating that sparse networks with local connections can perform comparably to dense networks, this research offers crucial guidelines for scaling up Equilibrium Propagation-based architectures in real-world scenarios. This work contributes to the development of more energy-efficient and experimentally viable physical learning machines by highlighting the potential of architecturally constrained networks.