The Spatial Complexity of Inhomogeneous Multi-layer Neural Networks

摘要

Inhomogeneous multi-layer neural networks (IHMNNs) have been applied in various fields, for example, biological and ecological contexts. This work studies the learning problem of IHMNNs with an activation function \(f(x) = \dfrac{1}{2} (|x+1| - |x-1|)\) that derives from cellular neural networks, which can be adapted to the study of the vision systems of mammals. Applying the well-developed theory of symbolic dynamics, the explicit formulae of the topological entropy of the output and hidden spaces are given. We also demonstrate that, for any \(\lambda \in [0, \log 2]\) and \(\epsilon > 0\), parameters such that the topological entropy \(h\) of the hidden/output space of IHMNN that satisfies \(|h - \lambda | < \epsilon \) exists. This means that the collection of topological entropies is dense in the closed interval \([0, \log 2]\), which leads to the fact that IHMNNs are universal machines in some sense and hence are more efficient in learning algorithms. This paper aims to provide a mathematical foundation for the illustration of the capability of machine learning, while the method we have adopted can be extended to the investigation of multi-layer neural networks with other activation functions.