How Online Learning Approaches Ornstein Uhlenbeck Processes

摘要

We show that under reasonable conditions, online learning near a local minimum is similar to a multivariate Ornstein Uhlenbeck process. This implies that the parameter state oscillates randomly around the minimum point, with a Gaussian limiting distribution. We also develop a simple hypothesis test that detects Ornstein Uhlenbeck properties without storing the history of the learning process.