Selective Recurrent Neural Network

摘要

It is known that recurrent neural networks may have difficulties remembering data over long time lags. To overcome this problem, we propose an extended architecture of recurrent neural networks, which is able to deal with long time lags between relevant input signals. A register of latches at the input layer of the network is applied to bypass irrelevant input information and to propagate relevant inputs. The latches are implemented with differentiable multiplexers, thus enabling the derivatives to be propagated through the network. The relevance of input vectors is learned concurrently with the weights of the network using a gradient-based algorithm.