Logic Journal of IGPL Advance Access published online on September 3, 2009
Logic Journal of IGPL, doi:10.1093/jigpal/jzp044
Reservoir optimization in recurrent neural networks using properties of Kronecker product
École Polytechnique Fédéral de Lausanne, Laboratory of Nonlinear Systems, School of Computer and Communication Science, 1015 Lausanne, Switzerland.
E-mail: (ali.ajdarirad{at}epfl.ch, martin.hasler{at}epfl.ch)
Department of Computer Engineering, Sharif University of Technology, Tehran, Iran.
E-mail: mjalili{at}sharif.edu
 |
Abstract |
|---|
Recurrent neural networks based on reservoir computing are increasingly being used in many applications. Optimization of the topological structure of the reservoir and the internal connection weights for a given task is one of the most important problems in reservoir computing. In this paper, considering the fact that one can construct a large matrix using Kronecker products of several small-size matrices, we propose a method to optimize the reservoir. Having a small number of parameters to optimize, a gradient based algorithm is applied to optimize parameters, and consequently the reservoir. In addition to reducing the number of parameters for optimization, potentially, the method is able to control several other properties of the reservoir such as spectral radius, sparsity, weight distribution and underlying connections, i.e. connection topology. To reveal the effectiveness of the proposed optimization method, the application to the following tasks are considered: Nonlinear autoregressive moving average and multiple superimposed oscillators. Simulation results show satisfactory performance of the method.
Key Words: Recurrent Neural Networks • Reservoir Computing • Echo State Networks • Optimization • Kronecker Product