Logic Journal of IGPL Advance Access published online on September 3, 2009
Logic Journal of IGPL, doi:10.1093/jigpal/jzp044
Reservoir optimization in recurrent neural networks using properties of Kronecker product
École Polytechnique Fédéral de Lausanne, Laboratory of Nonlinear Systems, School of Computer and Communication Science, 1015 Lausanne, Switzerland.
E-mail: (ali.ajdarirad{at}epfl.ch, martin.hasler{at}epfl.ch)
Department of Computer Engineering, Sharif University of Technology, Tehran, Iran.
E-mail: mjalili{at}sharif.edu
Recurrent neural networks based on reservoir computing are increasingly being used in many applications. Optimization of the topological structure of the reservoir and the internal connection weights for a given task is one of the most important problems in reservoir computing. In this paper, considering the fact that one can construct a large matrix using Kronecker products of several small-size matrices, we propose a method to optimize the reservoir. Having a small number of parameters to optimize, a gradient based algorithm is applied to optimize parameters, and consequently the reservoir. In addition to reducing the number of parameters for optimization, potentially, the method is able to control several other properties of the reservoir such as spectral radius, sparsity, weight distribution and underlying connections, i.e. connection topology. To reveal the effectiveness of the proposed optimization method, the application to the following tasks are considered: Nonlinear autoregressive moving average and multiple superimposed oscillators. Simulation results show satisfactory performance of the method.
Key Words: Recurrent Neural Networks • Reservoir Computing • Echo State Networks • Optimization • Kronecker Product
References
-
[1] Ajdari Rad A, Jalili M, Hasler M. Reservoir optimization in recurrent neural networks using Kronecker kernels. (2008) 868–871. In Proceedings of the IEEE International Symposium on Circuits and Systems.
[2] Atiya AF, Parlos AG. New results on recurrent network training: unifying the algorithms and accelerating convergence, IEEE Transaction on Neural Networks (2000) 11(3):697–709.[CrossRef]
[3] Boyd S, Chua LO. Fading memory and the problem of approximating nonlinear operators with Volterra series, IEEE Transactions on Circuits and Systems (1985) 32:1150–1161.[CrossRef][Web of Science]
[4] Buehner M, Young P. A tighter bound for the echo state property, IEEE Transactions on Neural Networks (2006) 17(3):820–824.[CrossRef][Web of Science][Medline]
[5] Bush K, Tsendjav B. Improving the richness of echo state features using next ascent local search. (2005) 227–232. In Proceedings of the Artificial Neural Networks in Engineering Conference.
[6] Dutoit X, Van H, Nuttin M. Reservoir size reduction in echo state networks for classification problems. In: Proceedings of the Neural Information Processing Systems: Workshop on Echo State Networks and Liquid State Machines (2006).
[7] Dutoit X, Schrauwen B, Campenhout J, Van Stroobandt D, Van Brussel H, Nuttin M. Pruning and regularization in reservoir computing, Neurocomputing (2009) 72(7–9):1534–1546.[CrossRef][Web of Science]
[8] Feldkamp LA, Prokhorov DV, Eagen CF, Yuan F. Enhanced multi-stream Kalman filter training for recurrent neural networks. In: Nonlinear Modeling: Advanced Black-Box Techniques—Suykens J, Vandewalle J, eds. (1998) Kluwer. 29–54.
[9] Hammer B, Schrauwen B, Steil J. Recent advances in efficient learning of recurrent networks. (2009) Brugge: d-facto. 213–226. In Proceedings of European Symposium on Artificial Neural Networks (ESANN 2009).
[10] Hopfield J. Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences of the USA (1982) 79(8):2554–2558.
[11] Ishii K, van der Zant T, Vlatko Becanovic V, Ploger P. Identification of motion with echo state network. In: Proceedings of the IEEE OCEANS (2004) 1205–1210.
[12] Jaeger H. Short term memory in echo state networks. (2001) German National Research Center for Information Technology. GMD report 152.
[13] Jaeger H, Luko
evi
ius M, Popovici D, Siewert U. Optimization and applications of echo state networks with leaky integrator neurons, Neural Networks (2007) 20(3):335–352.[CrossRef][Web of Science][Medline]
[14] Jaeger H. The echo state approach to analysing and training recurrent neural networks. (2001) German National Research Center for Information Technology. GMD report 148.
[15] Jaeger H, Haas H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless telecommunication, Science (2004) 308(5667):78–80.[CrossRef][Web of Science]
[16] Jiang F, Berry H, Schoenauer M. Supervised and evolutionary learning of echo state networks. (2008) 215–224. Proceedings of 10th International Conference on Parallel Problem Solving From Nature.
[17] Joshi P, Maass W. Movement generation and control with generic neural microcircuits. In: Biologically Inspired Approaches to Advanced Information Technology, Lecture Notes in Computer Science—Ijspeert A, Murata M, Wakamiya N, eds. (2004) 3141:258–273.
[18] Kilian J, Siegelmann H. The dynamic universality of sigmoidal neural networks, Information and Computation (1996) 128:48–56.[CrossRef][Web of Science]
[19] Laub A. Matrix Analysis for Scientists and Engineers (2005) Philadelphia, PA: SIAM Publications.
[20] Leskovec J, Faloutsos C. Scalable modeling of real graphs using kronecker multiplication. (2007) 497–504. In Proceedings of the International Conference on Machine Learning.
[21] Lukoeviius M, Jaeger H. Overview of Reservoir Recipes. (2007) 11. Jacobs University Technical Report no.
[22] Lukoeviius M, Jaeger H. Reservoir Computing Approaches to Recurrent Neural Network Training, to appear. Computer Science Review (2009).
[23] Maass W, Natschlager T, Markram H. Fading memory and kernel properties of generic cortical microcircuit models, Journal of Physiology (2004) 98(4–6):315–330.
[24] Maass W, Natschlager T, Markram H. Realtime computing without stable states: a new framework for neural computation based on perturbations, Neural Computing (2002) 14(11):2531–2560.[CrossRef]
[25] Ozturk M, Xu D, Principe J. Analysis and design of echo state networks, Neural Computation (2007) 19(1):111–138.[CrossRef][Web of Science][Medline]
[26] Reinhart RF, Steil JJ. Recurrent Neural Associative Learning of Forward and Inverse Kinematics for Movement Generation of the Redundant PA-10 Robot. (2008) 35–40. ECSIS Symposium on Learning and Adaptive Behaviors for Robotic Systems.
[27] Schiller U, Steil J. Analyzing the weight dynamics of recurrent learning algorithms, Neurocomputing (2005) 63:5–23.[CrossRef][Web of Science]
[28] Schrauwen B, Verstraeten D, Van Campenhout J. An overview of reservoir computing: theory, applications and implementations. (2007) 471–482. In Proceedings of the 15th European Symposium on Artificial Neural Networks.
[29] Schrauwen B, Warderman M, Verstraeten D, Steil J, Stroobandt D. Improving reservoirs using intrinsic plasticity, Neurocomputing (2008) 71(7–9):1159–1171.[CrossRef][Web of Science]
[30] Steil J. Backpropagation-decorrelation: online recurrent learning with O(N) complexity. (2004) 843–848. In Proceedings of the International Joint Conference on Neural Networks.
[31] Steil J. Online stability of backpropagation-decorrelation recurrent learning, Neurocomputing (2006) 69:7–9.
[32] Steil J. Several ways to solve the MSO problem. (2007) 489–494. In Proceedings of the European Symposium on Artificial Neural Networks (ESANN).
[33] Steil J. Online reservoir adaptation by intrinsic plasticity for backpropagationdecorrelation and echo state learning, Neural Networks (2007) 20(3):353–364.[CrossRef][Web of Science][Medline]
[34] Verstraeten D, Schrauwen B, D‘Haene M, Stroobandt D. An experimental unification of reservoir computing methods, Neural Networks (2007) 20(3):391–403.[CrossRef][Web of Science][Medline]
[35] Werbos PJ. Backpropagation through time: what it does and how to do it, Proceedings of the IEEE (1990) 78(10):1550–1560.[CrossRef][Web of Science]
[36] Williams RJ, Zipper D. A learning algorithm for continuosly running fully recurrent neural networks, Neural Computation (1989) 1:270–280.[CrossRef]
[37] Wyffels F, Schrauwen B, Verstraeten D, Dirk S. Band-pass reservoir computing. Hou Z, Zhang N, eds. (2008) 3204–3209. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2008).
[38] Wyffels F, Schrauwen B, Dirk S. Stable Output Feedback in Reservoir Computing Using Ridge Regression, Lecture Notes In Computer Science. (2008) 5163:807–818. Proceedings of the 18th International Conference on Artificial Neural Networks (ICANN 2008).
[39] Xue Y, Yang L, Haykin S. Decoupled echo state networks with lateral inhibition, Neural Networks (2007) 20(3):365–376.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||