Logic Journal of IGPL Advance Access published online on September 14, 2009
Logic Journal of IGPL, doi:10.1093/jigpal/jzp045
Semantic learning in autonomously active recurrent neural networks
Institute for Theoretical Physics, Goethe University Frankfurt, 60054 Frankfurt/Main, Germany.
E-mail: gros07{at}itp.uni-frankfurt.de
The human brain is autonomously active, being characterized by a self-sustained neural activity which would be present even in the absence of external sensory stimuli. Here we study the interrelation between the self-sustained activity in autonomously active recurrent neural nets and external sensory stimuli.
There is no a priori semantical relation between the influx of external stimuli and the patterns generated internally by the autonomous and ongoing brain dynamics. The question then arises when and how are semantic correlations between internal and external dynamical processes learned and built up?
We study this problem within the paradigm of transient state dynamics for the neural activity in recurrent neural nets, i.e. for an autonomous neural activity characterized by an infinite time-series of transiently stable attractor states. We propose that external stimuli will be relevant during the sensitive periods, viz the transition period between one transient state and the subsequent semi-stable attractor. A diffusive learning signal is generated unsupervised whenever the stimulus influences the internal dynamics qualitatively.
For testing we have presented to the model system stimuli corresponding to the bars and stripes problem. We found that the system performs a non-linear independent component analysis on its own, being continuously and autonomously active. This emergent cognitive capability results here from a general principle for the neural dynamics, the competition between neural ensembles.
Key Words: recurrent neural networks • autonomous neural dynamics • transient state dynamics • emergent cognitive capabilities
References
-
[1] Fiser J, Chiu C, Weliky M. Small modulation of ongoing cortical dynamics by sensory input during natural vision, Nature (2004) 431:573–578.[CrossRef][Medline]
[2] Abeles M, et al. Cortical activity flips among quasi-stationary states, PNAS (1995) 92:8616–8620.
[3] Ringach DL. States of mind, Nature (2003) 425:912–913.[Medline]
[4] Kenet T, Bibitchkov D, Tsodyks M, Grinvald A, Arieli A. Spontaneously emerging cortical representations of visual attributes, Nature (2003) 425:954–956.[CrossRef][Medline]
[5] Damoiseaux JS, Rombouts SARB, Barkhof F, Scheltens P, Stam CJ, Smith SM, Beckmann CF. Consistent resting-state networks across healthy subjects, PNAS (2006) 103:13848–13853.
[6] Honey CJ, Kötter R, Breakspear M, Sporns O. Network structure of cerebral cortex shapes functional connectivity on multiple time scales, PNAS (2007) 104:10240–10245.
[7] VanRullen R, Koch C. Is perception discrete or continuous?. Trends in Cognitive Sciences (2003) 5:207–213.
[8] Gros C. Complex and Adaptive Dynamical Systems, A Primer (2008) Springer.
[9] Edelman GM, Tononi GA. A Universe of Consciousness (2000) New York: Basic Books.
[10] Edelman GM. Naturalizing consciousness: A theoretical framework, PNAS (2003) 100:5520–5524.
[11] Dehaene S, Naccache L. Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework, Cognition (2003) 79:1–37.[CrossRef]
[12] Baars BJ, Franklin S. How conscious experience and working memory interact, Trends Coginitve Science (2003) 7:166–172.[CrossRef]
[13] Crick FC, Koch C. A framework for consciousness, Nature Neuroscience (2003) 6:119–126.[CrossRef][Web of Science][Medline]
[14] Koch C. The Quest for Consciousness - A Neurobiological Approach (2004) Robert and Company.
[15] Haykin S. Neural Networks: A Comprehensive Foundation (1994) Prentice Hall.
[16] Hopfield JJ. Neural Networks and Physical Systems with Emergent Collective Computational Abilities, PNAS (1982) 79:2554–2558.
[17] Gros C. Cognitive computation with autonomously active neural networks: An emerging field, Cognitive Computation (2009) in press.
[18] Eggermont JJ. Is There a Neural Code?. Neuroscience and Biobehavioral Reviews (1998) 22:355–370.[CrossRef][Web of Science][Medline]
[19] Jaeger H, Haas H. Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication, Science (2004) 304:78–80.
[20] Maass W, Natschlager T, Markram H. Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations, Neural Computation (2002) 14:2531–2560.[CrossRef][Web of Science][Medline]
[21] Rabinovich M, Volkovskii A, Lecanda P, Huerta R, Abarbanel HDI, Laurent G. Dynamical Encoding by Networks of Competing Neuron Groups: Winnerless Competition, Physical Review Letters (2001) 87:68102.[CrossRef]
[22] Olshausen BA, Field DJ. Sparse coding of sensory inputs, Current Opinion in Neurobiology (2004) 14:481–487.[CrossRef][Web of Science][Medline]
[23] Lin L, Osan R, Tsien JZ. Organizing principles of real-time memory encoding: neural clique assemblies and universal neural codes, Trends in Neurosciences (2006) 29:48–57.[CrossRef][Web of Science][Medline]
[24] Gros C. Self-Sustained Thought Processes in a Dense Associative Network—Furbach U, ed. (2005) Springer Lecture Notes in Artificial Intelligence 3698. 366–379. in KI 2005, also available as http://arxiv.org/abs/q-bio.NC/0508032.
[25] Gros C. Neural networks with transient state dynamics, New Journal of Physics (2007) 9:109.[CrossRef]
[26] Hyvärinen A, Oja E. Independent component analysis: Algorithms and applications, Neural Networks (2000) 13:411–430.[CrossRef][Web of Science][Medline]
[27] Chialvo DR, Bak P. Learning from mistakes, Neuroscience (1999) 90:1137–1148.[CrossRef][Web of Science][Medline]
[28] Choi S, Cichocki A, Park HM, Lee SY. Blind Source Separation and Independent Component Analysis: A Review, Neural Information Processing (2005) 6:1–57.
[29] Földiák P. Forming sparse representations by local anti-Hebbian learning, Biological Cybernetics (1990) 64:165–170.[CrossRef][Web of Science][Medline]
[30] Butko N, Triesch J. Learning Sensory Representations with Intrinsic Plasticity, Neurocomputing (2007) 70:1130–1138.[CrossRef][Web of Science]
[31] Dreyfus G. Neural Networks: Methodology and Applications (2005) Springer.
[32] Arbib MA. The Handbook of Brain Theory and Neural Networks (2002) MIT Press.
[33] Müller-Schloer C, von der Malsburg C, Würtz RP. ‘Organic Computing’, Informatik Spektrum (2004) 27:2–6.[CrossRef]
[34] Würtz RP. Organic Computing (2008) Springer Verlag.
[35] Elman JL. Finding structure in time, Cognitive Science (1990) 14:179–211.[CrossRef][Web of Science]
[36] Elman JL. An alternative view of the mental lexicon, Trends in Cognitive Sciences (2004) 8:301–306.[CrossRef][Web of Science][Medline]
[37] Seth AK, Edelman GM. Environment and Behavior Influence the Complexity of Evolved Neural Networks, Adaptive Behavior (2004) 12:5–20.[Abstract]
[38] Sporns O, Lungarella M. Evolving coordinated behavior by maximizing information structure. In: Proceedings of Artificial Life X—Rocha L, et al, eds. (2006) 3–7.
[39] Ay N, Bertschinger N, Der R, Güttler F, Olbrich E. Predictive information and explorative behavior of autonomous robots, European Physical Journal B (2008) 63:329–339.[CrossRef][Web of Science]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||