Abstract
We have recently shown that when initiated with “small” weights, recurrent neural networks (RNNs) with standard sigmoid-type activation functions are inherently biased towards Markov models, i.e. even prior to any training, RNN dynamics can be readily used to extract finite memory machines [6,8]. Following [2], we refer to this phenomenon as the architectural bias of RNNs. In this paper we further extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We obtain both lower and upper bounds on various types of fractal dimensions, such as box-counting and Hausdorff dimensions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
M.F. Barnsley: Fractals everywhere. Academic Press, New York (1988)
M.H. Christiansen, N. Chater: Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23 (1999) 157–205
K. Falconer: Fractal Geometry-Mathematical Foundations and Applications. John Wiley & Sons, Chichester, UK (1990)
J.F. Kolen: Recurrent networks: state machines or iterated function systems? In: Proceedings of the 1993 Connectionist Models Summer School. Lawrence Erlbaum Associates, Hillsdale, NJ (1994) 203–210
J.F. Kolen: The origin of clusters in recurrent neural network state space. In: Proceedings from the Sixteenth Annual Conference of the Cognitive Science Society. Lawrence Erlbaum Associates, Hillsdale, NJ (1994) 508–513
B. Hammer, P. Tiňo: Neural networks with small weights implement finite memory machines. Technical Report P-241, Dept. of Math./Comp.Science, University of Osnabrück, Germany (2002)
P. Manolios and R. Fanelli: First order recurrent neural networks and deterministic finite state automata. Neural Computation 6 (1994) 1155–1173
P. Tiňo, M. Čerňanský, L. Beňušková: Markovian Architectural Bias of Recurrent Neural Networks. In: 2nd Euro-International Symposium on Computational Intelligence. Springer-Verlag Studies in CI (2002) to appear
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tiňo, P., Hammer, B. (2002). Architectural Bias in Recurrent Neural Networks — Fractal Analysis. In: Dorronsoro, J.R. (eds) Artificial Neural Networks — ICANN 2002. ICANN 2002. Lecture Notes in Computer Science, vol 2415. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-46084-5_219
Download citation
DOI: https://doi.org/10.1007/3-540-46084-5_219
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44074-1
Online ISBN: 978-3-540-46084-8
eBook Packages: Springer Book Archive