Equipartition and Geometry of Generative Representations

A growing number of results in artificial and biologic learning systems point at a possibility of spontaneous emergence of higher-level structures in representations of models of unsupervised generative self-learning. In this work we attempted to establish connections between principles of exchange of information in learning processes with origins in statistical thermodynamics of near-equilibrium ensembles and geometrical characteristics of distributions of data in the representations of generative learning models. It was demonstrated that under a set of identified common assumptions, the dynamics of information processes in the learning systems near equilibrium can lead to statistical preference for configurations with decoupled or categorized representations of common patterns or concepts in the sensory data. Observations of emergence of such structures were reported in the systems in different disciplines, from artificial learning systems to social networks and other artificial and natural learning systems.