Speakers
Description
In the machine learning community, structured representations have demonstrated themselves to be hugely beneficial for efficient learning from limited data and generalization far beyond the training set. Examples of such structured representations include the spatially organized feature maps of convolutional neural networks, and the group structured activations of other equivariant models. To date however, the integration of such structured representations with deep neural networks has been limited to explicitly geometric transformations (such as spatial translation or rotation), known a priori to model developers. In the real world, we know that natural intelligence is able to efficiently handle novel transformations and flexibly generalize in a manner reminiscent of these structured artificial models, but crucially with respect to a broader class of non-geometric transformations. In this talk, we investigate how naturally intelligent systems might accomplish this through what we denote ‘fluid representations’. Specifically, we introduce a notion of generalized equivariance based on local reference frames at each point in representation space, and show how this novel type of structure can be induced in artificial neural networks through inductive biases originating from fluid dynamics and inspired by observed traveling waves in the brain. We show empirically that such models indeed learn 'approximately equivariant’ representations, similar to their explicitly geometrically structured counterparts, but in a much more flexible manner, where structure is learned directly from the data itself. We show that this structure both improves artificial neural networks, and simultaneously helps to understand observations from neuroscience itself.
