CANCELED: xLSTM and the age of recurrent neural networks

Thursday 16 Jan 2025, 15:00 → 17:00 Europe/Vienna

To participate in this event, please book at https://www.termino.gv.at/meet/de/b/7c0542521d5ca5b877fbd2315a20f393-392393

 

Location: Campus Akademie Seminar Room, Bäckerstraße

15:00 → 16:00 xLSTM and the age of recurrent neural networks

Since their inception in the early 1990, long short-term memory recurrent neural networks (LSTMs) revolutionized AI with their ability to manage long-term dependencies, playing a key role in early language models. However, neural networks based on Transformers later outperformed LSTMs by leveraging parallel processing and self-attention. This work revisits LSTMs and a potential new age of recurrent neural networks, asking: how well can they scale with billions of parameters and modern techniques? We introduce new methods to enhance LSTMs, including exponential gating and memory structure updates. These lead to sLSTM (simplified memory) and mLSTM (parallelizable memory). Combined into xLSTM architectures, these innovations make LSTMs competitive with state-of-the-art Transformers in both performance and scalability. We demonstrate applications of xLSTM beyond natural language processing, such as robotics, molecular biology, genetics, and chemistry.

Speaker: Günter Klambauer (JKU Linz)