xLSTM and the age of recurrent neural networks

• Günter Klambauer (JKU Linz)

Since their inception in the early 1990, long short-term memory recurrent neural networks (LSTMs) revolutionized AI with their ability to manage long-term dependencies, playing a key role in early language models. However, neural networks based on Transformers later outperformed LSTMs by leveraging parallel processing and self-attention. This work revisits LSTMs and a potential new age of recurrent neural networks, asking: how well can they scale with billions of parameters and modern techniques? We introduce new methods to enhance LSTMs, including exponential gating and memory structure updates. These lead to sLSTM (simplified memory) and mLSTM (parallelizable memory). Combined into xLSTM architectures, these innovations make LSTMs competitive with state-of-the-art Transformers in both performance and scalability. We demonstrate applications of xLSTM beyond natural language processing, such as robotics, molecular biology, genetics, and chemistry.