State-space models (SSMs) are a powerful mathematical framework for describing dynamic systems using sets of first-order differential or difference equations governed by state variables. Recently, deep learning researchers have unlocked a fascinating capability of SSMs: constructing efficient memory representations for sequential data, including text, time series, and audio. The beauty of SSM-based architectures lies in their duality. On one hand, they can be trained in a fully parallel regime, similar to Transformers, allowing for the utilization of massive datasets and computational resources. On the other hand, inference can be performed with the efficiency and speed of Recurrent Neural Networks (RNNs), even for extremely long sequences like DNAs, audio, and lengthy texts.

In the past four years, several innovative variants of SSM-based deep learning architectures have been proposed, including S3, S4, H3, and S5, each bringing its unique contributions and advancements. Among these, Mamba, or S6, stands out as a cutting-edge SSM recently introduced by leading researchers Albert Gu and Tri Dao in their paper, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." Mamba is specifically designed to tackle sequences with highly complex structures, showcasing its advanced capabilities in this domain.

In this presentation, we embark on a journey through the evolution of SSMs, starting from their early beginnings and delving into the mathematical foundations that underpin their remarkable capabilities. We explore how SSMs have revolutionized our approach to sequential data, offering new avenues for efficient and effective processing of highly-complex sequential data.