Traveling waves are a fundamental phenomenon in the brain, playing a crucial role in short-term information storage. In this work, we leverage the concept of traveling wave dynamics within a neural lattice to formulate a theoretical model of neural working memory, study its properties, and its real-world implications in practical neural networks. We first investigate the model’s capabilities in representing and learning state histories, vital for learning in history-dependent dynamical systems. The findings reveal that the wave memory stores external information and enhances the learning process by addressing the diminishing gradient problem. To understand the model’s real-world applicability, we explore two cases of the wave theory: linear boundary condition and non-linear, self-attention-driven boundary condition. The experiments reveal that the linear case emerges in Recurrent Neural Networks (RNNs) when trained on history-dependent dynamical systems by the backpropagation algorithm showing that RNNs leverage waves as a working memory store. Conversely, the non-linear scenario parallels the autoregressive loop of an attention-only transformer. Collectively, our findings suggest the broader relevance of traveling waves in AI and its potential in advancing our understanding and improving neural network architectures.