In October 25, we had the pleasure to co-organize the Schloss Dagstuhl – Leibniz-Zentrum für Informatik (LZI) seminar on “Deep Continual Learning in the Foundation Model Era” with Joost van de Weijer (University of Barcelona), Tinne Tuytelaars (KU Leuven), and Christopher Kanan (University of Rochester), where we discussed the future of continual learning with a group of world-class researchers. Today, we are happy that our jointly developed roadmap, following a commonly identified theme of modular memory as the central element to unlock continual adaptation at scale, has been accepted as a spotlight position paper at ICML 2026!
Specifically, in our paper: “Modular Memory is the Key to Continual Learning Agents“, we outline the need to combine the strengths of In-Weight Learning and the emerging capabilities of In-Context learning through design of a modular memory framework. Thanks to everyone who contributed to this work! In particular to Vaggelis Dorovatas (Toyota Europe) as the lead author and OWL-ML’s Malte Schwerin as the second author, and to Rahaf Aljundi (Toyota Europe) & Jonghyun Choi (Seoul National University), for the joint efforts in coordinating the paper.
For more information, read the full paper. The abstract is provided below:
Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in continuous operation, experience accumulation, and personalization, capabilities that are central to adaptive intelligence. While continual learning research has long targeted these goals, its historical focus on in-weight learning (IWL), i.e., updating a single model’s parameters to absorb new knowledge, has rendered catastrophic forgetting a persistent challenge. Our position is that combining the strengths of In-Weight Learning (IWL) and the newly emerged capabilities of In-Context Learning (ICL) through the design of modular memory is the missing piece for continual adaptation at scale. We outline a conceptual framework for modular memory-centric architectures that leverage ICL for rapid adaptation and knowledge accumulation, and IWL for stable updates to model capabilities, charting a practical roadmap toward continually learning agents.