Our paper titled “Return of the normal distribution: Flexible deep continual learning with variational auto-encoders” is now published in Neural Networks, Volume 154, Pages 397-412, October 2022.
In the paper we propose a novel two-stage variational auto-encoder approach to continual learning. Our approach comes with the advantage that previous assumptions in continual learning are optional, but can be accommodated if desired. Respectively, our proposed architecture allows for flexible choices to enable unsupervised, semi-supervised, or fully supervised continual learning. In essence, this fits into our view of continual learning: a framework should allow for flexible inclusion of additional assumptions, while arguably should not require strong simplifications per definition.
For more information, read the full paper or see the abstract below:
Learning continually from sequentially arriving data has been a long standing challenge in machine learning. An emergent body of deep learning literature suggests various solutions, through introduction of significant simplifications to the problem statement. As a consequence of a growing focus on particular tasks and their respective benchmark assumptions, these efforts are thus becoming increasingly tailored to specific settings. Whereas approaches that leverage Variational Bayesian techniques seem to provide a more general perspective of key continual learning mechanisms, they however entail their own caveats. Inspired by prior theoretical work on solving the prevalent mismatch between prior and aggregate posterior in deep generative models, we return to a generic variational auto- encoder based formulation and investigate its utility for continual learning. Specifically, we propose to adapt a two-stage training framework towards a context conditioned variant for continual learning, where we then formulate mechanisms to alleviate catastrophic forgetting through choices of generative rehearsal or well-motivated extraction of data exemplar subsets. Although the proposed generic two-stage variational auto-encoder is not tailored towards a particular task and allows for flexible amounts of supervision, we empirically demonstrate it to surpass task-tailored methods in both supervised classification, as well as unsupervised representation learning.
Abstract of our paper “Return of the normal distribution: Flexible deep continual learning with variational auto-encoders”