|
Deep neural networks usually suffer from significant catastrophic forgetting when faced with class incremental tasks. Existing methods mostly use generative models to synthesize pseudo samples or save old exemplars to overcome the forgetting, but they are difficult to deal with the memory-constrained scenarios. In this paper we propose a Multi-Level Distillation and Continual Normalization (MLDCN) method which applies the framework of the exemplar-free method PASS. We first analyze the two bottlenecks in PASS: prototype mismatching problem and normalization preference for statistical properties of the current task. Hence, we propose MLDCN which contains a multi-level distillation framework to improve the model's ability to retain old knowledge. In addition, we introduce the continual normalization layer in the backbone to further enhance the stability of the model. Experimental results on CIFAR-100, ImageNet-sub show that our method can effectively alleviate the problem of catastrophic forgetting without saving old exemplars, and better preserve the knowledge of old categories in the incremental process. The performance of the proposed method outperforms many exemplar-free methods and several exemplar-replay methods.
|
