A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application [paper] | [blog]
Continual learning, also known as incremental learning or life-long learning, stands at the forefront of deep learning and AI systems. It breaks through the obstacle of one-way training on close sets and enables continuous adaptive learning on open-set conditions. In the recent decade, continual learning has been explored and applied in multiple fields especially in computer vision covering classification, detection and segmentation tasks. Continual semantic segmentation (CSS), of which the dense prediction peculiarity makes it a challenging, intricate and burgeoning task. In this paper, we present a review of CSS, committing to building a comprehensive survey on problem formulations, primary challenges, universal datasets, neoteric theories and multifarious applications. Concretely, we begin by elucidating the problem definitions and primary challenges. Based on an in-depth investigation of relevant approaches, we sort out and categorize current CSS models into two main branches including data-replay and data-free sets. In each branch, the corresponding approaches are similarity-based clustered and thoroughly analyzed, following qualitative comparison and quantitative reproductions on relevant datasets. Besides, we also introduce four CSS specialities with diverse application scenarios and development tendencies. Furthermore, we develop a benchmark for CSS encompassing representative references, evaluation results and reproductions. We hope this survey can serve as a reference-worthy and stimulating contribution to the advancement of the life-long learning field, while also providing valuable perspectives for related fields.
- Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation [WACV 2025] [paper]
- Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images [ACMMM 2024] [paper] [code]
- Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation [ECCV 2024] [paper] [code]
- Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation [MICCAI 2024] [paper]
- Federated Cross-Incremental Self-Supervised Learning for Medical Image Segmentation [TNNLS 2024] [paper]
- CLMS: Bridging Domain Gaps in Medical Imaging Segmentation with Source-Free Continual Learning for Robust Knowledge Transfer and Adaptation [Medical Image Analysis 2024] [paper]
- AWF: Adaptive Weight Fusion for Enhanced Class Incremental Semantic Segmentation [ArXiv 2024] [paper]
- CIT: Rethinking Class-incremental Semantic Segmentation with a Class Independent Transformation [ArXiv 2024] [paper]
- Domain-Incremental Learning for Remote Sensing Semantic Segmentation With Multifeature Constraints in Graph Space [TGRS 2024] [paper]
- kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies [TMLR 2024] [paper]
- Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentation. [ECCV 2024] [paper] [code]
- Mitigating Background Shift in Class-Incremental Semantic Segmentation [ECCV 2024] [paper] [code]
- Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation [ECCV 2024] [paper] [code]
- L2A: Learning Affinity from Attention for Weakly Supervised Continual Semantic Segmentation [TCSVT 2024] [paper]
- Prompt-Guided Semantic-aware Distillation for
Weakly Supervised Incremental Semantic
Segmentation. [TCSVT 2024] [paper] [code]
- Cs2K: Class-specific and Class-shared Knowledge Guidance for Incremental Semantic Segmentation. [ArXiv 2024] [paper]
- Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation [CVPR 2024] [paper]
- CoMasTRe: Continual Segmentation with Disentangled Objectness Learning and Class Recognition [CVPR 2024] [paper]
- ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning [CVPR 2024] [paper]
- SimCS: Simulation for Domain Incremental Online Continual Segmentation [AAAI 2024] [paper]
- SegViT v2: [SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers] [IJCV 2024] [paper]
- Tendency-driven Mutual Exclusivity for Weakly Supervised Incremental Semantic Segmentation [Arxiv 2024] [paper]
- Layer-Specific Knowledge Distillation for Class Incremental Semantic Segmentation [TIP 2024] [paper]
- Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery [TMI 2024] [paper]
- ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer [ArXiv 2024] [paper]
- A Domain Adaptive Semantic Segmentation Method Using Contrastive Learning and Data Augmentation [Neural Processing Letters 2024] [paper]
- Boosting knowledge diversity, accuracy, and stability via tri-enhanced distillation for domain continual medical image segmentation [Medical Image Analysis 2024] [paper]
- Cross-Domain Few-Shot Incremental Learning for Point-Cloud Recognition. [WACV2024] [paper]
- Towards Domain-Aware Knowledge Distillation for Continual Model Generalization. [WACV 2024] [paper]
- MDINet: Multi-Domain Incremental Network for Change Detection. [TGRS 2024] [paper]
- Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments [NeurIPS 2023] [paper]
- CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation. [ArXiv 2023] [paper]
- FALCON: Fairness Learning via Contrastive Attention Approach to Continual Semantic Scene Understanding in Open World. [ArXiv 2023] [paper]
- L2MNet: Enhancing Continual Semantic Segmentation with Mask Matching [PRCV 2023] [paper]
- SDCL: Subspace Distillation for Continual Learning [[Neural Networks 2024]] [paper] [code]
- Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory [TPAMI 2023] [paper]
- CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation [ICCV 2023] [paper] [code]
- Class-incremental Continual Learning for Instance Segmentation with Image-level Weak Supervision. [ICCV 2023] [paper] [code]
- Effects of architectures on continual semantic segmentation. [ArXiv 2023] [paper]
- LGKD: [Label-Guided Knowledge Distillation for Continual Semantic Segmentation on 2D Images and 3D Point Clouds] [ICCV 2023] [paper]
- Preparing the Future for Continual Semantic Segmentation. [ICCV 2023] [paper]
- GSC: [Gradient-Semantic Compensation for Incremental Semantic Segmentation] [TMM 2023] [paper]
- MiCro: [MiCro: Modeling Cross-Image Semantic Relationship Dependencies for Class-Incremental Semantic Segmentation in Remote Sensing Images.] [TGRS 2023] [paper]
- CL-PCSS: [Continual Learning for LiDAR Semantic Segmentation: Class-Incremental and Coarse-to-Fine strategies on Sparse Data] [CVPR 2023] [paper]
- AWT: [Attribution-aware Weight Transfer: A Warm-Start Initialization for Class-Incremental Semantic Segmentation] [WACV 2023] [paper]
- EWF: [Endpoints Weight Fusion for Class Incremental Semantic Segmentation] [CVPR 2023] [paper] [code]
- SATS: [SATS: Self-attention transfer for continual semantic segmentation] [PR 2023] [paper] [code]
- Incrementer: [Incrementer: Transformer for Class-Incremental Semantic Segmentation with Knowledge Distillation Focusing on Old Class][CVPR 2023][paper]
- S3R: [S3R: Shape and Semantics-Based Selective Regularization for Explainable Continual Segmentation Across Multiple Sites] [TMI 2023] [paper]
- ContinualPMF:[Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network] [ArXiv 2023] [paper]
- CoMFormer: [CoMFormer: Continual Learning in Semantic and Panoptic Segmentation] [CVPR 2023] [paper]
- DILRS: [DILRS: Domain-Incremental Learning for Semantic Segmentation in Multi-Source Remote Sensing Data] [Remote Sensing 2023] [paper]
- RaSP: [RaSP: Relation-aware Semantic Prior for Weakly Supervised Incremental Segmentation] [ArXiv 2023] [paper]
- Efficient Multi-Grained Knowledge Reuse for Class Incremental Segmentation. [ArXiv 2023] [paper]
- CISDQ: Continual Learning for Image Segmentation with Dynamic Query. [TCSVT 2023] [paper]
- Multi-Head Distillation for Continual Unsupervised Domain Adaptation in Semantic Segmentation. [CVPRW 2022] [paper]
- CONDA: Continual Unsupervised Domain Adaptation Learning in Visual Perception for Self-Driving Cars. [ArXiv 2022] [paper]
- iFS-RCNN: An Incremental Few-shot Instance Segmenter [CVPR 2022] [paper]
- WILSON: [Incremental Learning in Semantic Segmentation From Image Labels] [CVPR 2022] [paper] [code]
- RBC: [RBC: Rectifying the Biased Context in Continual Semantic Segmentation] [ECCV 2022] [paper] [code]
- CBNA: [Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation] [TITS 2022] [paper] [code]
- ACD: [A contrastive distillation approach for incremental semantic segmentation in aerial images] [ICIAP 2022] [paper] [code]
- TANet: [Class-Incremental Learning Network for Small Objects Enhancing of Semantic Segmentation in Aerial Imagery] [TGRS 2022] [paper]
- ST-CISS: [Self-Training for Class-Incremental Semantic Segmentation] [TNNLS 2022] [paper]
- RCIL: [Representation Compensation Networks for Continual Semantic Segmentation] [CVPR 2022] [paper]
- CAF：[Continual attentive fusion for incremental learning in semantic segmentation] [TMM 2022] [paper]
- UCD: [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation] [TPAMI 2022] [paper]
- REMINDER: [Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation] [CVPR 2022] [paper]
- DFD-LM: [Class-Incremental Learning for Semantic Segmentation in Aerial Imagery via Distillation in All Aspects] [TGRS 2022] [paper]
- SDR: [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations] [CVPR 2021] [paper]
- PLOP: [PLOP: Learning without Forgetting for Continual Semantic Segmentation] [CVPR 2021] [paper]
- MiB: [Modeling the Background for Incremental Learning in Semantic Segmentation] [CVPR 2020] [paper]
- ILT: [Incremental Learning Techniques for Semantic Segmentation] [ICCVW 2019] [paper]
- Learning At a Glance: Towards Interpretable Data-Limited Continual Semantic Segmentation Via Semantic-Invariance Modelling. [TPAMI 2024] [paper] [code]
- Task Consistent Prototype Learning for Incremental Few-Shot Semantic Segmentation [ICPR 2024] [paper]
- Recall-Based Knowledge Distillation for Data Distribution Based Catastrophic Forgetting in Semantic Segmentation [ICPR 2024] [paper]
- Learning from the Web: Language Drives Weakly-Supervised Incremental Learning for Semantic Segmentation. [ECCV 2024] [paper]
- Learning with Style: Continual Semantic Segmentation Across Tasks and Domains. [TPAMI 2024] [[paper]]
- Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting. [MICCAI24] [paper] [code]
- Incremental Model Enhancement via Memory-based Contrastive Learning. [IJCV 2024] [paper]
- Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting [ArXiv 2024] [paper]
- A Dual Enrichment Synergistic Strategy to Handle Data Heterogeneity for Domain Incremental Cardiac Segmentation [TMI 2024] [paper]
- BACS: Background Aware Continual Semantic Segmentation. [ArXiv 2024] [paper]
- MiSSNet: Memory-inspired Semantic Segmentation Augmentation Network for Class-Incremental Learning in Remote Sensing Images [TGRS 2024] [paper]
- ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer [ArXiv 2024] [paper]
- TIKP: Text-to-Image Knowledge Preservation for Continual Semantic Segmentation [AAAI 2024] [paper]
- Continual atlas-based segmentation of prostate MRI. [WACV 2024] [paper] [code]
- Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation [NeurIPS 2023] [paper]
- Replaying Styles for Continual Semantic Segmentation Across Domains [ACPR 2023] [paper]
- Continual Semantic Segmentation via Scalable Contrastive Clustering and Background Diversity [ICDM 2023] [paper]
- FairCL: [Fairness Continual Learning Approach to Semantic Scene Understanding in Open-World Environments] [NeurIPS 2023] [paper]
- DiffusePast: [DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation] [ArXiv 2023] [paper]
- FMWILSS: [Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation] [CVPR 2023] [paper]
- AMSS: [Continual Semantic Segmentation With Automatic Memory Sample Selection] [CVPR 2023] [paper]
- RECALL+: [Adversarial Web-based Replay for Continual Learning in Semantic Segmentation] [ArXiv 2023] [paper]
- EndoCSS: [Rethinking exemplars for continual semantic segmentation in endoscopy scenes: Entropy-based mini-batch pseudo-replay] [CBM 2023] [paper]
- Domain-Incremental Cardiac Image Segmentation With Style-Oriented Replay and Domain-Sensitive Feature Whitening [TMI 2023] [paper]
- GarDA: [Generative appearance replay for continual unsupervised domain adaptation] [Medical Image Analysis 2023] [paper]
- ProCA: [Prototype-guided continual adaptation for class-incremental unsupervised domain adaptation] [ECCV 2022] [paper] [code]
- SPPA: [Continual Semantic Segmentation via Structure Preserving and Projected Feature Alignment] [ECCV 2022] [paper] [code]
- ALIFE: [Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation] [NeurIPS 2022] [paper] [code]
- DKD: [Decomposed knowledge distillation for class-incremental semantic segmentation] [NeurIPS 2022] [paper] [code]
- MicroSeg: [Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation] [NeurIPS 2022] [paper] [code]
- CCDA: [Continual coarse-to-fine domain adaptation in semantic segmentation] [IVC 2022] [paper]
- Continual Adaptation of Semantic Segmentation Using Complementary 2D-3D Data Representations [LRA 2022] [paper]
- Improving replay-based continual semantic segmentation with smart data selection [ITSC 2022] [paper]
- SIL-LAND: Segmentation Incremental Learning in Aerial Imagery via LAbel Number Distribution Consistency [TGRS 2022] [paper]
- SSUL: [SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning] [NeurIPS 2021] [paper] [code]
- RECALL: [RECALL: Replay-based Continual Learning in Semantic Segmentation] [ICCV 2021] [paper] [code]
- FSS: [Prototype-based Incremental Few-Shot Semantic Segmentation] [BMVC 2021] [paper] [code]
- SRAA: [Advancing Incremental Few-shot Semantic Segmentation via Semantic-guided Relation Alignment and Adaptation] [MMM 2024] [paper]
- FSCILSS: [Few-Shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge Distillation] [ISPDS 2023] [paper]
- GAPS: [GAPS: Few-Shot Incremental Semantic Segmentation via Guided Copy-Paste Synthesis] [CVPRW 2023] [paper]
- EHNet: [Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation] [ACM MM 2022] [paper]
- PIFS: [Prototype-based Incremental Few-Shot Semantic Segmentation] [BMVC 2021] [paper]
- Towards Realistic Incremental Scenario in Class Incremental Semantic Segmentation [ArXiv 2024] [paper]
- UnCLe SAM: Unleashing SAM’s Potential for Continual Prostate MRI Segmentation [MIDL 2024] [paper]
- Cross-Domain Few-Shot Incremental Learning for Point-Cloud Recognition [WACV 2024] [paper]
- Attacks on Continual Semantic Segmentation by Perturbing Incremental Samples. [AAAI 2024] [paper]
- Federated Incremental Semantic Segmentation. [CVPR 2023] [paper] [code]
- Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions [CVPR 2023] [paper]
- Continual segment: Towards a single, unified and accessible continual segmentation model of 143 whole-body organs in ct scans. [ICCV 2023] [paper]
- Continual Nuclei Segmentation via Prototype-wise Relation Distillation and Contrastive Learning. [TMI 2023] [paper]
- Continual learning for abdominal multi-organ and tumor segmentation. [MICCAI 2023] [paper]
The evaluation of anti-forgetting ability and robustness of CSS methods.
Here are some qualitative visualizations of recent CSS methods.
Feature-based visualization is an explicit way to visualize the semantic clusters during CL steps. For example, TSNE and CAM-series are potential manners to assist the result analysis.
If this project is helpful, please consider citing it as
@ARTICLE{SurveyCSS,
author={Yuan, Bo and Zhao, Danpei},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
title={A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application},
year={2024},
volume={},
number={},
pages={1-20},
doi={10.1109/TPAMI.2024.3446949}}
- Brain-inspired Continual Semantic Segmentation [paper] | [blog]
- Inherit With Distillation and Evolve With Contrast [paper]
- Continual Panoptic Perception [paper]
☑ Method characteristics summary
☑ Performance comparison
☑ Roadmap update