A curated list of awesome in-context segmentation (ICS) papers, benchmarks, datasets, and resources. Inspired by classic awesome lists in computer vision, this repository aims to collect and organize the rapidly growing body of research in ICS.
In-context segmentation is a paradigm where a model segments arbitrary concepts—objects, parts, or personalized instances—from a single annotated visual example, without retraining. This list tracks foundational works, state-of-the-art methods, and emerging applications in this field.
The foundational models that pioneered visual in-context learning and established the core paradigms for in-context segmentation.
Painter: Images Speak in Images: A Generalist Painter for In-Context Visual Learning (arXiv 2022, CVPR 2023)
- Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, Tiejun Huang.
- Links: arXiv 2212.02499 | CVPR 2023 | Code
- Highlights: Presented a generalist model that redefines core vision task outputs as images and task prompts also as images. Uses masked image modeling on input-output image pairs. Outperforms recent generalist models on several challenging tasks and shows capabilities for out-of-domain tasks like open-category keypoint detection and object segmentation. This work laid the foundation for SegGPT.
SegGPT: Segmenting Everything In Context (arXiv 2023, ICCV 2023)
- Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, Tiejun Huang.
- Links: arXiv 2304.03284 | ICCV 2023 | Code
- Highlights: A generalist model for segmenting everything in context. Unifies various segmentation tasks into a generalist in-context learning framework by transforming them into the same image format. Performs arbitrary segmentation tasks in images or videos via in-context inference, including object instance, stuff, part, contour, and text. The training is formulated as an in-context coloring problem with random color mapping for each data sample, relying on context rather than specific colors.
SINE: A Simple Image Segmentation Framework via In-Context Examples (NeurIPS 2024)
- Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, Chunhua Shen.
- Links: NeurIPS 2024 | Code
- Highlights: The first in-context segmentation model to address task ambiguity in instance segmentation. Uses a Transformer encoder-decoder structure where the encoder provides high-quality image representations and the decoder yields multiple task-specific output masks. Solves the ambiguity among instance, semantic, and panoptic segmentation tasks. Trains only 19M parameters while freezing a frozen DINOv2 (304M) encoder. Provides a baseline for in-context segmentation models to promote the development of this field.
Fully trainable frameworks that learn segmentation rules conditioned on few-shot examples.
SegIC: Unleashing the Emergent Correspondence for In-Context Segmentation (ECCV 2024)
- Lingchen Meng, Shiyi Lan, Hengduo Li, Jose M. Alvarez, Zuxuan Wu, Yu-Gang Jiang.
- Links: arXiv 2311.14671 | ECCV 2024 | Code
- Highlights: End-to-end SegIC framework built on a single vision foundation model. Leverages emergent correspondence within VFM to capture dense relationships between target images and in-context samples. Extracts information into three types of instructions—geometric, visual, and meta—as explicit conditions for final mask prediction. Achieves SOTA on one-shot segmentation benchmarks and generalizes to video object segmentation and open-vocabulary segmentation. ECCV 2024.
Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025)
- Chaoyang Wang, Xiangtai Li, Henghui Ding, Lu Qi, Jiangning Zhang, Yunhai Tong, Chen Change Loy, Shuicheng Yan.
- Links: arXiv 2403.09616 | AAAI 2025 | Project
- Highlights: First exploration of latent diffusion models for in-context segmentation. Proposes two meta-architectures and design several output alignment and optimization strategies. Builds a new fair benchmark covering both image and video datasets. Shows LDMs can achieve comparable or even stronger results than specialist models and visual foundation models. Demonstrates that segmentation quality counts on output alignment and in-context instructions. (Note: The AAAI paper and arXiv version 2 are 2025; the initial arXiv version v1 is 2024.)
DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency (TPAMI 2025)
- Links: arXiv 2504.12080 | TPAMI 2025 | Code
- Highlights: Prompt-tuning framework that adapts SAM and SAM2 for both image and video in-context segmentation. Enhances prompt encoder features and employs dual-consistency and cycle-consistent cross-attention. Achieves SOTA by forcing spatial and temporal consistency without extensive scene fine-tuning. Builds the first video in-context segmentation benchmark (IC-VOS).
E-InMeMo: Enhanced Prompting for Visual In-Context Learning (2025)
- Links: arXiv 2504.18158 | Code
- Highlights: Lightweight strategy for enhancing visual ICL prompts. Demonstrates superior performance over existing SOTA methods on standard vision tasks, particularly for image segmentation and medical image analysis.
UNICL-SAM: Uncertainty-Driven In-Context Segmentation with Part Prototype Discovery (CVPR 2025)
- Links: CVPR 2025 | Code
- Highlights: Addresses the reliability of in-context examples by incorporating an uncertainty probabilistic module to quantify each example‘s reliability during both training and testing phases. Extracts robust representations by modeling example distribution differences and discovering part prototypes.
CDICS: Delving Into Fine-Grained Attribute for In-Context Segmentation via Compositional Prompts and Phased Decoupling (CVPR 2026)
- Zhiyu Li, Dianmo Sheng, Qi Chu, Shilong Chen, Tao Gong, Zhou Wei, Nenghai Yu.
- Links: CVPR 2026 |
- Highlights: Introduces compositional prompts derived from reference prompts, combining semantic, part and color attributes to dynamically define segmentation targets via a phased decoupling mechanism. Extends traditional in-context segmentation to support compositional prompt control, making an attempt toward real-world fine-grained segmentation. Demonstrates superior performance on compositional prompt-controlled in-context segmentation tasks. CVPR 2026.
These approaches require no training or fine-tuning, leveraging frozen features from vision foundation models to perform segmentation directly.
INSID3: Training-Free In-Context Segmentation with DINOv3 (CVPR 2026 Oral)
- Claudia Cuttano, Gabriele Trivigno, Christian Reich, Daniel Cremers, Carlo Masone, Stefan Roth.
- Links: arXiv 2603.28480 | Project | Code
- Highlights: First purely training-free ICS method that uses only frozen DINOv3 features for semantic matching and segmentation without any mask or category-level supervision. Achieves state-of-the-art on one-shot semantic, part, and personalized segmentation, outperforming prior work by +7.5% mIoU with 3× fewer parameters. Demonstrates that vision systems are moving away from “training determines everything” industrial processes. CVPR 2026 Oral. Note: The arXiv publication date is March 30, 2026.
PR-MaGIC: Prompt Refinement Via Mask Decoder Gradient Flow For In-Context Segmentation (CVPR 2026 Oral)
- Minjae Lee, Sungwoo Hur, Soojin Hwang, Won Hwa Kim.
- Links: arXiv 2604.12113 | CVPR 2026 | Project | Code
- Highlights: A training-free test-time framework that refines prompts via gradient flow derived from SAM's mask decoder. Seamlessly integrates into in-context segmentation frameworks, being theoretically grounded yet practically stabilized through a simple top-1 selection strategy. Consistently improves segmentation quality across various benchmarks, effectively mitigating inadequate prompts without requiring additional training or architectural modifications. CVPR 2026 Oral.
Matcher: Segment Anything in Context with Vision Foundation Models (ICLR 2024, IJCV 2025)
- Links: ICLR 2024 | IJCV 2025 | Code
- Highlights: Training-free perception paradigm that utilizes off-the-shelf vision foundation models to address various segmentation tasks. Matcher can segment anything by using in-context examples without training. Designs three effective components: bidirectional matching strategy, robust prompt sampler, and instance-level matching to filter false-positive mask fragments. Demonstrates impressive generalization without any additional training. (Also available on arXiv v4 as “Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching”)
Stable Diffusion Models are Secretly Good at Visual In-Context Learning (ICCV 2025)
- Links: arXiv 2508.09949 | ICCV 2025
- Highlights: Off-the-shelf Stable Diffusion models can be repurposed for visual in-context learning. Formulates an in-place attention recomputation within the self-attention layers that explicitly incorporates context between query and example prompts. Requires no additional fine-tuning.
SkySense-VITA: Towards Universal In-context Segmentation of Multi-modal Remote Sensing Imagery (CVPR 2026)
- Kang Wu, Lei Yu, Junwei Luo, Bo Dang, Junjian Zhang, Xiangyuan Cai, Hongwei Hu, Jingdong Chen, Yansheng Li.
- Links: CVPR 2026 | Project
- Highlights: A unified in-context segmentation model that synergistically processes both optical and SAR imagery using visual, textual, or fused prompts. Proposes VITA-Former and VITA-Decoder to decouple multi-modal prompt fusion and prediction process. Trained with a progressive two-stage strategy including Image-Level Alignment Pretraining and Pixel-Level In-context Pretraining using Semantic Granularity Annealing (SGA). Establishes new SOTA on 18 datasets with an average performance lead of over 10% mIoU. Introduces the large-scale, multi-modal Sky-VT-300k dataset. CVPR 2026.
A significant application area where segmenting new anatomical structures from few examples is clinically valuable. This field has seen particularly strong growth.
SegMIC: A Universal Model for Medical Image Segmentation through In-Context Learning (Pattern Recognition 2025)
- Jianwei Zhao, Fan Yang, Xin Li, Zhicheng Jiao, et al.
- Links: Pattern Recognition | Code
- Highlights: First framework to leverage ICL for universal medical segmentation through masked image modeling. Tailored Jointly Masked Image Modeling (JMIM) strategy enabling ICL. Segments any medical image solely using a reference pair. Superior Dice (0.919) and IoU (0.870) on in-domain tasks. Introduces UniMedDB—49k images / 82k annotations across 14+ modalities and 56 datasets.
MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance (ICCV 2025)
- Hallee E. Wong, Jose Javier Gonzalez Ortiz, John Guttag, Adrian V. Dalca.
- Links: ICCV 2025 | arXiv 2412.15058 | Project | Code
- Highlights: Enables users to rapidly segment new biomedical imaging datasets without access to existing labeled data. Combines user interaction with historical segmentation results as context. Introduces CrossBlock mechanism for target-context fusion. Reduces user interactions by 36% (clicks) and 25% (scribbles) to reach same accuracy. Evaluated on 79 datasets covering MRI, CT, ultrasound, and more.
PatchICL: Scaling In-Context Segmentation with Hierarchical Supervision (arXiv 2026)
- Tidiane Camaret Ndir, Marco Reisert, Robin T. Schirrmeister.
- Links: arXiv 2604.12752 | Code
- Highlights: Hierarchical framework combining selective image patching with multi-level supervision. Reduces compute by 44% at 512×512 resolution while maintaining competitive in-domain CT segmentation accuracy. Particularly strong on modalities dominated by localized pathology such as OCT and dermoscopy.
Extending ICS capabilities from static images to temporal sequences.
DC-SAM (see above under End-to-End Methods) – Adapts SAM and SAM2 to ICS via prompt-tuning and constructs the first video ICS benchmark (IC-VOS) via mask-tube training.
Temporal: Time-Contrastive Pretraining for In-Context Image and Video Segmentation (MICCAI 2025)
- Links: arXiv 2506.17837 | MICCAI 2025
- Highlights: Reformulates ICL as a Video Object Segmentation (VOS) problem. Introduces a time-contrastive self-supervised objective for prompt retrieval. Supports variable number of context images while preserving full resolution. On MICCAI FLARE 2022, achieves 90.95% Dice for image segmentation (+10.64%) and 92.45% Dice for video segmentation (+14.88%).
Specialized applications focused on text region detection and manipulation.
ConText: Driving In-context Learning for Text Removal and Segmentation (ICML 2025)
- Fei Zhang, Pei Zhang, Baosong Yang, Fei Huang, Yanfeng Wang, Ya Zhang.
- Links: ICML 2025 | arXiv 2506.03799 | Code
- Highlights: First V-ICL framework for OCR tasks—text removal and text segmentation. Proposes task-chaining compositor and context-aware aggregation. Achieves SOTA on text removal (+4.50 PSNR) and segmentation (+3.34% fgIoU).
Research dedicated to understanding and improving example selection for in-context segmentation.
Rethinking and Improving Visual Prompt Selection for In-Context Learning Segmentation (ECCV 2024)
- Wei Suo, Lanqing Lai, Mengyang Sun, Hanwang Zhang, Peng Wang, Yanning Zhang.
- Links: ECCV 2024 | Code
- Highlights: Demonstrates that ICL-based segmentation models are sensitive to different contexts. Shows diversity of contextual prompts plays a crucial role in guiding segmentation. Proposes stepwise context search method that constructs a small yet rich candidate pool and adaptively searches well-matched contexts. Reduces annotation cost by compacting the search space.
Retriever Encoder Selection Matters for In-Context Learning-based Medical Segmentation (AAAI 2026)
- Fan Wang, Zhongyi Han, Yongshun Gong, Yilong Yin.
- Links: AAAI 2026 | Code
- Highlights: Demonstrates that retriever encoder selection significantly impacts ICL-based medical segmentation performance. Extends the understanding of visual prompt selection to medical domains.
Emerging paradigm that reduces annotation costs by using weak prompts (bounding boxes, points) instead of dense masks.
WS-ICL: Efficient Universal Models for Medical Image Segmentation via Weakly Supervised In-Context Learning (arXiv 2025)
- Jiesi Hu, Yanwu Yang, Zhiyu Ye, Jinyan Zhou, Jianfeng Cao, Hanyang Peng, Ting Ma
- Links: arXiv 2510.05899 | Code
- Highlights: Proposes Weakly Supervised In-Context Learning (WS-ICL), a new ICL paradigm that leverages weak prompts (e.g., bounding boxes or points) instead of dense labels for context. Drastically cuts annotation effort by removing the need for fine-grained segmentation masks and avoiding repeated prompting for every image. (arXiv submission: October 8, 2025)
| Benchmark | Description | Scale | Key Papers |
|---|---|---|---|
| COCO-20ⁱ | Few-shot segmentation on 20 classes from COCO | 20 categories | Classic FSS benchmark |
| PASCAL-5ⁱ | Few-shot segmentation on 20 classes from PASCAL VOC | 20 categories | Classic FSS benchmark |
| FSS-1000 | 1000-class few-shot segmentation dataset | 1000 classes | Standard benchmark |
| IC-VOS | First video ICS benchmark | Video segmentation | DC-SAM |
| UniMedDB | Universal medical segmentation benchmark | 49k images, 82k annotations, 14+ modalities, 56 datasets | SegMIC |
| LDIS Benchmark | Image + video ICS benchmark (from LDM work) | Multi-domain | LDM paper (arXiv 2403.09616) |
| Repository | Description | Link |
|---|---|---|
| Painter / SegGPT | SegGPT generalist segmentation framework | https://github.com/baaivision/Painter |
| INSID3 | Training-free ICS with DINOv3 (CVPR 2026 Oral) | https://github.com/visinf/INSID3 |
| SegIC | End-to-end SegIC framework (ECCV 2024) | https://github.com/MengLcool/SEGIC |
| SINE | Simple ICS framework (NeurIPS 2024) | https://github.com/aim-uofa/sine |
| ConText | Text removal & segmentation (ICML 2025) | https://github.com/Ferenas/ConText |
| PatchICL | Hierarchical medical image segmentation (arXiv 2026) | https://github.com/tidiane-camaret/ic_segmentation |
| SegMIC | Universal medical segmentation (Pattern Recognition 2025) | https://github.com/JWZhao-uestc/SegMIC |
| MultiverSeg | Interactive segmentation (ICCV 2025) | https://github.com/halleewong/MultiverSeg |
| SCS | Stepwise Context Search for prompt selection (ECCV 2024) | https://github.com/LanqingL/SCS |