Medical Image Segmentation
Segmenting organs and abnormalities in medical images.
Medical image segmentation delineates anatomical structures and pathologies in 3D medical scans — critical for treatment planning, tumor measurement, and surgical guidance. nnU-Net remains the dominant auto-configuring baseline, while SAM-Med2D and Med-SA bring foundation model capabilities to medical segmentation.
History
U-Net (Ronneberger et al.) establishes the encoder-decoder architecture for biomedical segmentation
V-Net extends U-Net to 3D volumetric segmentation for CT/MRI
nnU-Net auto-configures preprocessing, architecture, and training for any medical segmentation task
Attention U-Net adds attention gates for focusing on relevant anatomical regions
UNETR applies vision transformers to 3D medical image segmentation
nnU-Net wins multiple Medical Segmentation Decathlon challenges
SAM (Segment Anything) adapted for medical images — SAM-Med2D and MedSAM
STU-Net scales 3D medical segmentation to 1.4B parameters
TotalSegmentator provides 117-class whole-body CT segmentation
Foundation segmentation models enable zero-shot organ and tumor delineation
How Medical Image Segmentation Works
Image Acquisition
3D medical volumes (CT, MRI, PET) or 2D images (X-ray, ultrasound, pathology slides) are acquired and standardized.
Preprocessing
nnU-Net-style preprocessing: intensity normalization, resampling to target spacing, and cropping to the region of interest.
Encoder Path
A CNN (ResNet blocks) or ViT encoder compresses the image into a hierarchy of feature maps at decreasing resolutions.
Decoder Path
The decoder upsamples features back to input resolution, with skip connections from the encoder preserving fine-grained spatial detail.
Segmentation Output
A pixel/voxel-wise classification produces a segmentation mask, with each pixel labeled as background or one of the target structures.
Current Landscape
Medical image segmentation in 2025 is dominated by nnU-Net — the auto-configuring framework that consistently achieves top results across diverse tasks without manual tuning. Foundation models (MedSAM, SAM-Med2D) are enabling interactive and few-shot segmentation, reducing the annotation burden. TotalSegmentator demonstrates that comprehensive whole-body segmentation is now production-ready. The field is mature for common structures (organs, major vessels) but still challenging for small lesions, rare pathologies, and domains with limited training data.
Key Challenges
Annotation cost — expert radiologist/pathologist annotation of 3D volumes is extremely expensive ($50-200 per scan)
Class imbalance — lesions and tumors are tiny relative to background, requiring focal loss or oversampling
3D memory constraints — processing full-resolution 3D volumes requires patch-based training with careful aggregation
Domain gaps — different scanners, protocols, and patient populations cause segmentation failures on new data
Evaluation metrics — Dice score alone is insufficient; surface distance and volumetric accuracy matter for clinical use
Quick Recommendations
Any new segmentation task
nnU-Net
Auto-configuring, consistently wins or ties SOTA on new datasets with zero manual tuning
Interactive segmentation
MedSAM / SAM-Med2D
Click-based prompting for quick annotation and novel structure segmentation
Whole-body CT
TotalSegmentator
117 anatomical structures in one pass, production-ready
Research with transformers
SwinUNETR / UNETR
Best transformer-based architectures for 3D medical volumes
What's Next
The frontier is universal medical segmentation — models that can segment any anatomical structure or pathology from any imaging modality with minimal or zero task-specific training. Expect foundation models pretrained on massive multi-institutional datasets, federated learning to access data across hospitals, and real-time segmentation for surgical guidance.
Benchmarks & SOTA
Synapse Multi-Organ CT
Synapse Multi-Organ Abdominal CT Segmentation
18 abdominal CT scans with 13 organ labels. Most widely used multi-organ CT segmentation benchmark. MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge.
State of the Art
SegMamba
Xing et al.
86.45
mean-dsc
ACDC
Automated Cardiac Diagnosis Challenge
150 cardiac MRI cases segmenting right ventricle, myocardium, and left ventricle. MICCAI 2017 challenge.
State of the Art
MedNeXt-L
German Cancer Research Center (DKFZ)
92.65
mean-dsc
BTCV
Beyond The Cranial Vault Multi-Organ CT Segmentation
30 abdominal CT scans with 13 organ annotations. MICCAI 2015 challenge hosted on Synapse platform.
State of the Art
STU-Net-H
Ziyan Huang et al.
85.38
mean-dsc
BraTS 2023
Brain Tumor Segmentation Challenge 2023
1,251 multi-institutional MRI cases for brain tumor segmentation. Segments whole tumor (WT), tumor core (TC), and enhancing tumor (ET). Annual MICCAI challenge.
State of the Art
MedNeXt-L
German Cancer Research Center (DKFZ)
0.896
mean-dice-wt-tc-et
Related Tasks
Get notified when these results update
New models drop weekly. We track them so you don't have to.
Something wrong or missing?
Help keep Medical Image Segmentation benchmarks accurate. Report outdated results, missing benchmarks, or errors.