MedVision: Dataset and Benchmark for Quantitative Medical Image Analysis

🌟 Highlights

• Research Gap: modern VLMs are not reliably able to produce precise quantitative measurements from medical images.
• Dataset: MedVision, a large-scale, multi-modality dataset for quantitative medical image analysis, covering 22 public datasets and 30.8M images with structured measurement annotations.
• Benchmark: the first comprehensive evaluation of contemporary VLMs on detection, tumor/lesion size estimation, and angle/distance measurement.
• Supervised Finetuning (SFT): SFT can improve the performance of VLMs on quantitative medical image analysis tasks.
• Code and Models: data, model checkpoints, and code (training and evaluation) are available

📀 Dataset

MedVision includes 22 public datasets and 30.8M image-annotation pairs. The dataset is available at here. Details are as follows:

Table 1: The MedVision dataset consists of public medical images and quantitative annotations from this study. MRI: Magnetic Resonance Imaging; CT: Computed Tomography; PET: positron emission tomography; US: Ultrasound; b-box: bounding box; T/L: tumor/lesion size; A/D: angle/distance; HF: HuggingFace; GC: Grand-Challenge; ^† redistributed.

Q: How to use the dataset?

• Quick start
• Prepare dataset for MedVision benchmarking and finetuning: here

1️⃣ Detection

Q: Can VLMs localize healthy anatomical structures and abnormalities from medical images?

• Pretrained VLMs show limited ability in medical image detection tasks.
• With SFT, VLMs achieve dramatic improvement in the detection of both healthy anatomical structures and tumors/lesions.
• Small object and tumors/lesions detection is still challenging.
• SFT improves models’ generalizability (details in paper)

Group-level detection performance

Table 2: VLM performance on detection tasks. Targets are grouped into health anatomy and tumor/lesion detection tasks. Mean metrics weighted by sample sizes are reported in %. R: recall; P: precision; F1: F1 score; IoU: intersection over union; SR: success rate.

Label-level detection performance

This figure shows the detection metrics per label and box-to-image ratio (that indicates relative target size) for each models.

2️⃣ Tumor/Lesion Size Estimation

Q: Can VLMs estimate the size of tumors/lesions from medical images?

• Mean relative error (MRE) for SFT models are ∼ 30%, while it is 50~120% for off-the-shelf VLMs
• SFT improves models’ generalizability (details in paper)

VLMs are asked to estimate the longest diameter and its perpendicular diameter of tumors/lesions. An example of the quantitative annotations from this work is shown below.

Figure 2: Tumor/lesion size annotation. An ellipse is fitted to the tumor/lesion mask and 4 landmarks are recorded.

Table 3: VLM performance on tumor/lesion size estimation tasks. Mean relative error (MRE), success rate (SR), and MRE_<k are reported in %, while mean absolute error (MAE) is in millimeters.

3️⃣ Angle/Distance Measurement

Q: Can VLMs measure angles or distances from medical image?

• Off-the-shelf VLMs fail to measure angles and distances accurately.
• SFT can significantly improve performance.
• Small-angle measurements remain challenging.

VLMs are prompted with task description and definition of angle or distance. Examples of landmarks in the Ceph-Bio-400 and FeTA24 datasets are shown below.

Figure 3: Landmarks in the Ceph-Bio-400 (top-left) and FeTA24 datasets. Ground truth angle and distance measurements are calculated from these landmarks.

Table 4: VLM performance on angle and distance measurement tasks. Mean relative errors (MRE), MRE_<0.1, and success rate (SR) are reported in %. Mean absolute errors (MAE) are given in millimeters (distance) and degrees (angle).