MedVision: Benchmarking Quantitative Medical Image Analysis
🌟 Highlights
- Research gap. Modern VLMs cannot reliably produce precise quantitative measurements from medical images.
- Dataset. MedVision
v1.0.0— a large-scale, multi-anatomy, multi-modality dataset for quantitative medical image analysis (22 public datasets, 29.0K 3D images, 11.2M annotated 2D slices, 24.3M single-instance annotations, and 45.3M multi-instance annotations); see the Dataset Explorer for later releases. - Benchmark. The first comprehensive evaluation of contemporary VLMs on detection, tumor/lesion (T/L) size estimation, and angle/distance (A/D) measurement in medical images.
- Model. MedVision-V0, a 7B model trained on MedVision via supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT); it significantly outperforms all evaluated VLMs across all three tasks — a strong, open baseline.
- Open release. Data, model, and code (training and evaluation) are all publicly available.
🎯 Problem & Tasks
Clinical decisions rely on quantitative assessment — measuring a tumor to stage disease, a joint angle to plan surgery, an anatomical distance to track development. We therefore target a concrete model ability: given a medical image, produce precise numeric measurements in real-world physical units (millimeters and degrees, not pixels).
MedVision evaluates this ability across three quantitative tasks:
1️⃣ Detection
Localize healthy anatomical structures and abnormalities with bounding boxes.
2️⃣ Tumor/Lesion Size
Estimate the longest diameter (major axis) and its perpendicular diameter (minor axis) of a tumor/lesion, reported in millimeters.
3️⃣ Angle/Distance
Measure angles (degrees) and distances (mm) from anatomical landmarks.
📈 Leaderboard
Last updated: Sep 3, 2026
MedVision-V0 outperforms all 17 evaluated off-the-shelf VLMs across all three quantitative task families. Each task below leads with the full leaderboard (🥇/🥈/🥉 mark the best three per column; click any metric to rank the table by it, best first; underlined metrics belong to a sub-task whose success rate fell below 50%, meaning they are computed on a minority of samples and should be read with care), followed by an interactive viewer of real predictions — the complete prompt, the model’s chain-of-thought response, and the error metrics, beside the image with a ground-truth-vs-prediction overlay.
1️⃣ Detection
Table 2: Detection performance (%), grouped into anatomy and tumor/lesion targets. R: recall; P: precision; F1: F1 score; IoU: intersection over union; SR: success rate.
| Model | Anatomy (18 regions, 13.4K samples) | Tumor/Lesion (8 regions, 8.5K samples) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R ↑ | P ↑ | F1 ↑ | IoU ↑ | SR ↑ | IoU>0.5 ↑ | R ↑ | P ↑ | F1 ↑ | IoU ↑ | SR ↑ | IoU>0.5 ↑ | |
| MedVision-V0 (7B) | 81.3🥇 | 80.4🥇 | 79.1🥇 | 72.0🥇 | 100 | 80.1🥇 | 52.4 | 50.5🥇 | 46.9🥇 | 38.2🥇 | 100 | 40.7🥇 |
| Gemma-4 (31B) | 34.8 | 20.7🥈 | 22.6🥈 | 16.7🥈 | 98.5 | 13.5🥈 | 43.6 | 14.6🥈 | 18.1🥈 | 12.7🥈 | 99.5 | 8.4🥈 |
| Lingshu (32B) | 37.4 | 20.2🥉 | 20.2🥉 | 13.7🥉 | 100 | 6.7 | 40.2 | 6.0 | 8.6 | 5.1 | 100 | 0.2 |
| GLM-4.6V (106B) | 46.5 | 16.4 | 20.2🥉 | 13.5 | 99.9 | 6.0 | 52.4 | 5.1 | 7.7 | 4.6 | 99.8 | 0.5 |
| Qwen3-VL-Thinking (32B) | 35.8 | 17.4 | 19.5 | 13.4 | 96.5 | 7.5🥉 | 33.9 | 6.6 | 8.2 | 5.2 | 98.0 | 1.3 |
| MedGemma (27B) | 57.2 | 15.7 | 19.1 | 12.9 | 98.6 | 6.6 | 54.1 | 4.6 | 7.5 | 4.3 | 97.0 | 0.1 |
| MedGemma (4B) | 68.7🥉 | 14.7 | 18.6 | 12.4 | 98.9 | 6.4 | 77.7🥇 | 4.4 | 7.5 | 4.2 | 99.3 | 0.0 |
| Qwen2.5-VL (32B) | 44.8 | 14.9 | 18.4 | 12.5 | 100 | 6.3 | 38.5 | 5.7 | 7.7 | 4.7 | 100 | 0.6 |
| LLaVA-OneVision (72B) | 34.9 | 19.0 | 18.1 | 11.8 | 100 | 2.4 | 34.1 | 6.3 | 8.4 | 5.0 | 100 | 0.4 |
| MiniMax-M3 (428B, int4) | 32.5 | 15.4 | 17.5 | 11.8 | 99.9 | 5.6 | 35.7 | 7.8🥉 | 10.2🥉 | 6.5🥉 | 100 | 1.6🥉 |
| InternVL3 (38B) | 31.1 | 17.0 | 17.2 | 11.5 | 100 | 5.3 | 29.5 | 6.6 | 7.9 | 4.9 | 100 | 0.8 |
| Qwen2.5-VL (7B) | 69.7🥈 | 12.2 | 16.7 | 11.3 | 99.4 | 5.6 | 77.4🥈 | 3.8 | 6.5 | 3.6 | 99.6 | 0.0 |
| HealthGPT (14B) | 29.6 | 20.7🥈 | 16.0 | 10.2 | 98.7 | 1.8 | 29.2 | 7.0 | 8.4 | 5.1 | 97.8 | 0.5 |
| GLM-4.6V-Flash (9B) | 22.6 | 17.6 | 15.2 | 9.8 | 100 | 2.5 | 26.2 | 6.1 | 7.5 | 4.7 | 100 | 1.0 |
| HuatuoGPT-Vision (34B) | 25.6 | 17.7 | 15.1 | 9.8 | 99.5 | 2.7 | 21.6 | 5.0 | 6.4 | 3.8 | 99.0 | 0.3 |
| Gemma-3 (27B) | 37.1 | 12.4 | 14.9 | 10.1 | 100 | 4.6 | 34.3 | 4.3 | 6.1 | 3.6 | 100 | 0.3 |
| MedDr (40B) | 53.6 | 11.1 | 14.6 | 9.7 | 96.8 | 4.2 | 63.4🥉 | 3.7 | 6.2 | 3.5 | 98.7 | 0.1 |
| Llama-3.2-Vision (11B) | 52.2 | 12.7 | 14.3 | 9.4 | 89.4 | 3.2 | 49.0 | 3.6 | 5.9 | 3.4 | 85.4 | 0.1 |
2️⃣ Tumor/Lesion Size
Table 3: Tumor/lesion size estimation (2K samples). MAE in millimeters; MRE, SR, and MRE<0.1 in %.
| Model | MAE ↓ | MRE ↓ | SR ↑ | MRE<0.1 ↑ |
|---|---|---|---|---|
| MedVision-V0 (7B) | 10.5🥇 | 26.0🥇 | 100.0 | 23.5🥇 |
| Gemma-4 (31B) | 21.7🥈 | 72.6🥈 | 98.9 | 16.8🥈 |
| GLM-4.6V (106B) | 31.4🥉 | 107.3🥉 | 92.2 | 4.1 |
| MiniMax-M3 (428B, int4) | 32.7 | 108.1 | 95.8 | 5.6 |
| GLM-4.6V-Flash (9B) | 34.4 | 109.2 | 95.2 | 3.0 |
| Lingshu (32B) | 35.7 | 118.6 | 99.5 | 4.5 |
| HealthGPT (14B) | 51.8 | 176.0 | 100.0 | 2.2 |
| Qwen3-VL-Thinking (32B) | 53.2 | 140.6 | 94.0 | 6.2🥉 |
| MedDr (40B) | 78.0 | 240.7 | 87.3 | 1.9 |
| Llama-3.2-Vision (11B) | 101.2 | 329.4 | 98.3 | 0.7 |
| Gemma-3 (27B) | 225.8 | 611.4 | 99.0 | 0.5 |
| MedGemma (27B) | 523.4 | 1905.0 | 57.0 | 0.5 |
| HuatuoGPT-Vision (34B) | 1018.6 | 2847.2 | 98.3 | 1.5 |
| LLaVA-OneVision (72B) | 1089.1 | 3368.0 | 100.0 | 1.2 |
| Qwen2.5-VL (32B) | 2054.6 | 6989.2 | 99.9 | 1.3 |
| Qwen2.5-VL (7B) | 2897.2 | 7680.9 | 95.4 | 0.7 |
| InternVL3 (38B) | 8606.8 | 25285.1 | 100.0 | 0.2 |
| MedGemma (4B) | 1109532.7 | 3733526.8 | 90.9 | 0.1 |
3️⃣ Angle/Distance
Table 4: Angle/distance measurement across all 17 off-the-shelf VLMs and MedVision-V0, for each sub-task. MAE in millimeters (distance) and degrees (angle); MRE, SR, and MRE<0.1 in %.
| Model | Ceph-Bio-400 + FeTA24 · Distance (1,100 samples) | |||
|---|---|---|---|---|
| MAE ↓ | MRE ↓ | SR ↑ | MRE<0.1 ↑ | |
| MedVision-V0 (7B) | 3.6🥇 | 6.4🥇 | 100 | 81.4🥇 |
| MiniMax-M3 (428B, int4) | 18.0🥈 | 30.4🥈 | 97.5 | 22.0🥈 |
| GLM-4.6V (106B) | 22.7🥉 | 41.5 | 88.4 | 14.2 |
| HealthGPT (14B) | 23.1 | 42.4 | 97.6 | 18.0 |
| Gemma-4 (31B) | 23.8 | 37.9🥉 | 99.8 | 10.8 |
| GLM-4.6V-Flash (9B) | 30.3 | 56.0 | 99.7 | 12.5 |
| MedDr (40B) | 106.3 | 197.6 | 91.2 | 8.5 |
| Qwen3-VL-Thinking (32B) | 119.0 | 197.8 | 98.4 | 18.6 |
| Lingshu (32B) | 198.8 | 247.7 | 100 | 21.4🥉 |
| LLaVA-OneVision (72B) | 2541.7 | 4557.2 | 100 | 6.1 |
| HuatuoGPT-Vision (34B) | 3921.6 | 6251.5 | 98.2 | 5.4 |
| Llama-3.2-Vision (11B) | 4666.2 | 7065.8 | 98.6 | 2.1 |
| Gemma-3 (27B) | 5054.2 | 6606.9 | 99.0 | 13.3 |
| MedGemma (27B) | 6239.0 | 8616.4 | 46.2 | 6.7 |
| MedGemma (4B) | 15222.0 | 24918.0 | 95.1 | 0.1 |
| Qwen2.5-VL (32B) | 56045.7 | 71734.7 | 99.9 | 6.3 |
| InternVL3 (38B) | 62066.9 | 62191.8 | 99.8 | 8.0 |
| Qwen2.5-VL (7B) | 63603.8 | 96542.0 | 98.3 | 0.5 |
| Model | Ceph-Bio-400 · Angle (960 samples) | |||
|---|---|---|---|---|
| MAE ↓ | MRE ↓ | SR ↑ | MRE<0.1 ↑ | |
| MedVision-V0 (7B) | 4.7🥇 | 52.1🥇 | 99.9 | 52.0🥇 |
| MiniMax-M3 (428B, int4) | 17.9🥈 | 477.1 | 48.6 | 8.5 |
| GLM-4.6V (106B) | 19.7🥉 | 322.7 | 88.8 | 20.9🥈 |
| Gemma-4 (31B) | 23.9 | 428.0 | 88.5 | 11.7 |
| HealthGPT (14B) | 25.9 | 463.2 | 99.3 | 16.7 |
| InternVL3 (38B) | 30.3 | 616.8 | 100 | 20.4🥉 |
| Qwen3-VL-Thinking (32B) | 32.4 | 378.2 | 69.6 | 3.6 |
| LLaVA-OneVision (72B) | 33.6 | 473.5 | 100 | 2.9 |
| Llama-3.2-Vision (11B) | 34.2 | 287.1🥉 | 100 | 3.2 |
| Lingshu (32B) | 35.0 | 512.5 | 100 | 6.3 |
| GLM-4.6V-Flash (9B) | 35.0 | 531.7 | 99.1 | 8.1 |
| MedGemma (4B) | 35.8 | 298.8 | 96.7 | 6.0 |
| Gemma-3 (27B) | 36.3 | 702.2 | 99.9 | 6.7 |
| Qwen2.5-VL (32B) | 41.2 | 258.3🥈 | 99.2 | 1.0 |
| MedDr (40B) | 45.7 | 591.5 | 94.5 | 5.3 |
| MedGemma (27B) | 46.4 | 1024.8 | 93.0 | 6.8 |
| Qwen2.5-VL (7B) | 48.0 | 724.9 | 97.6 | 2.0 |
| HuatuoGPT-Vision (34B) | 7070.0 | 9032.7 | 95.4 | 3.7 |
🔬 Pilot Study: Frontier API Models
Running API-served frontier VLMs across the entire benchmark is prohibitively costly — the test set spans multiple tasks, each with a large number of samples. We therefore conduct a pilot study that evaluates frontier API models on a small testing subset per task (Tumor/Lesion Size for now), reusing the exact prompts and samples from the full benchmark. This pilot study benchmarks how capable today’s frontier models are at quantitative medical image measurement, facilitating the design of agentic AI systems for biomedical applications.
Table 5: Pilot study on tumor/lesion size estimation using a small testing subset (750 samples). MAE in millimeters; MRE, SR, and MRE<0.1 in %. Cost is the total API evaluation spend in USD.
| Model | MAE ↓ | MRE ↓ | SR ↑ | MRE<0.1 ↑ | Cost |
|---|---|---|---|---|---|
| MedVision-V0 (7B) | 9.6🥇 | 26.9🥇 | 100.0 | 24.1🥇 | $0 |
| Claude-Fable-5 | 12.5 | 46.5 | 100.0 | 23.7 | $63.9 |
| Gemini-3.1-Pro | 14.9 | 48.8 | 79.2 † | 18.1 | $101.3 |
| GPT-5.5-Pro ‡490 of 750 samples | 13.7 | 52.4 | 100.0 | 23.7 | $959 |
† All API models were run with the same 16,000-token output budget. Gemini-3.1-Pro's thinking (non-disableable, default level) shares that budget with the answer; ~20% of responses exhaust it on reasoning and return empty/truncated output, lowering SR.
‡ GPT-5.5-Pro was evaluated on 490 of the 750 samples: the run stopped after 6 of the 10 dataset tasks when its API spending budget was exhausted (a cost limit, not the token limit). Its metrics are computed over those 490 samples and are not directly comparable to the full-subset rows. Liver tumor and enhancing brain tumor were not evaluated, and kidney tumor covers KiPA22 only (100 of 199 samples); Table 6 and the radar below leave all three targets empty as incomplete.
Table 6: Per-target MRE and SR (%) of the pilot-study models; n is the number of samples per target in the 750-sample subset; the miscellaneous tumor/lesion group (52 samples) is omitted, as in the radar below.
| Model | kidney tumorCT (A) · n=199 | liver tumorCT (A) · n=118 | brain tumorMR (A) · n=96 | brain resection cavityMR (A) · n=91 | non-enhancing brain tumorMR (A) · n=88 | metastatic lymph nodeMR (A) · n=63 | enhancing brain tumorMR (A) · n=43 | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MRE ↓ | SR ↑ | MRE ↓ | SR ↑ | MRE ↓ | SR ↑ | MRE ↓ | SR ↑ | MRE ↓ | SR ↑ | MRE ↓ | SR ↑ | MRE ↓ | SR ↑ | |
| MedVision-V0 (7B) | 35.3 | 100.0 | 31.4 | 100.0 | 25.9 | 100.0 | 21.5 | 100.0 | 15.2 | 100.0 | 28.3 | 100.0 | 11.7 | 100.0 |
| Claude-Fable-5 | 55.5 | 100.0 | 64.4 | 100.0 | 59.5 | 100.0 | 14.1 | 100.0 | 40.3 | 100.0 | 42.2 | 100.0 | 20.8 | 100.0 |
| Gemini-3.1-Pro † | 64.9 | 73.9 | 63.2 | 70.3 | 42.8 | 83.3 | 22.5 | 82.4 | 50.8 | 86.4 | 37.6 | 84.1 | 14.7 | 90.7 |
| GPT-5.5-Pro | — | — | — | — | 76.4 | 100.0 | 24.4 | 100.0 | 55.8 | 100.0 | 97.4 | 100.0 | — | — |
† SR below 100% for the reason given under Table 5.
— Not evaluated or incomplete: GPT-5.5-Pro's run stopped after 6 of the 10 dataset tasks (490 of 750 samples) when the API budget ran out, skipping the KiTS23 and MSD tasks. Liver tumor and enhancing brain tumor were never evaluated, and its kidney-tumor result (KiPA22 only, 100 of 199 samples) is omitted as incomplete.
🤝 Call for Models
MedVision is an open, growing leaderboard — we keep adding models as the field moves. If there is a vision language model you would like to see evaluated on the benchmark, please 🧑🏻💻open a GitHub issue with the model name and a link to its weights or API, and we will consider it for a future evaluation round.
Pull requests are equally welcome. Our 📚 New Models Guide walks through everything needed to plug a new VLM into the benchmark — a PR contributing complete, working inference code for a model is the fastest route onto the leaderboard.