MobileCLIP2 S0 vs S2: a reproducible CPU test

MOBILECLIP2 · SITE MEASUREMENTS

On 24 synthetic images, S0 and S2 classified the same 21 correctly. S0 had lower warm latency on this machine. This small diagnostic set did not reveal an accuracy benefit from S2.

Measurement date: September 12, 2026 · one machine · six classes · no tuning on these examples

Measured by FastVLM.net

Test scope and limits

Apple M5 Pro host with 48 GiB unified memory, CPU only, float32, 4 PyTorch threads, batch size 1. Each model ran in its own process: 3 warmups, then 5 fixed-order repeats of all 24 images. The six candidate text embeddings were computed once. Timing includes image file decode, preprocessing, image encoding and scoring; excludes download, model loading and text encoding.

Peak RSS includes Python, library imports, checkpoint loading and initialization. Parameter-tensor bytes exclude buffers and runtime overhead. Neither figure is a minimum RAM requirement or GPU-memory measurement.

Six synthetic template families with four variants each, using six fixed English candidate labels. All three errors were small-text documents classified as interfaces. A different candidate-label set, photographs or real documents can change the result. Equal performance here does not mean equal general accuracy.

Apple-published results: a separate benchmark

Apple reports ImageNet zero-shot top-1 of 71.5% for S0 and 77.2% for S2. Those values are from a different dataset and measurement procedure; do not combine them with this 24-image result or compare our CPU latency to Apple’s device benchmarks. Apple S0 · Apple S2

How to choose

S0 is a reasonable starting point when these document-category distinctions and lower CPU latency fit the task. Choose S2 after testing whether it improves your own labels and images enough to justify the extra resources. Neither model generates a written answer.

Reproduce and inspect

Download fixtures, results and scripts ↓ · Classification setup guide →Run each variant in a separate process

python -m pip install "open_clip_torch==3.3.0" "timm==1.0.29" torch pillow
python scripts/evaluate-mobileclip.py --model S0
python scripts/evaluate-mobileclip.py --model S2

S0 JSON · S2 JSON · Manifest · Provenance

Inspect every case

Category Only missing-answer / misclassified cases 24 cases shown

receipt-1

Expected class: a receipt

S0 Prediction: a receipt · 108.7 ms

S2 Prediction: a receipt · 270.8 ms

receipt-2

Expected class: a receipt

S0 Prediction: a receipt · 110.7 ms

S2 Prediction: a receipt · 267.6 ms

receipt-3

Expected class: a receipt

S0 Prediction: a receipt · 110.7 ms

S2 Prediction: a receipt · 263.7 ms

receipt-4

Expected class: a receipt

S0 Prediction: a receipt · 110.4 ms

S2 Prediction: a receipt · 268.9 ms

chart-1

Expected class: a bar chart

S0 Prediction: a bar chart · 109.9 ms

S2 Prediction: a bar chart · 264.2 ms

chart-2

Expected class: a bar chart

S0 Prediction: a bar chart · 114.8 ms

S2 Prediction: a bar chart · 285.3 ms

chart-3

Expected class: a bar chart

S0 Prediction: a bar chart · 112.7 ms

S2 Prediction: a bar chart · 280.6 ms

chart-4

Expected class: a bar chart

S0 Prediction: a bar chart · 109.5 ms

S2 Prediction: a bar chart · 268.2 ms

interface-1

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 110.3 ms

S2 Prediction: a user interface screenshot · 271.0 ms

interface-2

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 110.5 ms

S2 Prediction: a user interface screenshot · 271.5 ms

interface-3

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 110.6 ms

S2 Prediction: a user interface screenshot · 284.5 ms

interface-4

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 111.0 ms

S2 Prediction: a user interface screenshot · 267.8 ms

small-text-1

Expected class: an English text document

S0 Prediction: an English text document · 111.0 ms

S2 Prediction: an English text document · 270.9 ms

small-text-2

Expected class: an English text document

S0 Prediction: a user interface screenshot · 108.1 ms

S2 Prediction: a user interface screenshot · 284.5 ms

small-text-3

Expected class: an English text document

S0 Prediction: a user interface screenshot · 109.0 ms

S2 Prediction: a user interface screenshot · 282.1 ms

small-text-4

Expected class: an English text document

S0 Prediction: a user interface screenshot · 109.4 ms

S2 Prediction: a user interface screenshot · 277.1 ms

chinese-1

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 108.7 ms

S2 Prediction: a Chinese text document · 272.5 ms

chinese-2

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 108.1 ms

S2 Prediction: a Chinese text document · 271.8 ms

chinese-3

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 109.6 ms

S2 Prediction: a Chinese text document · 287.6 ms

chinese-4

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 109.0 ms

S2 Prediction: a Chinese text document · 274.4 ms

shapes-1

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 108.4 ms

S2 Prediction: an illustration of geometric shapes · 274.3 ms

shapes-2

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 110.7 ms

S2 Prediction: an illustration of geometric shapes · 264.1 ms

shapes-3

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 109.7 ms

S2 Prediction: an illustration of geometric shapes · 267.7 ms

shapes-4

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 108.1 ms

S2 Prediction: an illustration of geometric shapes · 265.2 ms