Run FastVLM with Python and Transformers: Complete Image Q&A Example

This guide takes one local image and returns an answer using the native FastVLM integration in Transformers. The downloadable script defaults to the community conversion KamilaMila/FastVLM-0.5B, uses a bounded generation length, and includes compatibility handling for that conversion.

For a quick browser trial, use the FastVLM playground. For Apple's original checkpoints, read the separate repository path below before choosing files.

Choose a compatible runtime

A model identifier is not a drop-in replacement for every runtime. The Transformers FastVLM documentation describes its native interface; Apple's model cards also document their own usage paths.

Prepare an isolated environment

Run these commands in your working directory on macOS or Linux:

python -m venv .venv
source .venv/bin/activate
python -m pip install "transformers==5.0.0" "timm==1.0.29" torch pillow

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 instead. Select a PyTorch build appropriate for your device if you need CUDA support. The script selects CUDA first, then Apple MPS, then CPU. It uses float16 on CUDA and float32 on MPS/CPU; performance and memory requirements differ between those paths.

The version pins preserve the original example's compatibility baseline. Keep this environment separate from an existing application before changing its dependencies.

Download and run the example

Download fastvlm_transformers.py, save it in your working directory, and supply a readable local image:

python fastvlm_transformers.py image.png \
  --prompt "What is the total on this receipt? Answer with the amount only."

The first run downloads model files and creates the inference session. Allow for the model download and free memory. Later runs may use the downloaded files, but a new process still has to load the model.

The script prepares an image-and-text chat message, passes it through the processor and prints only tokens generated after the input. It caps generation at 192 new tokens. Use a short task-specific question when you need a field rather than a general description.

Complete script

The downloadable file and this listing are the same implementation.

"""Single-image FastVLM inference using the native Transformers integration.

Setup: python -m pip install 'transformers==5.0.0' 'timm==1.0.29' torch pillow
Run: python fastvlm_transformers.py image.png --prompt 'What is in this image?'
Source: https://huggingface.co/docs/transformers/model_doc/fast_vlm
The default is a community conversion used by that guide, not Apple's original ZIP.
"""

import argparse
from pathlib import Path

import torch
from transformers import AutoConfig, AutoProcessor, FastVlmForConditionalGeneration


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument('image', type=Path)
    parser.add_argument('--prompt', default='Describe this image briefly.')
    parser.add_argument('--model', default='KamilaMila/FastVLM-0.5B')
    args = parser.parse_args()
    if not args.image.is_file():
        parser.error('The image file does not exist.')

    device = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
    dtype = torch.float16 if device == 'cuda' else torch.float32
    config = AutoConfig.from_pretrained(args.model)
    if args.model == 'KamilaMila/FastVLM-0.5B':
        # This older conversion records weight tying only in text_config.
        # Transformers 5 also needs it on the outer config to restore lm_head.
        config.tie_word_embeddings = config.text_config.tie_word_embeddings
    model = FastVlmForConditionalGeneration.from_pretrained(args.model, config=config, dtype=dtype).to(device).eval()
    processor = AutoProcessor.from_pretrained(args.model, use_fast=False)
    messages = [{'role': 'user', 'content': [
        {'type': 'image', 'path': str(args.image.resolve())},
        {'type': 'text', 'text': args.prompt},
    ]}]
    inputs = processor.apply_chat_template(
        messages, add_generation_prompt=True, tokenize=True,
        return_dict=True, return_tensors='pt',
    ).to(device)
    if 'pixel_values' in inputs:
        inputs['pixel_values'] = inputs['pixel_values'].to(dtype=dtype)
    # Qwen chat turns may end with im_end instead of the legacy endoftext.
    stop_tokens = [processor.tokenizer.eos_token_id]
    chat_end = processor.tokenizer.get_vocab().get('<|im_end|>')
    if chat_end is not None:
        stop_tokens.append(chat_end)
    with torch.inference_mode():
        output = model.generate(
            **inputs, max_new_tokens=192, do_sample=False, repetition_penalty=1.2,
            eos_token_id=stop_tokens,
            pad_token_id=processor.tokenizer.pad_token_id,
        )
    answer = output[:, inputs['input_ids'].shape[1]:]
    print(processor.batch_decode(answer, skip_special_tokens=True)[0].strip())


if __name__ == '__main__':
    main()

Troubleshooting

Missing output weights or unreadable answers

For the older KamilaMila/FastVLM-0.5B conversion with Transformers 5.0.0, the original project reproduced a missing lm_head.weight problem. The conversion records weight tying in text_config; the script copies that setting to the outer configuration before loading. This workaround is deliberately scoped to that model identifier.

Repeated answers or another chat turn

The script includes the tokenizer's end token and the Qwen im_end marker when available, decodes only newly generated tokens and limits output length. These measures control generation; they do not guarantee a factually correct answer.

Missing timm or the FastVLM class

Check that installation and execution use the same environment:

python -c "import sys, transformers, timm; print(sys.executable, transformers.__version__, timm.__version__)"

An older Transformers installation may not expose the native FastVLM class. The vision backbone also needs its compatible dependencies.

A stalled model download

Check disk space, network access and any required repository permissions. For an Xet-related transfer problem, the original project's retry used the HTTP path:

HF_HUB_DISABLE_XET=1 python fastvlm_transformers.py image.png

This is a targeted retry option, not a fix for every network failure.

Out of memory

Close other model processes, check that you selected the intended model size, and shorten generation. Moving to CPU changes performance and memory behavior rather than making the model files smaller.

Apple's original repository path

Use a separate environment for the original implementation. Apple's repository documents Python 3.10 and an editable installation:

git clone https://github.com/apple-aiml-research/ml-fastvlm.git
cd ml-fastvlm
conda create -n fastvlm python=3.10
conda activate fastvlm
python -m pip install -e .

Download and extract your chosen checkpoint from the official model zoo, then run:

python predict.py \
  --model-path /path/to/checkpoint-directory \
  --image-file /path/to/image.png \
  --prompt "Describe this image briefly."

This path uses the repository's model implementation. Choose files using the model/download comparison, and check the Stage3 guide if you need the 1.5B checkpoint.

Validation scope

The original FastVLM project recorded a successful $15.00 answer for a synthetic receipt on September 12, 2026 using the pinned Transformers example. This historical example is not a general OCR accuracy result and has not been presented as a new test of every runtime. Check your own images, exact environment and output format before relying on answers.

Article revised September 23, 2026. Return to the FastVLM hub or try the browser version.