On This Page8 sections
Build a small image-question application with the same FastVLM inference worker used by this site's playground. The starter includes image input, a receipt sample, streamed text, model-load progress, cancellation and JSON export.
Download the complete starter ZIP or try the running playground.
Requirements
- Node.js 22 or newer for this starter's development environment.
- A browser exposing WebGPU and an adapter with
shader-f16. - Enough free GPU memory for the model and its runtime buffers.
- Localhost during development and HTTPS when deployed.
- A network connection for the initial model download, approximately 1.1 GB.
The project pins Transformers.js 4.3.0 and uses onnx-community/FastVLM-0.5B-ONNX. This is a community export of Apple's model. See the ONNX model card and Transformers.js WebGPU documentation for the model and runtime context.
Start the app
Extract the ZIP, open a terminal in its fastvlm-webgpu directory, and run:
npm install
npm run devOpen the localhost address printed by Vite. Select Use sample receipt, keep the amount question, and press Run. Inspect whether the answer matches the receipt's $15.00 total; generated text may vary and must be checked.
Export JSON to inspect the full answer and timings. Also try Cancel during model preparation and run again to check recovery.
How the files fit together
| File | Responsibility |
|---|---|
index.html | Image controls, question, answer and status UI |
src/main.ts | Decodes input, manages controls, handles worker messages and exports JSON |
src/worker.ts | Loads the model and processor, runs inference and streams text |
src/protocol.ts | Typed input, output and timing messages |
src/capabilities.ts | Secure-context, WebGPU adapter and shader-f16 checks |
public/sample.png | Synthetic receipt used for a first-run check |
The ZIP is packaged from this project's worker and capability code. The example uses a small Vite/TypeScript UI so it can be adapted to React, Svelte or another framework without copying a whole website.
Run inference in a worker
The main thread resizes the image to a maximum of 1024 pixels on its longest side, fills transparent areas with white and prepares a PNG data URL. It passes that image and the question to a worker only when the user requests a run.
const worker = new Worker(new URL('./worker.ts', import.meta.url), {
type: 'module',
});
worker.postMessage({ image: preparedImage, prompt: question });The worker sends loading, progress, running, text, result and error messages. A result contains the model identifier, answer and measured timings. Keep the worker alive between completed questions to reuse its loaded session.
Cancel terminates the worker. The next run builds a new session; whether weight files need downloading again depends on the browser's cache. Do not promise that a user changing origin or clearing storage will retain downloaded models.
Show useful progress
Model preparation includes both file reads and session initialization. The starter reports bytes read, then shows that preparation is still in progress. It does not invent a completion percentage when the total is unknown.
Keep a clear Cancel control throughout checking, downloading and inference. Prevent a second run from starting while the first is active, and ignore messages from a cancelled session. Reset the answer and exported result when the input image changes.
Understand the exported timings
| Field | What it measures |
|---|---|
load_ms | Cache/download work and model session initialization |
inference_ms | Image processing through completed generation, excluding preparation |
first_text_ms | Image processing through the first non-empty streamed text chunk |
generated_tokens | Number of newly generated output tokens |
First text is not a strict first-token benchmark. A single image run also does not establish stable device performance. Exported JSON contains the question and answer but does not include image pixels.
Build and deploy
npm run build
npm run previewDeploy the generated dist directory over HTTPS, preserving its worker JavaScript paths. This pinned Transformers.js release loads its ONNX runtime from versioned jsDelivr URLs; model weights come from Hugging Face. Both hosts must be reachable by the browser. This example does not require an inference server.
For Cloudflare Workers Static Assets, the included .assetsignore excludes Vite's unused local ONNX WASM fallback, which exceeds the platform's 25 MiB per-file limit. Keep that file in the root of the deployed asset directory. Recheck the runtime's network requests when upgrading Transformers.js before retaining this exclusion. See Cloudflare's asset configuration.
When it does not run
| Problem | Response |
|---|---|
| WebGPU is unavailable | Offer the Python workflow before model loading. |
| The adapter lacks shader-f16 | Explain the requirement. WebGPU availability alone is insufficient for this configuration. |
| Out of GPU memory | Close competing workloads or choose another runtime. Shrinking an image does not shrink the weights. |
| Worker or WASM request fails | Check generated worker paths, the site's base path and access to jsDelivr runtime files. |
| Model download fails | Check network access, free storage and the model host's response. |
| The answer is wrong | Inspect the original image and prompt. Successful inference does not guarantee factual correctness. |
For application telemetry, record action names, bounded categories and timings. Keep images, file names, questions and answers out of analytics. Add your application's own privacy and consent handling before collecting events.
Article revised September 23, 2026. Continue with model selection, Python integration or the FastVLM overview.
Continue Reading
More articles connected to the same themes, protocols, and tools.
Referenced Tools
Browse entries that are adjacent to the topics covered in this article.
