FastVLM: Efficient Vision Encoding for Vision Language Models

High-resolution images preserve small text and visual detail, but they increase both image-encoding cost and the number of tokens processed by a language model. Reducing only the token count does not address an expensive image encoder. FastVLM treats these components together when designing an efficient vision-language model.

FastViTHD introduces a hybrid convolution-and-attention vision backbone with additional downsampling for high-resolution image processing.

An architecture study jointly varies image resolution, vision encoding, and language-model size to examine the accuracy–latency design space.

The project releases model checkpoints, inference code, export utilities, and an application demonstrating execution on Apple devices.

FastVLM retains the image-encoder, projector, and language-model structure used in LLaVA-style systems. Its encoder begins with efficient convolutional processing and RepMixer blocks, then uses attention in later stages. Moving attention to smaller feature maps limits its spatial cost.

The high-definition backbone adds a further downsampling stage, producing a shorter visual sequence for the language model. The study also aggregates features from multiple encoder stages, using pooling or depthwise convolutions to align their spatial scales. A projector maps the resulting visual representation into the decoder's input space.

Training first prepares the vision encoder through image–text contrastive learning and then aligns and instruction-tunes the multimodal model. Experiments compare direct changes to input resolution with tiled image processing. At inference, the image must be encoded before the decoder can process visual and text tokens and begin generating its answer; the implementation measures these stages separately.

Benchmarks include GQA, TextVQA, DocVQA, POPE, and SEED-Bench, covering question answering, document reading, visual understanding, and object hallucination. Additional image–text evaluations characterize the encoder before its integration into the language model.

Latency experiments report image-encoding time and time to first token under a specified device setup. Ablations vary encoder architecture, feature aggregation, input resolution, tiling, and decoder size. These are general visual-language evaluations; they do not directly test GUI click accuracy or complete interface workflows.