If you have ever rewritten the same image prompt three times for three AI providers, vlmhub is built for you: a small open-source Python library that sends text and images to nine different vision-language model backends through one generate() call. It was posted to Hacker News as a Show HN on 19 October 2026. It is tiny and very new, but the problem it goes after is one most multimodal developers know well.
Key takeaways
- Vlmhub wraps Ollama, MLX-VLM, vLLM, Hugging Face Transformers, Gemini, Vertex AI, OpenAI, Anthropic and OpenRouter behind one interface.
- Prompts are lists of
TextBlockandImageBlockobjects that you can mix in any order. - Models and their default settings live in one editable
models.jsonregistry. - It is MIT-licensed and built on LiteLLM and Transformers. When it was posted on 19 October 2026 it had 1 GitHub star and 35 commits.
What is vlmhub, and what problem does it solve?
A vision-language model (VLM) takes images and text as input and writes text back. Hugging Face’s VLM explainer lists the usual jobs: captioning, answering questions about images, reading documents and locating objects.
The trouble is that each backend wants images packaged differently. A local Ollama server, an in-process Transformers model and the Anthropic API each have their own way to attach a picture, set the temperature or limit output length. Benchmarking the same prompt across five models can turn into five small integration projects.
The vlmhub repository, published by GitHub user dbiswas55, describes itself as a lightweight, config-driven framework that hides those differences. Think of it as a universal travel adapter: the plug in the wall changes, but your charger stays the same.
How does a vlmhub call actually look?
The README example is short. You create a model by its registry name, such as Model("ollama/gemma3-4b"), Model("gemini/flash-2.5") or Model("transformers/qwen3vl-8b"). Then you pass a list of blocks to model.generate().
A two-image comparison prompt reads like a sentence: a TextBlock saying “Here are two images:”, an ImageBlock pointing to a file, a second ImageBlock holding an in-memory PIL image, and a final TextBlock asking what the main differences are. You read the answer from reply["text"].
The reply is a dictionary that also includes logs, model and params, so you can see which settings were really used. That matters, because vlmhub quietly changes some requests. Models marked "sampling": false in the registry drop sampling parameters such as temperature and top_p, and the logs record that they were dropped.
Which backends and models does vlmhub support?
The repository groups its nine providers into three kinds of setup:
| Type | Providers | Where it runs |
|---|---|---|
| Local server | Ollama, MLX-VLM, vLLM | Your machine. MLX-VLM targets Apple Silicon and vLLM targets CUDA GPUs |
| In-process | Hugging Face Transformers | Inside your Python process on CUDA, MPS or CPU |
| Cloud API | Gemini, Vertex AI, OpenAI, Anthropic, OpenRouter | The provider’s servers, billed per call |
The registry comes with models already configured. For local use there are Gemma 3 (4B and 12B) and Qwen3-VL (4B and 8B). The Transformers backend adds Qwen2.5-VL, InternVL3.5, Molmo2, Idefics3 and LLaVA-OneVision, among others. There is also a set of vLLM “judge” models, including Prometheus and CompassJudger, which are used to grade other models’ answers.
The Transformers backend also takes your hardware into account. It can load models in 4-bit quantized form, which stores the weights at lower precision so large models fit in less memory. Its fallback_dtype setting covers older GPUs such as the V100, which don’t natively support the bfloat16 number format many new models expect.
Why not just use LiteLLM directly?
It’s a fair question, since vlmhub is built on LiteLLM. LiteLLM’s vision docs already show image parts passed in an OpenAI-style message format, and its supports_vision() helper tells you whether a model accepts images at all.
What vlmhub adds sits on top of that:
- In-process models. LiteLLM talks to servers and APIs. Vlmhub also loads Transformers models directly, with quantization handled for you.
- A model registry. Defaults for each model live in JSON, not in your code. Values you pass on a call override the registry.
- Simpler prompts. You pass file paths or PIL images, and vlmhub does the base64 encoding and message building.
My opinion: that mix suits researchers best. If you want to run the same image-reasoning benchmark across a local Qwen model, a quantized InternVL and GPT or Claude, writing models.json entries is much less work than writing glue code.
Should you use vlmhub in production?
Probably not yet, and the numbers explain why. When it was posted on 19 October 2026, the repository had 1 star, no forks, no tagged releases and 35 commits on main. The Hacker News post had 1 point and no comments.
Installation is also from source for now. You clone the repo and run pip install -e . for API backends, or pip install -e ".[transformers]" to add PyTorch and Transformers. The README also describes adding it to another project as a git submodule. There is no PyPI package, which tells you this is an early research tool, not a stable dependency.
For experiments, evaluations and side projects, the risk is low. The MIT licence lets you fork it, and the code is small enough to read in one sitting.
What happens next
Three things to watch: a PyPI release with version numbers, how fast the registry keeps up with new models, and whether anyone besides the author starts contributing. Single-maintainer wrappers often stall once the author’s own research project ends.
The wider trend is clearer. Local runtimes such as MLX-VLM and Ollama keep improving, so more developers will want to compare a free local model with a paid API on the same prompt. Tools that make that one line of config will find users, whether or not this one is the tool that wins.
FAQ
Is vlmhub free?
Yes. The library is MIT-licensed. Cloud providers such as OpenAI, Anthropic and Gemini still bill you for API usage, but local backends like Ollama cost nothing beyond your own hardware.
Can vlmhub run on a Mac?
Yes. It supports MLX-VLM on Apple Silicon and Ollama, and its Transformers backend can use Apple’s MPS GPU backend or fall back to the CPU.
Can I send more than one image in a prompt?
Yes. Prompts are lists of TextBlock and ImageBlock objects in any order, so you can send several images with text between them, from file paths or in-memory PIL images.
Want to try it? Clone the repo, point Model() at a small local model such as ollama/gemma3-4b, then send the same prompt to a cloud model and compare the answers.
Leave a comment