Install
Inference & Serving
Serving runtimes, Triton, vLLM, ONNX Runtime, and scalable endpoints.
- 3 Tracked terms
- Last 30 days Feed window
What this topic collects on
An article joins this feed when it matches these terms. Each one is also a search of its own.
Related topics
Latest in Inference & Serving
Gemma 4 QAT on One TPU v5e: What Runs and What Doesn't
22+ hour, 18+ min ago (770+ words) This article provides a step by step guide to repacking Google's quantization-aware-trained (QAT) Gemma 4 weights for vLLM and serving them on one Google Cloud TPU v5e chip, with every build scored for classification, math, tool calling, throughput and long prompts. Every…...
Repacked QAT Gemma 4 on One TPU v5e: 12B Serves at 675 Tokens per Second
22+ hour, 18+ min ago (737+ words) This article provides a step by step guide to repacking Google's quantization-aware-trained (QAT) Gemma 4 weights for vLLM and serving them on one Google Cloud TPU v5e chip, with every build scored for classification, math, tool calling, throughput and long prompts. Every…...
Gemma 4 E2B on an AMD MI300X: Which Weight Format Should You Serve?
22+ hour, 19+ min ago (1054+ words) This article provides a step by step guide to serving ten weight formats of Gemma 4 E2B on one AMD Instinct MI300X through vLLM, with every build timed across a grid of request counts and prompt lengths on the same card, image and…...
Podman Test Week: Help test the Rust-based conmon v3
1+ day, 21+ hour ago (465+ words) The Podman and Fedora Quality teams are organizing a test week from Monday, October 12, through Sunday, October 18, 2026. We want your help testing conmon v3, the Rust-based container monitor used with Podman. If you use Podman, try the same containers and commands…...
VRAM usage doesn't decrease even with 4-bit quantization? What I learned from actual measurements with vLLM
2+ week, 3+ day ago (545+ words) "If you quantize a model to 4-bit, VRAM usage will be roughly 1/4"—this is a feeling you naturally develop when working with local LLMs. However, when I actually ran FP16, AWQ, and GPTQ on vLLM and measured the VRAM usage with…...
Jevfire vs Gemma 4 E2B: Benchmarks & Cost
3+ day, 14+ hour ago (107+ words) Jevfire vs Gemma 4 E2B Estimated · Public rank #167 The page does not recommend a cost winner because at least one model cannot fit the stated workload in one request. Gemma 4 E2B does not fit this workload in one request. Jevfire has no comparable…...
JEV Qwen3.5-9B Base NVFP4 vs XLM-V large (1.2B) (Belebele Translate-Train-All)
4+ day, 14+ hour ago (144+ words) Updated October 5, 2026. We do not rank this pair: at least one has no public score. Public scores include evidence status and uncertainty. JEV Qwen3.5-9B Base NVFP4 vs XLM-V large (1.2B) (Belebele Translate-Train-All) The public evidence has no benchmark result shared by both models,…...
I Ran Gemma 4, Qwen3–8B, and GPT-OSS Locally. Here’s What I Found
4+ day, 19+ hour ago (160+ words) I used to treat local AI as a hobby for people with spare GPUs and too much free time. Then I got tired of sending every small prompt to a paid API and wondering where my data went. So I…...
vllm preload¶
6+ day, 6+ min ago (1709+ words) When passing JSON CLI arguments, the following sets of arguments are equivalent: Additionally, list elements can be passed individually using +: Configuration for the model. To swap the Rust BPE backend that powers HF fast tokenizers for the fastokens implementation, set…...
vLLM's weight cache can serve another checkpoint's weights when the tensor layout matches
6+ day, 6+ hour ago (262+ words) RTX 2070 (8 GB) on Windows 11, WSL2 Ubuntu, driver 591.86, vLLM 0.30.0 with torch 2.13.0+cu130, everything in fp16 since the card has no bf16. Two checkpoints, following the report: The files differ, and loaded normally each behaves like itself. Greedy decoding, 12 tokens, "The capital of France is": Then…...