---

license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license language:

• en base_model:
• google/gemma-4-26B-A4B-it base_model_relation: quantized tags:
• telecom
• telecommunications
• open-telco-ai
• classification
• typed-decisions
• decision-engine
• system-one
• gemma4
• mixture-of-experts
• awq
• w4a16
• compressed-tensors
• 4-bit
• amd
• rocm
• nvidia
• cuda
• h100
• vllm pipeline_tag: text-generation

---

OTel-2.0-Decision-26B-A4B-4bit

Versioning notice: This card describes the current OTel-2.0-Decision-26B-A4B-4bit checkpoint. The model weights may be updated over time. For reproducible evaluation or production deployment, pin a specific model revision, checkpoint hash, or release tag rather than relying on the floating latest version.

Details on the training data, training method and configuration, data-processing pipeline, and training infrastructure will be added soon.

OTel-2.0-Decision-26B-A4B-4bit is a 4-bit W4A16 build of Google's Gemma 4 26B-A4B-it for typed decisions: classification, Yes/No questions, and ordered levels. Questions are supplied at request time, and decisions are read from answer-label probabilities rather than generated prose.

Intended Use

Classify messages, call transcripts, tickets, and documents, or ask multiple questions about the same text. Use label probabilities for confidence thresholds and routing. This release is not intended for open-ended chat, retrieval, or reranking.

Evaluation

Benchmarks use the typed-decision readout (level=L0, no prior, shared-prefix execution, adaptive Yes/No ordering with margin 4.0). All runs reported zero errors and answer_mass of 1.00.

Benchmark	AMD MI350P	AMD Ryzen AI MAX+ PRO 395 (Strix Halo)
JevBench public (231 tasks)	86.58% to 87.45% (200 to 202 of 231)	87.45% (202/231)

JevBench by tier: 100% on easy, 94.44% to 95.83% on original, and 74.77% to 76.58% on hard tasks. The MoE kernel is not bit-deterministic, so results vary on a few low-margin tasks across runs. The Strix Halo run used vLLM 0.26.0 on ROCm 7.13 (48/48 easy, 71/72 original, 83/111 hard).

Use a system message that asks for a single label (see Prompting and Label Probabilities). Without one, the model sometimes answers hard tasks in prose instead of a label, which lowers hard-tier accuracy by about 6 points.

Inference Recipe

Serve on one AMD MI350P with vLLM 0.30.0. Both options use the same weights and API. For NVIDIA GPUs, see NVIDIA H100: vLLM on CUDA below.

Option	Image	Throughput
1. Official vLLM	vllm/vllm-openai-rocm:v0.30.0	22.7k tok/s
2. Prebuilt image	farbodatdocker/otel-decision-vllm-rocm:v0.30.0	32.3k tok/s (42% more)

Throughput figures are peak aggregate output tokens per second. The highest throughput is reached only when the GPU runs near maximum utilization, with many concurrent requests in flight. A single request runs much slower, so batch or parallelize requests to reach these numbers.

Download

pip install -U huggingface_hub
hf download farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit \
    --local-dir ./OTel-2.0-Decision-26B-A4B-4bit

Option 1: Official vLLM

docker run --rm --name otel_decision \
  --device /dev/kfd --device /dev/dri --group-add video \
  --ipc=host --security-opt seccomp=unconfined -p 8000:8000 \
  -v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \
  vllm/vllm-openai-rocm:v0.30.0 \
    /model \
    --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
    --max-model-len 16384 \
    --enable-prefix-caching \
    --max-logprobs 64 \
    --port 8000

With vLLM 0.30.0 already installed, the equivalent is:

vllm serve ./OTel-2.0-Decision-26B-A4B-4bit \
  --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
  --max-model-len 16384 --enable-prefix-caching --max-logprobs 64 --port 8000

Option 2: Prebuilt Image

The official image with a corrected Gemma 4 MoE kernel and MI350P tile configuration.

docker run --rm --name otel_decision \
  --device /dev/kfd --device /dev/dri --group-add video \
  --ipc=host --security-opt seccomp=unconfined -p 8000:8000 \
  -v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \
  farbodatdocker/otel-decision-vllm-rocm:v0.30.0 \
    /model \
    --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
    --max-model-len 16384 \
    --enable-prefix-caching \
    --max-logprobs 64 \
    --moe-backend flydsl \
    --port 8000

Use --moe-backend flydsl only with Option 2. The first start takes 4 to 10 minutes.

System One API (/v1/systemone)

vLLM serves this model on its System One endpoint with no conversion. One request carries a state and any number of named questions (choice, Yes/No, score), and the reply gives a probability for every option. The endpoint is on vLLM main and in nightly builds, not yet in a release. Tested on one NVIDIA H100 80GB with 0.33.0rc1.dev613+g44965a109, the vLLM main build of commit 44965a1; the command below installs exactly that build. A newer nightly or release that includes /v1/systemone should also work: replace the pinned install line with uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly.

uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install "vllm==0.33.0rc1.dev613+g44965a109" --torch-backend=auto \
  --extra-index-url https://wheels.vllm.ai/44965a10974ee417f6f937eee207409462b2a0ab

vllm serve farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit \
  --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
  --max-model-len 16384 --enable-prefix-caching \
  --logprobs-mode raw_logprobs --port 8000

--logprobs-mode raw_logprobs is required. With prefix caching, the state is prefilled once and each question only adds its own tokens.

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "OTel-2.0-Decision-26B-A4B-4bit",
  "instructions": "You are a decision function. You will be given a state and one question. Reply with the answer label only: no words, no punctuation, no explanation.",
  "state": "Caller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded. I already rebooted the router twice.",
  "questions": {
    "refund_requested": {"type": "noul", "instructions": "Did the caller explicitly ask for a refund or credit?"},
    "route": {"type": "choice", "instructions": "Which team should handle this call?",
              "criteria": {"billing": "Charges, refunds and payments", "technical": "Internet, TV and equipment faults", "sales": "New plans and upgrades"}},
    "urgency": {"type": "score", "instructions": "How urgent is this call?", "criteria": ["Routine", "Soon", "Today"]}
  }
}'

Each answer has probabilities for every option and confidence; a Yes/No answer's noul is the probability of yes. diagnostics.<id>.label_mass is the probability the model put on the answer labels and should be close to 1.0. Pass the system prompt in instructions as above: without it, label_mass falls close to zero on some hard multiple choice questions.

Measured on /v1/systemone (one request at a time)	Accuracy	Latency mean / median / p95
JevBench public, 231 tasks, one question per request	85.28% (197/231)	0.030 / 0.019 / 0.064 s
Synthetic call transcripts, 150 transcripts with 27 Yes/No questions each, one request per transcript	99.80%	0.114 / 0.112 / 0.148 s per transcript

JevBench is lower than in Evaluation above because the endpoint writes its own prompt and reads each question once, without the adaptive Yes/No ordering.

NVIDIA H100: vLLM on CUDA

Serve on one NVIDIA H100 80GB (SXM, HBM3) with the official CUDA image of vLLM 0.26.0. vLLM selects the Marlin W4A16 kernels for both the dense and MoE layers automatically, and no config shim or extra flags are needed.

docker run --rm --name otel_decision \
  --gpus '"device=0"' \
  --ipc=host -p 8000:8000 \
  -v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \
  vllm/vllm-openai:v0.26.0 \
    /model \
    --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
    --max-model-len 16384 \
    --enable-prefix-caching \
    --max-logprobs 64 \
    --port 8000

With vLLM 0.26.0 already installed (pip install vllm==0.26.0), the equivalent is:

CUDA_VISIBLE_DEVICES=0 vllm serve ./OTel-2.0-Decision-26B-A4B-4bit \
  --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
  --max-model-len 16384 --enable-prefix-caching --max-logprobs 64 --port 8000

Measured (1x H100 80GB, vLLM 0.26.0, 512 output tokens per request)	Value
Peak aggregate output, 256 concurrent requests	10.6k tok/s
1,024 concurrent requests	10.1k tok/s
Single request	223 tok/s

The weights use about 17 GB, which leaves about 51 GB for KV cache (about 242k tokens) at the default --gpu-memory-utilization 0.92. The first start takes about 4 minutes because of torch.compile and CUDA graph capture. Later starts reuse the compile cache. To serve several replicas on one multi-GPU node, add --data-parallel-size N.

AMD Ryzen AI MAX+ PRO 395 (Strix Halo): vLLM on ROCm

Serve on one AMD Ryzen AI MAX+ PRO 395 (Strix Halo: Radeon 8060S GPU, 128 GB unified memory, configured as 96 GB GPU and 32 GB NPU) with vLLM 0.26.0 and ROCm 7.13 for gfx1151.

# 1. ROCm 7.13 + vLLM for gfx1151 (managed by the ROCm CLI)
rocm engines install vllm
PY=~/.rocm/runtimes/wheel/release-wheel-gfx1151-7-13-0/bin/python

# 2. Config shim: transformers 5.19 drops Gemma 4's global head sizes that vLLM 0.26 reads
mkdir -p shim && cat > shim/sitecustomize.py <<'EOF'
from transformers import PretrainedConfig
from transformers.models.gemma4 import configuration_gemma4 as g4
PretrainedConfig.allow_global_per_layer_attribute_access = True
_init = g4.Gemma4TextConfig.__init__
def init(self, *a, **k):
    _init(self, *a, **k)
    lt, pl = list(getattr(self, "layer_types", []) or []), getattr(self, "per_layer_config", None)
    if pl is not None and "full_attention" in lt:
        full = pl[lt.index("full_attention")]
        for dst, src in (("global_head_dim", "head_dim"), ("num_global_key_value_heads", "num_key_value_heads")):
            if dst not in self.__dict__ and getattr(full, src, None) is not None:
                object.__setattr__(self, dst, getattr(full, src))
g4.Gemma4TextConfig.__init__ = init
EOF

# 3. Serve (the torch wheel links OpenMPI 4's libmpi_cxx.so.40; point at its lib dir)
LD_LIBRARY_PATH=/path/to/openmpi4/lib PYTHONPATH=$PWD/shim \
$PY -m vllm.entrypoints.cli.main serve ./OTel-2.0-Decision-26B-A4B-4bit \
  --served-model-name OTel-2.0-Decision-26B-A4B-4bit \
  --max-model-len 16384 --enable-prefix-caching --max-logprobs 64 \
  --gpu-memory-utilization 0.55 --max-num-seqs 256 --port 8000

Measured (AC power, performance profile)	Value
Peak aggregate output, 256 concurrent requests	377 tok/s
64 concurrent requests	257 tok/s
Single request	30 tok/s
JevBench public, 8 concurrent	1.27 tasks/s

Peak throughput again needs near-maximum utilization: aggregate output keeps rising up to 256 concurrent requests. Keep HIP graphs on (the default). --enforce-eager cuts single-request speed to about 19 tok/s. The shim is not needed with vLLM releases that read Gemma 4's per-layer config directly. torch fails to import without an OpenMPI 4.x runtime. Distributions that ship only OpenMPI 5 (for example Ubuntu 26.04) lack libmpi_cxx.so.40, so extract an OpenMPI 4 package into a local directory and use that directory; no sudo is needed.

Prompting and Label Probabilities

Put the text first, then the question and labels. Use letters (A, B, …) for options or Yes/No, and numbers for ordered levels. Add a system message that asks for exactly one label, and request only the label through the chat endpoint, which applies the repository's template with thinking disabled.

import math
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
question = """State:
Order #1182. Status: shipped on 3 September. Carrier: DHL.

Question: Has the order been shipped? Answer strictly from the facts stated in the text.
Options:
A. Yes
B. No
Answer with the letter only."""
labels = ["A", "B"]

r = client.chat.completions.create(
    model="OTel-2.0-Decision-26B-A4B-4bit",
    temperature=0,
    max_tokens=1,
    logprobs=True,
    top_logprobs=64,
    messages=[
        {"role": "system", "content": "You are a typed decision engine. Reply with exactly one answer label "
                                      "(a single letter or number) and nothing else. Do not explain."},
        {"role": "user", "content": question},
    ],
)
top = {t.token: t.logprob for t in r.choices[0].logprobs.content[0].top_logprobs}
probs = {label: math.exp(top[label]) if label in top else 0.0 for label in labels}
print("probabilities:", probs)
print("decision:", max(probs, key=probs.get))
print("answer_mass:", sum(probs.values()))
# Expected: decision A, answer_mass approximately 1.0

Choose the label with the highest probability. answer_mass is the summed probability of the answer labels and should be close to 1.0; a low value indicates an invalid decision readout. For multiple questions, keep the shared text byte-identical so prefix caching can reuse it.

Verify the Deployment

Check that http://localhost:8000/v1/models lists the branded model name, then run the example above and confirm decision A with answer_mass near 1.0. Validate representative prompts against a reference run before deployment.

Limitations and Responsible Use

Evaluation covers English text only. JevBench is small and does not guarantee production accuracy. Validate on your own human-labeled data, and do not assume label probabilities are calibrated for your workload. Keep human review for consequential decisions.

The recipe starts an unauthenticated server. Add authentication before exposing it beyond your machine.

Resources

• Project Page
• OTel code
• OTel Selected Coverage
• OTel Paper
• OTel 1.0 Coverage
• OTel 2.0 Coverage
• OTel Docker

Citation

@inproceedings{tavakkoli2026otel2,
  title  = {OTel 2.0: Open Telco AI Datasets, Benchmarks, and Models},
  author = {Tavakkoli, Farbod and others},
  booktitle = {TBD},
  year   = {2026},
  url    = {https://huggingface.co/farbodtavakkoli}
}

Contact

If you have any technical questions, please feel free to reach out to mark.austin@att.com or farbod.tavakkoli@att.com or farbodtavakoli@gmail.com