Benchmark Report 24 Oct 2026

The ARMv9 Speed Paradox: Auditing Real-World Code with Gemma 4 on Bare Metal

Running automated security audits on real-world codebases with frontier LLMs usually comes with two painful tradeoffs: cloud API bills and headaches or stripped-down context windows that miss cross-file vulnerabilities.

For Week 3 of Hacktoberfest 2026 ("Slow and Steady"), we tested what happens when you run Google's Gemma 4 family on bare-metal ARMv9 silicon with a massive 65,536-token context window, evaluating real open-source targets against verified ground-truth vulnerabilities.

The Test Targets: Verified Ground Truth

Rather than synthetic benchmarks or toy examples, we scanned two real-world Go projects where our team previously discovered and remediated zero-day security flaws:

jhaals/yopass at commit 16bc350a (pre-fix commit before the merged patch in PR #3973). Scanned files: pkg/server/oidc.go, pkg/server/apitoken.go, and pkg/server/middleware.go.
rs/curlie at commit 5dfcbb17 (pre-fix commit before patches PR #104 and PR #105). Scanned files: main.go, args/httpie.go, args/parse.go, formatter/json.go, formatter/color.go, and formatter/binaryfilter.go.

The ARMv9 Hardware Paradox: Why 4-bit Beats 2-bit

Conventional wisdom suggests that smaller quants are always faster because they read fewer bytes from memory. On our 8-core ARMv9 Neoverse-V2 instance, we hit a surprising hardware counterexample:

Model Architecture	Quantization Variant	RAM / Disk Size	Prompt Eval Speed	Token Gen Speed	yopass (Time / Triage)	curlie (Time / Triage)
Gemma-4-12B	`UD-Q2_K_XL`	4.34 GB	8.30 t/s	8.38 t/s	4,040 s (0 valid / 0 rejected)	8,720 s (2 valid / 0 rejected)
Gemma-4-12B	`IQ4_XS`	5.94 GB	10.83 t/s	11.16 t/s	9,260 s (1 valid / 3 rejected)	18,260 s (6 valid / 7 rejected)
Gemma-4-12B	`UD-Q4_K_XL`	6.86 GB	15.12 t/s	11.08 t/s	7,375 s (1 valid / 3 rejected)	12,715 s (4 valid / 6 rejected)
Gemma-4-12B	`UD-Q6_K_XL`	9.95 GB	14.92 t/s	8.06 t/s	8,730 s (1 valid / 3 rejected)	19,739 s (2 valid / 11 rejected)
ThinkingCap-26B-A4B	`Q4_K_M`	15.84 GB	55.24 t/s	21.78 t/s	3,170 s (0 valid / 5 rejected)	6,404 s (2 valid / 12 rejected)
ThinkingCap-26B-A4B	`Q6_K`	21.70 GB	54.37 t/s	21.04 t/s	3,213 s (1 valid / 4 rejected)	OOM crash at 2,500 s (32 GB RAM ceiling)
Ternary-Bonsai-27B	`PTQ1_0`	5.54 GB	6.85 t/s	3.68 t/s	22,970 s (1 valid / 4 rejected)	27,090 s (1 valid / 3 rejected)
Gemma-4-26B-A4B	Vertex MaaS (Cloud)	Cloud API	~650 t/s	55.10 t/s	62 s (0 valid / 4 rejected)	90 s (4 valid / 9 rejected)

*Bold values indicate best performance among local bare-metal CPU runs. Managed cloud API is included as an external reference baseline.

Compute Economics: Bare-Metal CPU vs. Cloud APIs

Benchmarking deep context windows often raises concerns about cloud infrastructure costs. For this study on Google Cloud Platform, our hardware setup consisted of c4a-standard-8 instances (8 vCPUs on ARM Neoverse-V2 silicon, 32 GB RAM) billed at $0.282 / hour with standard hyperdisk storage.

gemma-arm-26b (ThinkingCap-26B-A4B MoE on yopass and curlie across Q4_K_M and Q6_K): ~5.5 hours runtime = $1.55
gemma-arm-c4a (Gemma-4-12B dense across four quantization levels): ~13.5 hours runtime = $3.80
gemma-arm-bonsai (Ternary-Bonsai-27B 1.58-bit execution): ~15.0 hours runtime = $4.23

Running the full local benchmark across all seven model configurations cost under $10.00 in total GCP compute expenses.

For comparison, running the same audit against Google Cloud's managed Vertex AI Model Garden (google/gemma-4-26b-a4b-it-maas at $0.15/M input and $0.60/M output) completed both repository audits in 2.5 minutes with a combined API cost of $0.06. By contrast, running the same 65k multi-round skeptical triage through commercial frontier closed-source APIs regularly reaches $25 to $60 per repository.

For teams working within restricted networks or private VPCs where source code cannot leave the boundary, running sparse MoE models on self-hosted ARMv9 CPU hardware offers high-depth context inspection for pennies per run.

Taming the 65,536 Context Window in llama-server

nice -n 10 llama-server \
  -m ./models/gemma-4-12b-it-UD-Q4_K_XL.gguf \
  -c 65536 \
  -n 4096 \
  -np 1 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -t 8 \
  --reasoning-format deepseek \
  --reasoning-budget 1536 \
  --host 127.0.0.1 \
  --port 8080

Three configuration flags proved essential:

-np 1: By default, llama-server partitions -c 65536 across multiple client slots (e.g. 4 slots of 16k tokens). Setting -np 1 allocates the full 64k window to our single audit pipeline.
-fa on -ctk q8_0 -ctv q8_0: Flash Attention combined with 8-bit quantized KV cache reduced context memory overhead by 48%, keeping RAM usage under 18 GB even at 45,000 prompt tokens.
--reasoning-format deepseek --reasoning-budget 1536: Gemma 4 outputs structured thought tokens (<|thought|>). Separating thought tokens from the final JSON response prevented syntax corruption when applying strict JSON response grammar.

Two Pipeline Fixes for Local 65k Models

Deploying local 65k context models uncovered two subtle edge cases in our scanner pipeline:

Greedy Markdown Code Block Parsing: The original regex matched the first code block, clobbering the final report if the model quoted Go source code in its thought trace. Iterating backward through blocks and validating JSON resolved the issue.
Runaway Generation Loops: Adding strict -n 4096 flags and client-side token caps prevented infinite generation loops after exhausting reasoning budgets.

Detection Accuracy and Skeptical Triage

In single-pass scanning, models produced numerous warnings. Subjecting each finding to 3 independent rounds of skeptical triage successfully filtered out false positives (e.g. flagging Go's linear-time regexp for ReDoS) while preserving valid findings (such as unbounded buffer growth and OIDC fail-open handling).

Sources and Reproducibility

← Back to Success Stories