Local AI models on the Iknos evidence-reasoning tasks

Iknos is an evidence-reasoning engine. We tested six AI language models, running on our own GPUs, on the five language tasks Iknos depends on, using the exact instructions Iknos gives them in production. The test questions, the pass marks and the rule for picking a winner were all fixed and published before any candidate was tested.

Plan published 2026-10-10 · results 2026-10-13

How to read this page

Score — the share of test items a model got right (for R4, the share it got wrong — lower is better).
Safe-side figure (the number with ↑ or ↓) — a test of a few hundred items has a margin of error, so a model is judged on the cautious end of that margin, not its raw score. That way a lucky run can't pass. (Statisticians: the Wilson 95% interval bound.)
Pass mark — the safe-side figure must clear it; ✓ passes, ✗ fails.
Screen and confirmation sets — two separate sets of test items. A model is first tried on the screen set; only models still in contention go on to the confirmation set, which nobody had looked at. Models that clearly failed were stopped after the screen.
Compression formats (AWQ, FP8, Q4_K_XL, bf16 = uncompressed) — ways of shrinking a model so it fits on one graphics card; vLLM and llama.cpp are the programs that run it.

Models tested

Code	Model	Runs on · compression	Role
B0	Qwen3.5-4B	vLLM, AWQ 4-bit	current model
B0-bf16	Qwen3.5-4B	vLLM, bf16	same model, uncompressed (does compression hurt?)
A3	Gemma-4-26B-A4B	vLLM, AWQ INT4	candidate
A7	Granite-4.1-30B	llama.cpp, Q4_K_XL	candidate
A2	Qwen3.8-27B	llama.cpp, Q4_K_XL	candidate
A9	Ministral-3-14B	vLLM, FP8	candidate

One RTX 3090 graphics card per model; the models' step-by-step "thinking" mode was off.

Tasks and pass marks

Task	What is measured	Pass mark
R1 Negation rewrite	Rewrites a "not X" statement in positive form without changing its meaning	≥ 90% right, safe-side ≥ 85%
R2 Entity extraction	Finds the things (parts, people, places…) a text names	safe-side ≥ 85% found; invents ≤ 2%; finds things in text that has none ≤ 5%
R3 Verifier	Catches a claim that was subtly altered from its source	safe-side ≥ 90% caught; wrongly rejects correct claims ≤ 10%
R4 Claim extraction	Claims pulled from a text that the text doesn't actually support (lower is better)	safe-side ≤ 10%; passages it fails to process ≤ 10%
R5 Evidence judge	Decides whether a piece of evidence supports or contradicts an explanation	safe-side ≥ 80% right; calls contradicting evidence "supporting" ≤ 10%

Winner: among models passing every task, the best safe-side R5 score; if none passes everything, the best safe-side R5 score, with its failures listed.

How the test was kept fair

The test items were fixed and fingerprinted (codes below) before any candidate ran, and nobody looked at the confirmation items in advance. A few protocol changes (amendments 11–15) were made after the current model's first run, before any candidate ran. For R5, the correct answers are cases where three different models agreed, each then checked by an independent reviewer against written rules. The source documents are private, so only totals are published.

Fingerprints of the test sets (SHA-256)

Set	Items	SHA-256
R1 screen	220	`2762bafb01048ca51b983769ba9ab435e7e3831c9907902611265699ae5872be`
R1 confirmation	220	`ecb50c8892b9835e211be7e8cda5e08d78a03ed70fad2dc20112c15a8fd12c50`
R1 grader check items	40	`fb9b57b325c1c8993fa5f92f8c2329c7b9f5186d9d6e97dcc20f882bcf19e4b2`
R2 screen	220	`7cf84055dcbd45d0bbb68b7051648637dc6ab860d1bf8cefd2b40460ab1c981e`
R2 confirmation	220	`6dc68c2d4f1deb595b4734c1898fb8218e0ac923f29349b5b43e05832051daa1`
R3 screen	200	`65ea5bcb74222b4c941e77a19236ee47d996b5fee7a42595e3fef15a78e046f8`
R3 confirmation	200	`a9dd89422fa78e6cef20d7f50c4b6b9f2eb8eae48ef2e793a083b98e832969ad`
R4 screen	108	`f54fd9c6aff306bcef5b6c6575ab9fca642ea78df8b12a2356b24d3bbcf0cbdf`
R4 confirmation	108	`cc97053b65d5415c1bbf6d34b8b9e29c39be1c212e15b34d0f079edd5101f3af`
R4 grader check items	40	`b119a74227489f969b093dbb787a532b4d85dfbf0600fe27e07f9334873501e5`
R5 screen	60	`cfbaf7d81b41cb877e4e55c270fd5b4944124cf9f7c232d17a929a208e25d94b`
R5 confirmation	66	`1db2bc7a91d91fd1dd85bf9f09bbaa5aa828cb097d6c5c382e05c5415b262bbf`

Results at a glance

Each row is one model. The marker is its score; the line runs to its safe-side figure; the dashed line is the pass mark. A ✗ whose line still clears the pass mark failed one of the task's other limits (see "Tasks and pass marks"). Square marker: result from the screen set only.

Results table

Score, then the safe-side figure in brackets (↑ must be at least the pass mark, ↓ at most). ✓ passes, ✗ fails. ^s screen set only (the model was stopped there); otherwise the confirmation set.

Model · runs on	R1 meaning kept	R2 things found	R3 altered claims caught	R4 unsupported claims ↓	R5 evidence judged right
Qwen3.5-4B vLLM · AWQ 4-bit	90.0% (83.3%↑)^s ✗	90.9% (87.3%↑)^s ✗	89% (81.4%↑) ✗	9.8% (11.6%↓)^s ✗	91.7% (81.9%↑)^s ✗
Qwen3.5-4B vLLM · bf16	92.9% (86.6%↑)^s ✓	90.9% (87.3%↑)^s ✓	—	6.2% (7.7%↓)^s ✓	93.3% (84.1%↑)^s ✓
Gemma-4-26B-A4B vLLM · AWQ INT4	77.2% (67.6%↑)^s ✗	88.4% (84.5%↑)^s ✗	96% (90.2%↑) ✓	4.4% (5.8%↓)^s ✓	96.7% (88.6%↑)^s ✓
Granite-4.1-30B llama.cpp · Q4_K_XL	86.8% (80.3%↑) ✗	94.7% (91.7%↑) ✓	91% (83.8%↑) ✗	8.5% (10.5%↓) ✗	93.9% (85.4%↑) ✗
Qwen3.8-27B llama.cpp · Q4_K_XL	92.9% (87.4%↑) ✓	95.3% (92.4%↑) ✓	95% (88.8%↑) ✗	4.1% (5.2%↓) ✓	97.0% (89.6%↑) ✓
Ministral-3-14B vLLM · FP8	86.3% (79.1%↑) ✗	93.1% (89.8%↑) ✗	93% (86.3%↑) ✗	15.5% (17.2%↓) ✗	90.9% (81.6%↑) ✗

Selected: Qwen3.8-27B (llama.cpp, Q4_K_XL) for R1, R2, R4 and R5 — the only model that passed all four. Gemma-4-26B-A4B (vLLM, AWQ INT4) as the verifier (R3) — the only pass from a different model family, so the checker does not share the main model's blind spots.