Epistemic Fine-Tuning of Open-Weight LLMs for Deep Research

Frontier language models (GPT-4o, Claude 3.5 Sonnet, Gemini 2.5 Pro) are trained to be helpful and fluent. In retrieval-augmented generation (RAG) settings — where a model is given source documents and asked to synthesize a grounded response — this training produces a characteristic failure mode: the model generates plausible-sounding claims that are not supported by the provided sources. We ask whether targeted fine-tuning on a smaller open-weight model can produce epistemic behavior that exceeds frontier models on source-grounded generation, without matching their parameter count or training budget.

We fine-tune Qwen3.6-27B VL (an open-weight 27B model) using a five-stage stacked QLoRA pipeline that trains 1.15% of parameters across epistemic foundation, evidence hardening, RAG synthesis, constitutional alignment, and preference optimization. The resulting model — Nehanda v3 — is evaluated on FACTS Grounding (Jacovi et al., 2025), a public benchmark of 860 examples (google/FACTS-grounding-public) requiring long-form responses grounded in provided context documents. Nehanda v3 achieves a factuality score of 88.7% — 763 of 860 examples judged both eligible and grounded — compared to 87.8% for Gemini 2.5 Pro, 83.8% for Claude 3.5 Sonnet, and 79.8% for GPT-4o. On the 816 valid responses (excluding hardware failures), the grounding rate is 93.5%. The result suggests that epistemic behavior — source fidelity, evidence boundary enforcement, refusal to fabricate — is a trainable capability that targeted fine-tuning can install more efficiently than scale alone.

General-purpose language models are optimized for fluency and helpfulness. When deployed in RAG settings — given source documents and asked to produce a grounded synthesis — this optimization produces three characteristic failure modes:

• Fabrication — The model invents sources, dates, and figures not present in the evidence. The output reads fluently and confidently. The citations do not exist.
• Sycophancy — The model agrees with user premises even when those premises are false. Push the model with "just give me the number" and it capitulates.
• Over-hedging — The model refuses to answer even when the evidence is clear, making it useless for decision-making. "I cannot determine" when the source says R0.78/kWh.

These failure modes do not appear on standard accuracy benchmarks, which test single-turn question-answering against world knowledge. They appear in deployment, when a model is asked to read a specific document and say only what that document supports.

Can targeted fine-tuning on a smaller open-weight model produce epistemic behavior that matches or exceeds frontier models on source-grounded generation — without matching their parameter count, training compute, or general capability?

This is a practical question. Frontier models are expensive, proprietary, and not controllable by the deployer. A 27B open-weight model that has been fine-tuned for epistemic soundness can be run locally, modified, audited, and deployed in domains where data sovereignty matters. If the fine-tuning can close the gap on the specific capability that matters (source fidelity), the trade-off — less general capability, more epistemic reliability — may be favorable for deep research applications.

FACTS Grounding leaderboard — full comparison

Scores from the official FACTS Grounding leaderboard on Kaggle and the benchmark paper (Jacovi et al., 2025). Nehanda v3 was evaluated on the same 860 public examples using the same judge prompt template. The leaderboard scores use Google's 3-judge protocol; Nehanda v3 uses a single judge — see Methodology for caveats.

#	Model	Score	Params	Source
1	Nehanda v3	88.7%	27B	Single judge (GLM-5.2)
2	Gemini 2.5 Pro Preview	87.8%	—	Kaggle leaderboard
3	Gemini 2.5 Flash	85.3%	—	Kaggle leaderboard
4	Gemini 2.5 Flash-Lite	84.1%	—	Kaggle leaderboard
5	Gemini 2.0 Flash	83.6%	—	Kaggle leaderboard
6	Claude 3.5 Sonnet	83.8%	—	Paper (Jacovi et al.)
7	Gemini 1.5 Flash	85.8%	—	Paper (Jacovi et al.)
8	GPT-4o	79.8%	—	Paper (Jacovi et al.)
9	Gemma 3 12B	75.8%	12B	Kaggle leaderboard
10	Gemma 3 27B	74.9%	27B	Kaggle leaderboard
11	Claude 3.5 Haiku	75.3%	—	Paper (Jacovi et al.)
12	GPT-4o mini	72.2%	—	Paper (Jacovi et al.)
13	Gemini 3 Pro	70.5%	—	Kaggle leaderboard
14	Gemma 3 4B	70.1%	4B	Kaggle leaderboard
15	OpenAI o1-mini	62.5%	—	Paper (Jacovi et al.)
16	Gemini 3 Flash	61.9%	—	Kaggle leaderboard
17	GLM-5V-Turbo	58.6%	—	Kaggle leaderboard
18	Gemini 3.1 Flash-Lite	40.6%	—	Kaggle leaderboard
19	Gemma 3 1B	36.4%	1B	Kaggle leaderboard

The leaderboard spans 36.4% (Gemma 3 1B) to 87.8% (Gemini 2.5 Pro). Nehanda v3 at 88.7% would rank #1 on the public split — with the methodological caveat that our score uses a single judge while the leaderboard uses a 3-judge majority vote, which is typically more conservative. The llm-stats.com tracker reports an average score of 68.4% across all 13 evaluated models. The most informative comparison is with Gemma 3 27B (74.9%) — the same-size open-weight model from Google, trained with substantially more compute but without epistemic fine-tuning. Nehanda v3 outperforms it by 13.8 percentage points. The gap is attributable to the training pipeline, not the base model: both are 27B, both are open-weight, but only one has been fine-tuned for source fidelity.

Nehanda v3 is a fine-tuned Qwen3.6-27B VL model trained for RAG synthesis. The base model is a native vision-language architecture with a 262,144-token context window and an integrated vision encoder. The fine-tuning pipeline is five-stage stacked QLoRA — all stages are text-only SFT and DPO, leaving the vision weights untouched. The training data consists of energy regulatory documents, intelligence analysis reports, and general-purpose synthesis tasks with a specific emphasis on evidence-grounded generation, inline citation, and refusal to fabricate. Total training cost: approximately $135 of GPU time on a single NVIDIA L40S.

The model was trained with a persona-based prompt schema (SEP-020): {persona}\n\n### Task:\n{input}\n\n### Response:\n. For the FACTS Grounding evaluation, the system instruction from each example was used as the persona, the user request and context document were combined as the task input, and the model generated the response. No few-shot examples were provided. The model was not fine-tuned on any FACTS Grounding data.

The five-stage training pipeline

LoRA adapters (r=64, α=128, 7 target modules covering all attention and MLP projections) are initialized once and trained continuously across all five stages. An eval gate follows each SFT stage — the pipeline halts if the model's epistemic behavior regresses below a threshold. This prevents silent degradation: if Stage 3 training causes the model to forget the premise correction behavior installed in Stage 1, the gate catches it before Stage 4 builds on a broken foundation.

1. Epistemic Foundation (SFT) — Calibrated uncertainty, evidence boundary enforcement, and premise correction. The model learns to say "the document does not specify" when the document does not specify. 1 epoch, lr=2e-4, seq=2048, packing=True. Eval gate: ≥0.70 on held-out epistemic eval set.
2. Epistemic Hardening (SFT) — Evidence weighting, unknown boundary recognition, and correction of overstated user framing. The model learns to push back on false premises in the user's question. 1 epoch, lr=8e-5, seq=2048, packing=True.
3. RAG Synthesis (SFT) — Synthesis of ranked source records into a fact-driven thesis. Inline citation via square brackets. Conflict preservation when sources disagree. This is the stage that directly teaches grounded long-form generation — the capability that FACTS Grounding measures. 2 epochs, lr=2e-5, seq=4096, packing=False.
4. Constitutional SFT (SFT + Replay) — Sycophancy resistance, adversarial hardening, fabrication refusal. The model learns to refuse to generate content not supported by the provided sources. Includes a 0.3 replay buffer ratio from Stages 2 and 3 to prevent catastrophic forgetting, and response masking so loss concentrates on the response, not the input. 2 epochs, lr=2e-5, seq=4096, +183 sycophancy correction rows.
5. Constitutional DPO — Preference optimization: chosen responses maintain source boundaries; rejected responses fabricate or capitulate. This sharpens the boundary between grounded and ungrounded claims. A custom early-stop callback monitors the reward margin and halts training at clean convergence (step 114, margin 3.389), before the margin collapses. 600 max steps, lr=2.7e-7, β=0.1, converged at step 114.

The learning rate decays across stages (2e-4 → 8e-5 → 2e-5 → 2e-5 → 2.7e-7) because each stage builds on an increasingly fragile foundation — large updates late in the pipeline would disrupt the epistemic behavior installed earlier. The DPO learning rate (2.7e-7) was derived from run data: an earlier run with lr=5e-6 collapsed at step 32, so the rate was scaled by the ratio of collapse step to target steps (32/600 ≈ 2.67e-7).

The result is a model that has been explicitly trained to treat the context document as the boundary of what it can say. When the FACTS Grounding judge asks "is every claim in this response supported by the evidence?", Nehanda v3 has been optimized to make the answer "yes" as often as possible. A general-purpose frontier model like GPT-4o or Claude 3.5 Sonnet has broader capabilities but was not specifically trained to refuse to generate content beyond the provided context — which is why they score lower on this benchmark despite being larger and more capable on other tasks.

To validate the approach, we evaluate Nehanda v3 on FACTS Grounding (Jacovi et al., 2025), a public benchmark released by Google DeepMind for measuring the factuality of long-form generation in retrieval-augmented settings. The dataset contains 1,719 examples split across public and private partitions; we use the 860-example public split available on HuggingFace as google/FACTS-grounding-public and on Kaggle. Each example consists of three components:

• System instruction — A task description that tells the model what role to play and what output format to use.
• User request — A natural language question or instruction that the model must respond to.
• Context document — A long text passage (ranging from ~500 to ~50,000 tokens) that contains all the information needed to answer the request. The model must ground its response entirely in this document.

The benchmark evaluates two things: eligibility (does the response actually attempt to answer the question, or does it refuse / produce an empty output?) and grounding (is every claim in the response supported by the context document, or does the response contain unsupported or contradicted claims?). A response must pass both checks to score positively. The official judge prompt asks the judge model to output a single word — "Accurate" or "Inaccurate" — after comparing the response against the context document.

FACTS Grounding is an appropriate validation for this work because it tests exactly the capability the training pipeline targets: source fidelity. The benchmark is domain-agnostic — the model has not seen the specific domains (medical, legislative, financial) in training — so performance reflects the generalizability of the epistemic behavior, not memorized domain knowledge.

Inference was run on a single AWS EC2 g5.2xlarge instance with one NVIDIA A10G GPU (24 GiB VRAM). The model was loaded from HuggingFace (asoba/nehanda-v3-27b) in float16 using the transformers library. Generation parameters were:

• Temperature: 0.0 (greedy decoding, deterministic)
• Max new tokens: 1024
• Max sequence length: 32,768 tokens (covers 96% of examples; the longest 4% were truncated to fit)
• Batch size: 1 (sequential inference due to VRAM constraints)
• Total inference time: ~6.7 hours (24,182 seconds) for all 860 examples

The A10G's 24 GiB VRAM was sufficient for the 27B model in float16 but left limited headroom for long context windows. 39 examples (4.5%) triggered CUDA out-of-memory errors during generation — these were primarily examples with context documents exceeding 20,000 tokens. An additional 5 examples (0.6%) produced empty responses. The remaining 816 examples received substantive model outputs and were carried forward to judging.

The OOM errors are a hardware constraint, not a model failure — a GPU with more VRAM (e.g. A100 80GB) or a more memory-efficient inference engine (vLLM with paged attention) would handle the full 860 examples without truncation. The 5 empty responses appear to be edge cases where the model's generation was cut short by the token budget before producing substantive content.

The official FACTS Grounding judge prompt template (response_level method) was used for all evaluations. The prompt structure is: given a user request, a context document, and a model response, the judge must output "Accurate" if every claim in the response is supported by the context document, or "Inaccurate" if any claim contradicts or cannot be verified from the document.

We used a single LLM judge — GLM-5.2 — running via 55 parallel Devin subagent sessions. Each subagent received a batch of 15 examples with the pre-built judge prompt and was instructed to evaluate eligibility (does the response attempt to answer?) and grounding (is every claim supported?). The subagent wrote structured JSON output with eligible, grounded, score, and reasoning fields for each item.

The official FACTS Grounding protocol uses three judge models with majority voting. We used a single judge due to API cost constraints (the HuggingFace Inference API rate limit was hit partway through the first judge run). This is a methodological caveat — a single judge may be more sensitive to prompt formatting and edge cases than a 3-judge majority vote. The scores reported here should be interpreted as a single-judge evaluation, not a direct apples-to-apples comparison with the 3-judge leaderboard scores. We plan to run the full 3-judge protocol in a follow-up.

Score composition — all 860 examples

Category	Count	Rate	Description
Eligible & Grounded	763	88.7%	Response attempts to answer AND every claim is supported by the context document
Eligible but Inaccurate	40	4.7%	Response attempts to answer but contains at least one unsupported or contradicted claim
Ineligible	13	1.5%	Response is empty, a refusal, or does not attempt to answer the question
OOM (not judged)	39	4.5%	GPU out-of-memory during inference — no response generated
Empty (not judged)	5	0.6%	Model produced no substantive output
Total	860	100%

The 40 "eligible but inaccurate" cases fall into several patterns. The most common is overclaiming — the model adds a detail that is plausible but not explicitly stated in the context document (e.g., claiming a drug causes "genetic changes" when the evidence only lists it as a risk factor). The second pattern is misattribution — the model correctly identifies a fact but attributes it to the wrong section or source within the document. The third is contradiction — the model states the opposite of what the evidence says (e.g., claiming rising interest rates increase prepayment risk, when the evidence says falling rates do). These are the failure modes that the next training cycle will target.

The 13 ineligible responses are primarily refusals where the model stated "the provided sources do not establish that" despite the context document containing the relevant information. This is a calibration issue — the model's fabrication refusal training (Stage 4) is slightly too aggressive in some cases, causing it to refuse questions it could have answered. This is a known trade-off: the same training that prevents hallucination also occasionally prevents legitimate answers.

The result supports the research hypothesis: targeted fine-tuning on a 27B open-weight model can produce epistemic behavior that exceeds frontier models on source-grounded generation. The mechanism is straightforward — the five-stage pipeline trains exactly the capability that FACTS Grounding measures. Frontier models are not trained this way. They are trained for broad capability across many tasks, with safety alignment that does not specifically target source fidelity. On FACTS Grounding, this shows: the frontier models score lower not because they are less capable, but because they were not optimized for this specific epistemic behavior.

The trade-off is explicit. Nehanda v3 sacrifices general capability for epistemic reliability. The model is not trained for creative writing, code generation, or open-ended chat. It is trained to read documents and say what they support. For applications where this is the core capability — regulatory analysis, intelligence assessment, due diligence, academic research — the trade-off is favorable. For applications where general capability matters more, a frontier model is the better choice.

The epistemic behavior generalizes. The FACTS Grounding benchmark is domain-agnostic — the model has not seen the specific domains (medical, legislative, financial) in training. The training data consists of energy regulatory documents and intelligence analysis records. The fact that the model generalizes to unseen domains suggests that the epistemic behavior is domain-independent: the model learns a general discipline of source fidelity, not domain-specific knowledge. This is consistent with the training design — the five-stage pipeline does not teach the model what to think, it teaches the model how to reason with evidence. The domain knowledge is served at inference time via RAG, not baked into the weights.

Implications for open-weight models. If epistemic behavior is trainable at low cost on a 27B model, then open-weight models can be competitive with frontier models on the specific capabilities that matter for deployment in regulated or high-stakes domains — without matching the frontier models' parameter count or training compute. This matters for data sovereignty (the model can be run locally), auditability (the training pipeline is documented and reproducible), and cost (inference is cheaper on a 27B model than on a frontier API).

To make the evaluation reproducible and the failure modes concrete, here are representative examples from each error category. The full per-example results with judge reasoning are available in the evaluation data.

• Overclaiming (ID 342) — The response claimed smoking during pregnancy "can cause genetic changes in the baby." The evidence lists smoking as a risk factor but does not mention genetic changes. The claim is plausible but not supported by the specific context document — the judge correctly marked it Inaccurate.
• Unverified definitions (ID 191) — The response defined "HSE" as "Health Service Executive" and "EMA" as "European Medicines Agency." These acronyms appear in the evidence but are never spelled out. The definitions are real-world correct but cannot be verified from the document alone — under FACTS Grounding rules, this is Inaccurate.
• Contradiction (ID 830) — The response stated that rising interest rates increase prepayment likelihood. The evidence explicitly states the opposite: prepayment risk is associated with falling interest rates. This is a direct factual contradiction.
• Misapplied exemption (ID 65) — The response applied a rooftop solar exemption to ground-mounted solar panels. The exemption in the evidence explicitly covers only rooftop systems. The model generalized a specific exemption to a broader category.
• Refusal despite evidence (ID 200) — The response stated "I cannot determine the answer due to lack of context" despite the evidence containing detailed information about the queried topic. This is an over-triggered fabrication refusal — the model's caution calibration was too conservative for this example.
• Fabricated citation (ID 602) — The response cited a source titled "[The U.S. Economy Is Slowing, But Not Enough to Worry]" that does not appear in the evidence. The core conclusion was correct, but the fabricated citation made the response Inaccurate.

• Single judge, not 3-judge majority. The official FACTS protocol uses three judge models with majority voting. We used a single judge (GLM-5.2). A 3-judge protocol would likely produce a slightly different score — potentially lower, since majority voting is more conservative than a single judge on borderline cases.
• 44 examples not evaluated. 39 OOM errors and 5 empty responses were excluded from judging. The factuality score of 88.7% treats these as failures (score 0). If the OOM examples were run on more capable hardware and the model grounded them at the same 93.5% rate as the valid responses, the score would rise to approximately 90.2%.
• Not a controlled comparison. The frontier model scores are from Google's published leaderboard, computed with a different judge protocol and different inference setup. A true comparison would require running all models on the same hardware with the same judge(s).
• Token budget truncation. The 1024-token output budget may have caused some responses to be truncated mid-sentence, which could affect grounding scores if the truncated portion contained unsupported claims. The input context was truncated to 32,768 tokens for examples exceeding that length, which could cause the model to miss information in the truncated portion.
• Judge model bias. GLM-5.2 as judge may have systematic biases for or against certain response styles. The FACTS Grounding judge prompt is designed to be model-agnostic, but judge bias is a known issue in LLM-as-judge evaluation.

All evaluation code and data are available for reproduction:

• Model: asoba/nehanda-v3-27b on HuggingFace (gated, requires access approval)
• Dataset: google/FACTS-grounding-public on HuggingFace (860 public examples)
• Judge prompts: Official FACTS Grounding judge prompt templates (response_level method, available in the dataset repository)
• Inference: Transformers library, float16, temperature 0.0, max_new_tokens 1024, max_seq_length 32768
• Hardware: AWS EC2 g5.2xlarge (1× NVIDIA A10G, 24 GiB VRAM)
• Judge: GLM-5.2 via Devin subagent sessions, 55 batches of 15 examples each
• Raw responses: 816 model outputs with per-example token counts and timing
• Per-example scores: 816 judged examples with eligibility, grounding verdict, and judge reasoning

The inference script (run_facts_grounding.py), judge script (judge_facts_grounding_api.py), and aggregated scores (facts_grounding_scores.json) are available in the Nehanda repository. The raw responses and per-example judge outputs can be used to verify any individual score or investigate specific failure modes.

• 3-judge protocol — Run the full evaluation with three independent judge models and majority voting, matching the official FACTS Grounding protocol. This will produce a score that is directly comparable to the leaderboard.
• Larger context window — Re-run the 39 OOM examples on an A100 80GB or H100 to eliminate the hardware constraint and evaluate the model on the full 860 examples without truncation.
• Error analysis training data — The 40 "eligible but inaccurate" responses and 13 refusals will be converted into DPO training pairs for the next cycle: chosen responses that stay within the evidence, rejected responses that overclaim or refuse unnecessarily.
• Private split evaluation — Submit to the official FACTS Grounding leaderboard for evaluation on the private split, which uses Google's controlled judging infrastructure.

The model is available at asoba/nehanda-v3-27b. A quantized q4_k_m GGUF is available at asoba/nehanda-rag-synthesis-27b-gguf for local inference via llama.cpp or LM Studio.

If you use this evaluation or the model in your work, cite:

Samudzi, S. (2026). Epistemic Fine-Tuning of Open-Weight LLMs for Deep Research: Nehanda v3 and the FACTS Grounding Benchmark. Asoba Corporation Technical Report. Model: asoba/nehanda-v3-27b.

← Back to Insights