---
license: mit
tags:
- financial-services
- evaluation
- determinism
- regulatory-compliance
- sec-filings
- small-language-models
- llm
---
# LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows

**Authors:** Raffi Khatchadourian, Rolando Franco  
**Venue:** AI4F @ ACM ICAIF 2025 (Nov 15 in Singapore)

**Paper:** https://arxiv.org/abs/2511.07585  
**Code:** https://github.com/ibm-client-engineering/output-drift-financial-llms

This repository is a Hugging Face landing page for the paper and its open-source implementation.
It focuses on deterministic test harnesses, cross-provider validation, and risk-tiered deployment
for financial LLM workflows (SEC 10-Ks, RAG over filings, JSON/SQL tasks).

## 🔑 Key finding

> Well-engineered **7–8B models achieve 100% output consistency at T=0.0**, while a 120B model reaches only **12.5% consistency**, regardless of configuration.

Across **480 runs** (5 models, 3 tasks, 2 temperatures, 3 concurrency levels), we show an
inverse relationship between model size and determinism and map this to regulatory
requirements (FSB, BIS, CFTC).

---

## 📊 Model tier classification

| Tier | Models                      | Consistency @ T=0.0 | Status             | Recommended use |
|------|-----------------------------|---------------------|--------------------|-----------------|
| **1** | Granite-3-8B, Qwen2.5-7B   | **100%**            | ✅ Production-ready | All regulated tasks |
| **2** | Llama-3.3-70B, Mistral-Medium-2505 | 56–100%  | ⚠️ Task-specific     | SQL / structured only |
| **3** | GPT-OSS-120B               | **12.5%**           | ❌ Non-compliant    | Not for compliance |

*n = 480 runs (16 per condition), 95% Wilson CIs, p < 0.0001 (Fisher’s exact).*

---

## 🎯 Why this matters

Financial institutions face a **“verification tax”**: human review erodes AI productivity
gains when outputs are nondeterministic.

This framework shows:

- **Audit-ready determinism is achievable** with the right model + decoding setup.
- **Cross-provider consistency**: behavior transfers between local (Ollama) and cloud (IBM watsonx.ai).
- **Task-specific drift**: SQL and structured summaries remain stable even at T=0.2; RAG is far more sensitive.