|
|
--- |
|
|
license: mit |
|
|
tags: |
|
|
- financial-services |
|
|
- evaluation |
|
|
- determinism |
|
|
- regulatory-compliance |
|
|
- sec-filings |
|
|
- small-language-models |
|
|
- llm |
|
|
--- |
|
|
# LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
|
|
|
|
|
**Authors:** Raffi Khatchadourian, Rolando Franco |
|
|
**Venue:** AI4F @ ACM ICAIF 2025 (Nov 15 in Singapore) |
|
|
|
|
|
**Paper:** https://arxiv.org/abs/2511.07585 |
|
|
**Code:** https://github.com/ibm-client-engineering/output-drift-financial-llms |
|
|
|
|
|
This repository is a Hugging Face landing page for the paper and its open-source implementation. |
|
|
It focuses on deterministic test harnesses, cross-provider validation, and risk-tiered deployment |
|
|
for financial LLM workflows (SEC 10-Ks, RAG over filings, JSON/SQL tasks). |
|
|
|
|
|
## 🔑 Key finding |
|
|
|
|
|
> Well-engineered **7–8B models achieve 100% output consistency at T=0.0**, while a 120B model reaches only **12.5% consistency**, regardless of configuration. |
|
|
|
|
|
Across **480 runs** (5 models, 3 tasks, 2 temperatures, 3 concurrency levels), we show an |
|
|
inverse relationship between model size and determinism and map this to regulatory |
|
|
requirements (FSB, BIS, CFTC). |
|
|
|
|
|
--- |
|
|
|
|
|
## 📊 Model tier classification |
|
|
|
|
|
| Tier | Models | Consistency @ T=0.0 | Status | Recommended use | |
|
|
|------|-----------------------------|---------------------|--------------------|-----------------| |
|
|
| **1** | Granite-3-8B, Qwen2.5-7B | **100%** | ✅ Production-ready | All regulated tasks | |
|
|
| **2** | Llama-3.3-70B, Mistral-Medium-2505 | 56–100% | ⚠️ Task-specific | SQL / structured only | |
|
|
| **3** | GPT-OSS-120B | **12.5%** | ❌ Non-compliant | Not for compliance | |
|
|
|
|
|
*n = 480 runs (16 per condition), 95% Wilson CIs, p < 0.0001 (Fisher’s exact).* |
|
|
|
|
|
--- |
|
|
|
|
|
## 🎯 Why this matters |
|
|
|
|
|
Financial institutions face a **“verification tax”**: human review erodes AI productivity |
|
|
gains when outputs are nondeterministic. |
|
|
|
|
|
This framework shows: |
|
|
|
|
|
- **Audit-ready determinism is achievable** with the right model + decoding setup. |
|
|
- **Cross-provider consistency**: behavior transfers between local (Ollama) and cloud (IBM watsonx.ai). |
|
|
- **Task-specific drift**: SQL and structured summaries remain stable even at T=0.2; RAG is far more sensitive. |
|
|
|
|
|
|
