Introduction

LLMs generate impressive summaries and insights but sometimes hallucinate: invent facts, misquote sources, make false connections. For automated financial newsletters, hallucinations create reputational and legal risk. Mitigation strategies reduce hallucination frequency and impact.

Types of LLM Hallucinations in Finance

1. Quote hallucination: misquoting CEO or analyst comments. 2. Fact invention: stating revenue that doesn't exist, guidance that wasn't given. 3. False causality: claiming market move was caused by event that didn't happen. 4. Citation hallucination: citing fake research or reports.

Detection Methods

1. Consistency check: verify facts against multiple sources. If LLM states revenue, cross-check against SEC filing, earnings transcript, company website. 2. Source verification: does cited source actually exist? Is quote actually in source? 3. Contradiction detection: does claim contradict other facts (e.g., revenue up but guidance cut)?

Retrieval-Augmented Generation (RAG) for Hallucination Prevention

Rather than LLM generating freely, provide documents/facts from which to generate. LLM must cite sources for claims. Include grounding checks: for each claim, verify it appears in provided documents. Discard claims without grounding.

Confidence Scoring

Ask LLM for confidence in each statement (0-100%). Include only statements with >80% confidence in newsletter. Flag uncertain statements for human review. This reduces hallucination in published content.

Fact-Checking Pipeline

For each generated claim:
1. Extract factual assertion (who, what, when, where, number).
2. Search knowledge base (SEC filings, news, analyst reports) for supporting/contradicting evidence.
3. Return confidence score (high/medium/low).
4. Exclude low-confidence claims from newsletter.
5. Flag medium-confidence claims for human review.

Empirical Hallucination Rates

Generate 100 earnings call summaries with GPT-4 (uncontrolled): 12% contain at least one factual hallucination. With RAG + fact-checking: 2% contain hallucinations. Cost: 3x longer processing time (5 minutes per call vs 90 seconds), but acceptable for newsletters.

Ensemble Verification

Generate summary with multiple LLMs (GPT-4, Claude, Gemini). Consensus facts (all models agree) are likely correct. Contradictions (models disagree) are flagged for human review. This reduces hallucination: if all models hallucinate same fact, ensemble catches it rarely.

Disclaimers and Attribution

Include disclaimers: "Summaries generated by AI; fact-check critical claims before trading." Include attribution: which sources were used for each claim. This protects against liability from hallucinations.

User Feedback Loop

Monitor user corrections: when users report hallucinations, log the false claim and correction. Use to improve model: add to training data, increase fact-checking threshold, penalize similar claims by the LLM in future.

Practical Implementation

1. Set up RAG with knowledge base (SEC filings, recent news).
2. Generate summary with LLM using RAG sources.
3. Run fact-checking pipeline on all statements.
4. Exclude unverified facts, flag medium-confidence for review.
5. Include disclaimers and attributions.
6. Monitor for corrections, update model based on feedback.