Ensuring Safety in Financial APIs with LLMs

Motivation

Large Language Models (LLMs) are increasingly used to orchestrate actions in financial APIs: transfer, pay bill, check balance. However, the generative and non‑deterministic nature of LLMs introduces real risks – a model can hallucinate amounts, recipients, or even invent non‑existent transaction IDs.

In this article, I present a production‑tested approach that combines evals‑as‑code (LLM‑as‑a‑judge) + structured guardrails to validate LLM output before any financial API call. I also show how to measure groundedness (adherence to facts) and faithfulness (fidelity to the provided context).

Solution Architecture

The pipeline is divided into three layers:

• Structured extraction: LLM generates a JSON intent (e.g. {"action": "transfer", "amount": 150.00, "destination": "12345-6"}).
• Schema validation: Using Pydantic + Guardrails AI to enforce types and required fields.
• Groundedness evaluation: Checks if the generated information is supported by real context (e.g. bank statement, contact list).
• LLM‑as‑a‑judge: Fallback that classifies the response as “grounded” or “hallucinated”.

Implementation: Groundedness Validator

I used the Guardrails AI library with a custom validator that combines embeddings + LLM‑as‑a‑judge. The code below was used in a real payment authorisation pipeline.

# groundedness_validator.py
from guardrails import Guard
from guardrails.validators import Validator, register_validator
from sentence_transformers import SentenceTransformer, util
import openai

@register_validator(name="groundedness", data_type="string")
class GroundednessValidator(Validator):
    def __init__(self, threshold=0.7, fallback_llm="gpt-3.5-turbo", **kwargs):
        super().__init__(**kwargs)
        self.threshold = threshold
        self.model = SentenceTransformer('all-MiniLM-L6-v2')
        self.fallback_llm = fallback_llm

    def validate(self, value, metadata):
        context = metadata.get("context", "")
        if not context:
            return value  # no context → skip validation

        # Embedding similarity
        emb_value = self.model.encode(value, convert_to_tensor=True)
        emb_ctx = self.model.encode(context, convert_to_tensor=True)
        sim = util.pytorch_cos_sim(emb_value, emb_ctx).item()

        if sim >= self.threshold:
            return value

        # Fallback: LLM-as-judge
        prompt = f"""Real context: {context}
Model generation: {value}
Is the generation fully supported by the context? Answer only 'YES' or 'NO'."""
        response = openai.ChatCompletion.create(
            model=self.fallback_llm,
            messages=[{"role": "user", "content": prompt}],
            temperature=0
        )
        answer = response.choices[0].message.content.strip()
        if answer == "YES":
            return value
        else:
            raise ValueError(f"Groundedness failed. Similarity={sim:.2f}, Judge={answer}")

Example usage with Guard:

guard = Guard.from_string(
    validators=[GroundednessValidator(threshold=0.75, on_fail="exception")]
)
try:
    guard.validate(
        llm_output="Transfer R$ 500 to account 12345",
        metadata={"context": "Available balance: R$ 300. Account 12345 does not exist."}
    )
    # execute payment
except Exception as e:
    # block and request human review
    print(f"Blocked: {e}")

Observability: Tracking Decisions

For production, I integrated the validator with LangSmith and Weave. Each execution logs:

• Original prompt and LLM output
• Similarity score (embedding)
• LLM‑as‑judge decision (YES/NO)
• Validation latency
• Whether the payment was authorised or rejected

This enabled a dashboard with metrics like groundedness rate (by intent type) and false positives/negatives of the judge.

Offline Evals with Ragas

Before every deployment, I run an evals‑as‑code suite using the Ragas library to measure faithfulness and answer_relevancy on a labelled dataset of 500 examples.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

result = evaluate(
    dataset=dataset,  # contains question, answer, contexts
    metrics=[faithfulness, answer_relevancy]
)
print(result)  # e.g. {'faithfulness': 0.92, 'answer_relevancy': 0.88}

When faithfulness drops below 0.85, the CI/CD pipeline fails and requires prompt or validator adjustments.

Results & Lessons Learned

Conclusion

For financial systems, prompt engineering alone is insufficient. The combination of deterministic guardrails (schema) + non‑deterministic evals (groundedness/LLM‑as‑judge) provides the required reliability. The code presented is in production validating over 10,000 transactions per week.

As a next step, I am exploring agentic memory with LangGraph so the financial agent remembers session‑specific rules (e.g., daily limits) without resending the entire context.

References

• Guardrails AI – validation and correction of LLM outputs
• Ragas – evaluation metrics for RAG and groundedness
• Sentence Transformers – embeddings for semantic similarity
• LangSmith and Weave – LLM observability in production