Making AI Accountable in Complaint Handling Systems

The FCA’s Mills Review explored how AI could transform financial services. For complaint management, one of its most relevant themes is the move that’s happening right now: from systems that help people understand to systems that take action on the complaint.

An AI-powered complaint handling system helps teams respond to new information sooner and automatically progress work that currently depends on a person reviewing and initiating each step. Alongside those benefits, the Mills Review began to examine how firms can retain meaningful oversight and explain what happened when AI influences an outcome.

In our recent article, From Understanding to Action: The Next Step in Complaint Management AI, we explored how Complyr connects focused AI assessments directly to configurable complaint workflows. The Mills Review asks us to consider how we track the accountability behind those assessments.

Why automated complaint decisions need to be traceable

The review described how some complaints could become direct interactions purely between customers’ AI agents and firms’ AI agents. While this could lead to improved efficiency, that will only be true if those systems remain reliable and able to systematically escalate complex or sensitive cases to people when appropriate.

As the review notes: “A useful answer is not enough if the basis for it cannot be reconstructed.”

This matters whenever an AI assessment affects how a complaint is handled. If a case is assigned to a particular team, given a different priority or referred for specialist review, the complaint management system needs to record the assessment and any action taken. Firms need to be able to review and understand those records, not least when those decisions are challenged.

How System 1 AI models support complaint workflow automation

At Complyr, our complaint management software combines purpose-built System 1 models with workflows that firms can design, control and update. These models are able to make focused assessments on precisely defined questions; for example, a categorisation assessment selects the relevant category from a defined list of options.

Each assessment returns structured values and probabilities that the workflow can use directly. In our categorisation example, a complaint might be assessed as having probabilities of being related to billing at 93%, service delivery at 5% and other at 2%. The workflow then applies its configured rules to determine whether that assessment can automatically route the complaint or whether it needs human review.

Making AI assessments easier to review and explain

Restricting a model to predefined answers prevents it from inventing (hallucinating) an additional category, but, much like a human, it could still select an incorrect one. This is especially true where probabilities, which are really a measure of how certain the model is, are relatively low.

The practical benefit of the assessments is that the complaint process becomes easier to inspect. Complyr records the context, question, probabilities, confidence, thresholds and resulting decision, creating an audit trail of the assessment and workflow rules that produced each action. A reviewer can examine what information was available, what the model returned and which rule caused the complaint to progress in a particular way.

Humans and AI working together in complaint handling

Consider a customer reporting a duplicate charge after two unsuccessful contacts. A workflow assessment could automatically route the complaint to billing, flag the repeated contacts and initiate a transaction check. Alongside this, another AI model could provide a summary and policy guidance to help the handler understand the complaint.

Not all assessments need to lead to the same level of automation. The category may be clear enough to automate the assignment, but there could be enough uncertainty about the urgency to prompt a human review. Meanwhile, any missing information could trigger an automated request for more evidence, allowing the complaint to move forward while issues requiring human intervention are routed directly to the appropriate person.

This approach also gives human oversight a more specific role. The Mills Review emphasised that firms need to define what reviewers are expected to do, what information they receive and when they can intervene. Within a complaint handling system, those responsibilities can be connected to explicit review points and escalation conditions, giving handlers a clear task and the context needed to carry it out.

Monitoring AI decisions throughout the complaint lifecycle

The same approach can continue throughout the complaint lifecycle. A new message, uploaded document or updated case note can trigger another assessment where required. New information might change the apparent urgency or reveal a need for additional support, so this is a useful step to have in place.

The records created when an AI assessment runs also support the continuous improvement process. Comparing assessments with verified outcomes helps teams identify incorrect classifications, unsuitable thresholds or gaps in the information. Repeated referrals for human review might indicate that the categories need clarification or that a particular decision should always be routed to a human for review. Being able to do this verification process quickly in a single system can be a huge benefit.

Connecting complaint management software to better customer outcomes

For customers, the value lies in how this affects their complaint. Reaching the right team sooner, reducing avoidable delays and ensuring that ambiguity or sensitivity receives appropriate attention all benefit the customer.

The Mills Review highlights the importance of accountability as AI takes on more responsibility. For Complyr, this reinforces the value of connecting focused AI assessments to controlled workflows. Doing so produces fast decisions that help complaints progress promptly, while making sure that workflow rules are enforced with a full audit trail in place to help explain and improve the actions taken.