AI is reshaping finance faster than many organisations can govern it. As AI becomes increasingly embedded in financial reporting, audit, risk and decision making, finance leaders need to understand not only what AI can do, but who remains accountable when things go wrong.
Amid growing excitement about the positive changes that artificial intelligence (AI) brings, Siyanda Mqokoza, chief artificial intelligence officer at Waterfall Chartered Accountants Chambers, gave finance leaders a reality check when he asked who takes the blame when intelligent systems take over complex forecasting, risk modelling, and financial reporting and get them wrong.
Presenting a session titled: The buck stops where? Accountability in an AI-enabled finance, Siyanda conceded that although AI is now an integral part of the finance sector, the industry has not satisfactorily answered the question of who should shoulder the blame and take accountability when AI systems get something wrong and what evidence they would present to exonerate themselves.
Considering the increasing use of AI across various applications in finance, Siyanda presented several fictitious scenarios where finance professionals using the platforms without verification and oversight may get run into a sticky situation.
These scenarios included an AI tool rating a fraudulent transaction as low risk, or when the regulator requests for evidence of the auditability and traceability of AI-assisted reporting tools and when the board asks who is accountable for AI-assisted financial decisions.
“The reality is that we are using these AI platforms but if we were to be pressed by the regulator to demonstrate what we used these applications for, would we have evidence for that? Can we authoritatively tabulate all the AI tools that are used in finance, risk and compliance functions? And how many of these tools went through your approval and change management processes before they went live?” he asked.
He asked delegates whether responsible AI deployment and usage in their organisations is enforced as a technical control or whether it exists as a paragraph in the policy documents that no one reads.
Not all doom and gloom
Siyanda said responsible AI usage should rest on four critical pillars that can withstand regulatory oversight. Explaining these pillars, Siyanda said they are grounded in the Model AI governance framework for agentic AI. Each of the pillars closes a specific gap and together they provide a defensible position on AI usage.
The first pillar is clearly defining the scope of what AI can access and do. This should be informed by the fact that not all the tasks that we do on a day to day carry the same risk profile. Therefore, he said it becomes critical for us to define AI involvement accordingly.
The second pillar speaks to the importance of holding a human accountable and preserving professional scepticism at every critical step. “The true value of our function, whether its finance, risk, or assurance, lies with our professional judgement and accountability. If we are just utilising AI tools without effectively examining them, our integrity and credibility as professionals is put on the line,” said Siyanda.
The third pillar is implementing technical controls enforcing governance by architecture across the full finance life cycle. Siyanda said the final pillar is instilling end user responsibility by equipping your team to oversee AI effectively and not simply use it and accept its output without question.
AI doesn’t substitute human accountability
Siyanda said organisations should enforce rules that whoever signs an AI-generated output owns it. “‘The model said so’ is not a defence in front of an audit committee,” Siyanda said.
He said finance professionals should make it a habit to review the reasoning and not just the result. “They should interrogate the data used, the assumptions made and how the conclusion was reached. If you cannot explain the number, you do not sign the report,” he said.
He stressed that organisations deploying AI systems have a responsibility to ensure that end users have the information, training, and support required to use AI responsibly and to exercise effective oversight.
