Key Takeaways from the AI Working Group For Company Secretaries

Last week, we attended the AI Working Group For Company Secretaries Open Forum, which explored the rapidly developing relationship between artificial intelligence, corporate decision-making and governance.

The conversation was not just focussed on how companies can use AI more effectively but how we govern organisations in which AI is increasingly influencing the information we receive, the decisions we make and the way businesses operate.

For company secretaries, this raises some particularly important questions around board oversight, accountability, AI literacy and the role of human judgement.

Prompting is more important than it looks

One of the most interesting themes focused on the bias in questions and prompts we ask AI.

A simple demonstration brought this to life. When AI was asked to tell a joke, around 20% of the room received the same joke. This prompted the obvious question: if this is one of the world’s most sophisticated AI models, why are so many people receiving the same answer?

Part of the explanation lies in how language models work. When we submit a query, the model generates multiple possible branches of what could come next, based on statistical relationships and probabilities. There is randomness built into the process, but the output is still shaped by the patterns the model has learned.

Those models are also influenced by human feedback, with people grading and assessing responses and thereby helping determine what the model considers a desirable answer. The result is that AI is not a neutral source of information.

The question we ask, the way we phrase it and the context we provide can all influence the answer. For governance professionals, this is particularly important. If AI is being used to analyse a board paper, summarise a regulatory development or provide recommendations to management, the prompt itself can influence what the decision-maker ultimately sees.

“Learning to prompt” is often described as an AI productivity skill. But for company secretaries and boards, it can become a governance issue.

Consider asking AI to summarise a 100-page document. We may think we are simply saving time. But in reality, we are asking the model to decide what is important enough to include and what can be left out. That is a significant delegation of judgement.

This is why prompts for important work deserve considerably more thought. If AI is being used to summarise board papers, regulatory documents or risk reports, we should ask whether the process of reading and interpreting the original material remains an important control. There is a risk that we don’t just outsource the task of reading – we end up outsourcing the decision about what matters.

AI literacy therefore needs to go beyond knowing how to use a particular model. People need to understand how these systems behave, how outputs can be influenced and where human judgement remains essential. An organisation-wide AI literacy programme should be part of this response.

Beware the AI “yes man”

Another important theme was sycophancy – the tendency of AI models to be overly agreeable and to give users answers they are likely to respond positively to.

AI is increasingly capable of detecting our style, tone and emotional cues. That can be helpful when asking it to adapt a piece of writing, but it becomes more concerning when we are asking AI to assess our ideas or challenge our assumptions.

One example demonstrated how sensitive AI can be to context. An average essay about ChatGPT could be given to another AI model and receive a high score. But if the same essay was described as having been written by a lazy person, the assessment could fall significantly. The underlying work had not changed, only the framing had.

This creates an interesting governance risk. If AI is inclined to make users happy, are we creating systems that reinforce the views of those already in the room rather than challenging them?

Boards need to consider whether they are using AI as a genuine source of challenge – or as a sophisticated validation tool.

The risk of social atrophy

The Working Group also considered some of the less obvious consequences of increasingly sophisticated AI, including the risk of social atrophy.

If we increasingly interact with systems designed to be responsive, agreeable and available whenever we need them, what happens to our ability to form meaningful relationships with other people? Human relationships involve disagreement, compromise, uncertainty and friction. AI can remove much of that friction.

There is therefore a wider question about what capabilities we could lose if we increasingly delegate interactions, thinking and even elements of decision-making to machines.

For organisations, this is worth considering alongside the obvious productivity benefits of AI.

Is the board fit for purpose?

Perhaps the biggest governance question is whether boards are fit for purpose in an AI-enabled organisation.

Boards are increasingly overseeing companies where technology is no longer just supporting the business. In many organisations, technology is effectively the operating system of the business. That changes the nature of board oversight.

The discussion around Section 172 of the Companies Act was particularly interesting in this context. If directors have duties to promote the success of the company while considering wider stakeholder interests, how should those duties operate when AI increasingly influences corporate decisions? Is the existing framework still relevant? Or do we need to rethink the role and responsibilities of companies in a world where technology can make or influence decisions at scale?

Who is accountable when AI goes wrong?

AI also raises difficult questions around outsourcing and liability. If a company outsources an AI-enabled process and the AI makes a damaging decision, who is responsible?

Is it the AI provider? The outsourced service provider? The company using the system? Or ultimately the board responsible for overseeing the organisation?

Outsourcing the technology does not necessarily mean outsourcing accountability. This makes due diligence, contractual arrangements, oversight and assurance increasingly important when companies introduce AI through third parties.

Human judgement isn’t disappearing

One of the most interesting observations was that judgemental decisions don’t necessarily disappear as AI becomes more capable – they may actually increase.

If AI handles more routine decisions, the decisions left for humans may increasingly be those that are complex, sensitive or consequential. If more decisions are escalated upwards because AI is dealing with everything below them, could boards become overwhelmed by the volume and complexity of decisions reaching them?

Boards need AI literacy

A clear practical takeaway was the need for greater AI literacy at board level.

Many board members are unlikely to have sufficient understanding of AI to effectively challenge proposals involving it. As a result, responsibility for understanding AI can fall disproportionately on the company secretary, technology team or a small number of specialists.

There needs to be much greater exposure across the board. AI literacy should be considered as part of a board’s skills matrix, alongside financial, legal, regulatory, sector and technology expertise.

This does not mean every director needs to become an AI expert. It means directors need enough understanding to ask the right questions:

• What is the AI actually doing?
• What data is it using?
• What assumptions does it make?
• How could the output be biased?
• How has it been tested?
• Who checks the output?
• What happens when it is wrong?
• Who is accountable for the resulting decision?
• What decisions should never be delegated to AI?

For the foreseeable future, there will also need to be human specialists checking AI outputs, particularly where decisions carry significant legal, financial, regulatory or reputational consequences.

The changing role of the company secretary

This potentially creates a significant opportunity for company secretaries.

The company secretary sits at the intersection of the board, governance, regulation, risk and organisational processes. As AI becomes embedded in decision-making, company secretaries may increasingly need to understand not only what decision was made, but how AI contributed to that decision.

This could mean asking new questions when preparing board agendas and papers:

• Was AI used in preparing this analysis?
• Was the underlying information independently verified?
• Has management challenged the AI output?
• What human judgement has been applied?
• Does the board need specialist AI advice?
• Should the use of AI be recorded as part of the decision-making process?

This is an important evolution of the company secretary’s role: helping ensure that AI becomes part of a properly governed decision-making process rather than an invisible influence on board decisions.

Should AI use appear in board minutes?

One practical suggestion was that board minutes should record when AI has materially informed a decision. This could become an important part of the governance and audit trail.

If AI has generated an analysis, recommendation, forecast or summary that materially influenced the board’s decision, recording that fact provides greater transparency around how the decision was reached. It also reinforces an important principle: AI can inform a decision, but it does not remove the board’s responsibility for that decision.

From periodic audit to continuous assurance

Another implication is the changing nature of audit. Traditional governance has often relied on periodic checks: a system is reviewed, an audit takes place, controls are assessed and recommendations are made.

AI-enabled organisations may require something closer to continuous, live assurance. If AI systems are constantly processing information and influencing decisions, a periodic audit only provides a snapshot.

This could mean a shift towards continuous monitoring of systems, outputs, controls and exceptions – and will require new skills and potentially new approaches to internal audit and risk management.

Do we need to rethink the role of the company?

The discussion ultimately went beyond AI itself. It raised a broader question about what a company is for.

An example from Japan illustrated how companies can have a much broader social role. Some companies have historically provided employees with healthcare, sports facilities and other forms of support, reflecting a model in which the company can form part of a wider social safety net.

AI could force us to revisit these questions. If AI fundamentally changes productivity, employment and the relationship between people and organisations, should we rethink the role and responsibilities of companies? Do we need a new Companies Act? Or, more fundamentally, do we need to rethink what we expect companies to contribute to society?

Key takeaway

The biggest lesson from the AI Working Group For Company Secretaries was that AI governance is ultimately about human governance.

The technology may be new, but the fundamental principles remain familiar: accountability, transparency, oversight, challenge, judgement and responsibility.

What is changing is the environment in which those principles have to operate. Boards are now overseeing organisations where technology can increasingly influence – and potentially make – decisions. That means AI literacy can no longer sit solely within the technology function.

For company secretaries in particular, this creates an opportunity to play an increasingly important role in helping boards navigate this transition.

The question is no longer:

“How do we use AI?”

It is:

“How do we govern an organisation where AI is part of how decisions are made?”

That is a much bigger question – and one that company secretaries and boards need to be engaging with now.