Don’t get caught out by AI hallucinations

This case is all about one of our top firms of solicitors getting into trouble when using AI. But wait: I particularly want to coax my very enthusiastic QS friends who act as advocates in adjudication and arbitration to sit up and take note of what happened, because you may well mutter, “There, but for the grace of God, go I.” Adjudication, in particular, sees non-lawyer advocates nowadays arguing their clients’ cases, and some of you are very good at it… some not!

The case is Cork & Anor vs Smith [22 May 2026] EWHC 1199 (Ch). It is all about “hallucinated” AI content. The solicitors’ job here was to make an application to the court about an insolvency matter. It was merely administrative and uncontentious. The task was placed in the hands of a recently qualified solicitor, who put the papers into court. No problem with all that.

The judge spotted a snag and sent a question to the solicitors, asking: “What power is there to order release?” Promptly the solicitor replied by letter. It all looked sound. But the judge was very familiar with this insolvency territory, and the answers came as a surprise to him, he said. He did what some (only some) adjudicators do: he checked out the rules for himself. The rules said nothing of the sort. Nothing rang true.

The judge said, “Inevitably, my first thought was that this was an artificial intelligence (‘AI’) ‘hallucination’ that arose during [the solicitors’] research.” He went on, “‘Hallucination’ is a term used where AI generates false or misleading information, which it presents as fact.” So the judge asked in another letter, where did the information come from?

The firm of solicitors attempted to clarify what was meant and apologised for the confusion. But by now there was a very unhappy judge. “It struck me as likely to be an AI hallucination, which had not been checked,” he said. So, he ordered a hearing and disclosure of the use of AI.

Non-lawyer advocates are sometimes a real hazard… now we have an artificial contributor called AI – and those of us who decide cases are fretting about this new kid on the block

Seemingly, the junior solicitor began “chatting” with AI and eventually the chats ran to 59 pages. The judge said, “What is remarkable when reading these chats is, first, that one would think that the text attributed to the AI was produced by an intelligent human being.” I have looked at some of this chat myself. For all the world, this AI is talking back and forth as though it is a fellow lawyer, speaking very knowledgeably about the dilemma or principles. It looks very, very good.

The judge continued, “The second remarkable thing is that, on a large number of occasions, it is plainly wrong or, at the very least, extremely misleading”. He did acknowledge that the junior solicitor’s superiors had no intention to mislead, however, and indeed were unaware of the use of AI on this case. The firm has apologised.

Now let’s draw breath. Our adversarial system has, at base, a very important case principle; it is that the tribunal, whether judge, arbitrator or adjudicator, is owed a high-level obligation. It depends totally on being able to rely without question on the integrity of those who appear before it as representative and on their professionalism in only making submissions which can properly be supported. 

The system for many a long year gave that responsibility to barristers-at-law. Only they had rights of audience. That privileged position was eased in recent time to give rights of audience to solicitors as well. On occasions a litigant-in-person is given the court’s permission to represent himself or herself, or to use a non-lawyer friend. Great care is taken by a judge when giving such permission.

The reasoning behind all this is that the court is reliant on the advocate. Non-lawyer advocates are sometimes a real hazard to themselves, as well as to the person deciding the case. Do you follow? Now we have an artificial contributor called AI – and those of us who decide cases are fretting about this new kid on the block.

AI is a tool that carries with it risks as well as opportunities. Its use must take place therefore with an appropriate degree of oversight and within a regulatory framework that ensures compliance with well-established professional and ethical standards, if public confidence in the administration of justice is to be maintained.

But let me just pause again to reflect on all that. There is a regulatory framework of solicitors and also barristers. There is none, in regard to AI, for non-lawyers or consultants.

The duty on a solicitor, and so too a barrister, is to cross-check the commentary delivered by an AI chat-box or research. If the chat points to a case or even merely a legal principle, it has to be cross-checked for hallucination – not only that the case exists, but that it is relevant to the case at hand, rather than being made up or imagined.

Well now, the same has to apply to non-lawyers and consultants. If AI is used, then we will have to require a declaration signing off that the material has been checked for authenticity; and oust the fake nonsense.

One thing struck me: what responsibility does the AI producer have when wrong information is published to its customers on its AI platform? Their approach is to label all the AI “stuff” as a developing science or tool that is getting better but not perfect. They rely on the “human in the loop” (meaning the user, you or me) to take the risk of wrong information coming out and to check. So, use it and check it is the rule of thumb.

Tony Bingham is a barrister and arbitrator at 3 Paper Buildings, Temple