When AI Hallucinates the Law: Rethinking Verification and Accountability in AI-Assisted Judicial Research in India

Author

Harsh Gupta is a 2nd-year student pursuing the 5-year LL.B. programme at IIM Rohtak.

A fabricated judgment stops being a technical glitch the moment it enters a courtroom; it becomes a mistake of law.

In July 2026, the Supreme Court of India set aside an insolvency order after finding that the National Company Law Tribunal had relied on six “precedents” that were either non-existent or contained fabricated reasoning generated by artificial intelligence. Weeks earlier, the same Court had already confronted a trial court order built on four hallucinated judgments. Generative AI has quietly become a research assistant to advocates, judges and tribunals across the country, and its growing use has begun to produce more than convenience — it has begun to produce fiction dressed as authority. This is not an argument for banning AI from legal research; the technology is already too embedded, and too useful, for that. It is an argument that India’s response, while real and evolving, has not yet built the institutional machinery needed to catch a hallucination the moment it originates inside a court’s own research, rather than in a lawyer’s brief.

When AI Invents the Law

Generative AI models do not search a legal database the way a citation checker does. They predict the most statistically probable next word based on patterns learned from enormous volumes of text. This means an AI tool can produce a case name, citation, bench composition and paragraph number that read exactly like a real judgment, without any underlying document ever having existed.

India’s Draft Regulations for Use of Artificial Intelligence in Courts, 2026 recognise this directly: AI-generated output is treated as advisory in nature and must be independently verified before anyone relies on it. The danger is not that AI makes mistakes — human researchers make mistakes too. The danger is that AI presents a non-existent authority with exactly the same confidence, formatting and fluency as a real one, so that the fabrication stays invisible until someone actually opens the reporter.

From Gummadi Usha Rani to Pooja Ramesh Singh

The problem surfaced first in Gummadi Usha Rani v. Sure Mallikarjuna Rao. An Andhra Pradesh trial court, ruling on objections to an advocate-commissioner’s report, relied on four judgments that turned out not to exist. On appeal, the High Court accepted that the citations were AI-generated but declined to set the order aside, holding that a hallucinated citation vitiates a decision only where the legal principle actually applied is wrong or wrongly applied to the facts — a materiality-based approach that tolerated the fabrication so long as the underlying reasoning survived without it.

The Supreme Court was less willing to overlook the fabrication. Taking cognisance of the same matter in February 2026, it held that founding a decision on fabricated AI-generated judgments amounts not to a mere error but to misconduct carrying legal consequences, and it issued notice to the Attorney General, the Solicitor General and the Bar Council of India, appointing senior counsel as amicus curiae.

That warning became a concrete ruling five months later. In Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd., the Supreme Court examined citations relied upon by the NCLT and found that some cases did not exist at all, others existed but contained paragraphs that had never been written, and still others were correctly cited yet had fabricated reasoning inserted into them. What distinguishes this case from most AI-citation controversies is where the fabrication originated: the Court recorded that the disputed authorities had not been cited by the respondent’s counsel, but appeared to have entered the judgment through the Tribunal’s own research. The Supreme Court set aside both the NCLT and NCLAT orders and directed that the underlying application be reheard afresh.

What Does Indian Law Currently Provide?

India is not starting from a blank page. On the advocate’s side, Section 35 of the Advocates Act, 1961 already makes professional or other misconduct a ground for disciplinary action, and the Bar Council of India Rules impose an independent duty of candour toward the court; several commentators have argued these provisions are broad enough to reach a lawyer who files a fabricated citation without checking it. Following Pooja Ramesh Singh, the Supreme Court has itself asked the Bar Council to examine the issue formally.

On the institutional side, the Supreme Court’s Centre for Research and Planning published a White Paper on Artificial Intelligence and the Judiciary in November 2025, weighing the efficiency gains AI can offer an overburdened judiciary against the risks of hallucination, loss of confidentiality and erosion of judicial reasoning, and urging that any adoption preserve meaningful human oversight. That thinking has since fed into the Draft Regulations for Use of Artificial Intelligence in Courts, 2026, released by the Supreme Court’s AI Committee for public consultation. The Draft Regulations would permit AI for legal research, citation verification, drafting assistance, translation and case management across the Supreme Court, High Courts, subordinate courts and tribunals, while expressly reserving adjudication, bail decisions and credibility assessments for judges, and requiring that AI output be treated as advisory and independently verified. They also propose dedicated AI Committees, AI Secretariats, and annual technical and ethical audits of deployed systems.

The Missing Link: When the Court’s Own Research Hallucinates

Read together, these developments show that Indian law is not silent on AI hallucination, nor unaware of the need for human verification. What remains underdeveloped is something more specific: a workable institutional mechanism for allocating and recording responsibility when the hallucination originates inside a court’s or tribunal’s own research process, rather than in a party’s pleading. The Draft Regulations state that AI output must be verified, but do not yet specify who within a tribunal is accountable when verification fails, how that verification is to be recorded so it can be audited later, or what remedy a litigant is entitled to once a fabricated authority has already shaped a judgment before anyone notices.

Pooja Ramesh Singh makes this gap unusually visible, because the fabricated citations there did not come from an erring advocate who could be proceeded against under the existing professional-conduct regime — they came from the adjudicating body itself, an actor whose internal research process current law does not yet subject to the same granular scrutiny it is beginning to apply to the Bar. Existing law tells a court what not to do; it does not yet tell a court how to prove, after the fact, that it did the opposite.

Beyond “Human-in-the-Loop”: A Source-Grounded Verification Framework

This article proposes a source-grounded, auditable verification framework resting on four principles.

First, AI may discover but cannot authenticate: an AI tool may locate, summarise and organise material, but its output is a lead for research, never itself a legal authority capable of supporting a judgment. Second, every legally significant AI-suggested authority must be checked against a primary, authoritative source — an official law report, the court’s own record, or the enacted text of a statute — before it enters judicial reasoning. Third, that check should leave a verification record: which tool was used, what it suggested, which source was consulted, who verified it, and when. Such a record does for AI-assisted research what a court’s own docket already does for procedure — it creates an audit trail without which accountability remains theoretical. Fourth, the final legal determination must remain with a human judicial officer; AI can narrow the search, but it cannot decide what the law is.

Zero Tolerance, Proportionate Remedies

This framework also needs a companion principle on remedy. The Supreme Court’s zero-tolerance stance toward fabricated authorities is sound as a rule of verification, but a single remedy — setting the decision aside — need not be the proportionate response to every kind of failure. An AI-suggested error caught before it enters a judgment simply needs correction; a hallucination that appears in the reasoning but does not affect the outcome may call for review and correction rather than a full rehearing; a fabricated authority that materially shapes the actual conclusion, as it apparently did before the NCLT, justifies exactly what the Supreme Court ordered — setting the decision aside; and repeated or reckless reliance on unverified AI output should attract institutional or professional consequences beyond the individual case. Zero tolerance for fabrication can coexist with a proportionate, materiality-based scale of remedies, so that verification failures are treated as seriously as they actually turn out to be, and no more.

Why Not Simply Prohibit AI? The Constitutional Stakes

One might ask why India should not simply prohibit AI from judicial research altogether. The answer is that the technology’s benefits are real and already recognised at the highest level: the Supreme Court’s own White Paper and Draft Regulations treat AI as a legitimate aid to an overburdened judiciary, not a threat to be excluded outright. The better target for regulation is reliance, not assistance.

This carries a constitutional dimension too. Adjudication built on invented authority sits uneasily with the equal and rational application of law that Article 14 demands, and where a decision touches liberty or livelihood, an unverified fabrication raises a fair-procedure concern under Article 21. At bottom, AI hallucination becomes a rule-of-law problem only when a fabricated citation crosses the line from technological output into the actual reasoning of a judgment — which is exactly the line a source-grounded verification framework is designed to hold.

Way Forward

Four steps would operationalise this framework within the ongoing Draft Regulations process: mandatory independent verification of every AI-suggested authority before it is relied upon; a recorded audit trail for AI-assisted research in any judicially significant matter; structured training for judges, law clerks and tribunal researchers in recognising hallucinated material; and a graded remedial framework that ties consequences to the materiality of the error rather than treating every fabrication alike.

Conclusion

Pooja Ramesh Singh should not be read only as a cautionary tale about a careless tribunal. It previews a problem that will recur as AI tools become standard equipment in Indian courtrooms: verification cannot depend on who is embarrassed into checking — it must be built into the process itself. India has rightly chosen not to keep AI out of judicial research, and it should not reverse that choice now. But permission to assist is not permission to be believed. Until an AI-generated authority has been independently traced to a real source by an accountable human, it should carry no more weight in a judgment than a rumour carries in evidence — useful as a lead, worthless as proof.