Author
Arpita Anand is a 4th-year BBA LL.B. (Hons.) student at Maharishi Markandeshwar (Deemed to be University), Mullana–Ambala.
Introduction
Artificial Intelligence (AI) is rapidly changing the way knowledge is created, accessed and used. The legal profession is no exception. Lawyers increasingly use AI tools for legal research, summarising documents, identifying relevant authorities and assisting with drafting. Courts and tribunals are also exploring technology to improve efficiency and reduce the burden of repetitive work. Yet, as AI becomes more capable, a fundamental legal question emerges: how far can the justice system rely on a technology that can produce convincing but completely false information?
This question has moved from theoretical discussion to an actual judicial concern in India. In Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., decided on 2 July 2026, the Supreme Court dealt with a case in which the National Company Law Tribunal had relied upon non-existent and AI-generated legal material while deciding a matter. The Supreme Court subsequently set aside the decisions of the NCLT and NCLAT and emphasised the need for human control and verification when AI is used in adjudication.¹
The case presents a larger issue extending beyond one incorrect citation. If technology can generate fabricated judgments, paragraphs and authorities that appear legally authentic, what safeguards are required to preserve the integrity of judicial decision-making? This article examines India’s emerging approach to AI in the legal system, the dangers posed by AI hallucinations, the existing legal framework, and the need for responsible human oversight.
AI and the Changing Nature of Legal Research
Legal research has traditionally depended upon primary sources such as statutes, judicial decisions, regulations and official reports. Digital databases have already transformed this process by making thousands of authorities searchable within seconds. AI represents the next stage of this transformation because it can not only retrieve information but also generate summaries, explanations and apparently reasoned answers.
The difficulty is that generative AI does not necessarily distinguish between what is legally correct and what merely appears plausible. An AI system may produce a citation that looks genuine, attribute a paragraph to a real judgment that never contained it, or construct an entirely fictional case. This phenomenon is commonly described as an AI hallucination.
In ordinary conversation, an incorrect AI-generated statement may simply cause confusion. In legal proceedings, however, the consequences can be considerably more serious. A fabricated precedent may influence an argument, affect a judicial decision, consume judicial time and ultimately affect the rights of parties.
The legal profession therefore faces a unique problem: the more convincing AI becomes, the more difficult it may be for an unverified output to appear obviously false.
The Supreme Court’s 2026 Intervention
The significance of the issue became particularly clear in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., 2026 INSC 668.
The dispute arose in the context of insolvency proceedings. The NCLT had relied upon six purported judgments while deciding the matter. During proceedings before the Supreme Court, it became apparent that the authorities were problematic. The Supreme Court found that some citations were entirely non-existent, while other citations referred to genuine judgments but were accompanied by paragraphs that could not be found in those judgments. The Court also noted that the material had escaped scrutiny at the appellate stage.²
The Supreme Court set aside the NCLT and NCLAT decisions and restored the matter for fresh consideration. More importantly, the Court addressed the broader relationship between AI and adjudication.
The Court recognised that AI can assist with both routine and intellectual tasks but stressed that adjudication cannot be surrendered to technology. It emphasised that human control must remain at every stage where AI is used in the judicial process. The Court further adopted a strict position against the citation.
Opportunities don’t wait. Neither should you.
Join 1 Lakh+ law students connected with Lexibal and stay updated with internships, opportunities, competitions and important updates.
Join WhatsApp ChannelExisting Legal Framework: Is It Enough?
India’s legal system already recognises the importance of authenticity and reliability in legal proceedings. The Bharatiya Sakshya Adhiniyam, 2023 (BSA) provides a contemporary framework for electronic and digital evidence and came into force on 1 July 2024.³ Its provisions recognise electronic records within the law of evidence and prescribe conditions and certification requirements for specified electronic records.⁴
The framework is important because AI-generated content creates questions not only about whether something is digital, but also about whether the content can be trusted and how its origin can be established.
However, AI-generated legal research presents a somewhat different problem. A fabricated case cited by a lawyer may not necessarily be “evidence” in the conventional sense. It is an assertion about the existence and content of law. Consequently, traditional evidentiary safeguards cannot alone solve every problem created by generative AI.
Professional responsibility therefore becomes equally important. An advocate’s duty is not fulfilled merely by obtaining information from a technologically sophisticated source. The advocate remains responsible for ensuring that the authority actually exists and supports the proposition for which it is cited.
The Supreme Court’s 2026 decision reinforces precisely this principle.
The Human-in-the-Loop Principle
The most important principle emerging from the recent judicial discussion is that AI should remain assistive rather than authoritative.
AI can potentially help lawyers locate relevant cases, identify issues, organise large documents and generate preliminary research questions. These functions may save time and allow legal professionals to devote greater attention to analysis.
But the final responsibility must remain with a human legal professional.
A Practical Model Could Involve Three Stages
First, AI-assisted discovery: AI may be used to identify potentially relevant cases, statutes or legal issues.
Second, authoritative verification: Every authority identified by AI should be checked against an authoritative source, such as the official court database, recognised legal database or statutory text.
Third, human legal judgment: The lawyer or judge must independently determine whether the authority is relevant, current and applicable to the facts.
This model preserves the efficiency of AI without transferring legal responsibility to an algorithm.
The Need for Clear Professional Standards
The Supreme Court has already directed attention towards the need for institutional guidelines. However, a broader and more detailed framework may be necessary as AI use becomes routine.
Professional guidance should clearly address at least four questions.
1. Disclosure
Should advocates disclose material reliance on generative AI in legal research or drafting in specified circumstances?
2. Verification
Should professional rules expressly require advocates to verify every AI-generated authority before placing it before a court?
3. Confidentiality
Lawyers must be cautious about entering confidential client information into AI systems, particularly where the data may be processed or stored by external providers.
4. Accountability
Responsibility for an inaccurate filing should remain with the human professional who submits it. AI should not become a shield against professional responsibility.
Such standards would not require courts to prohibit AI. Instead, they would establish boundaries within which the technology can be safely used.
Beyond Hallucinations: The Larger Question of Judicial Independence
AI in courts raises a deeper issue than accuracy. Judicial decision-making involves interpretation, context, constitutional values and the application of law to particular facts. These are not simply information-retrieval exercises.
An AI system may identify patterns from previous decisions, but legal reasoning sometimes requires distinguishing a precedent rather than following the apparent majority pattern.
Constitutional adjudication may also involve balancing competing rights and values.
The danger, therefore, is not merely that AI may provide a wrong answer. It is that excessive dependence on automated reasoning could gradually influence how legal questions are framed and understood.
The Supreme Court’s insistence upon human control is significant in this context. It preserves the principle that technology may support adjudication but cannot become its independent decision-maker.
The Way Forward
India does not need to choose between technological innovation and judicial integrity. The more useful approach is to establish a framework in which the two operate together.
Courts and legal institutions should consider developing verified AI-assisted research systems trained or connected to reliable legal databases. Lawyers and judges should receive basic AI-literacy training, particularly concerning hallucinations, source verification, confidentiality and limitations of generative models.
Courts could also encourage the use of authoritative digital repositories for citation verification.
Law schools should introduce AI and legal ethics within legal research training so that future lawyers understand not only how to use AI but also when not to trust its output.
Most importantly, legal institutions should maintain a clear distinction between automation of legal work and automation of legal responsibility. The former may improve efficiency; the latter raises serious questions about accountability.
Conclusion
Artificial Intelligence is likely to become an increasingly important part of India’s legal ecosystem. The question is no longer whether lawyers and courts will encounter AI, but how responsibly they will use it.
The Supreme Court’s decision in Pooja Ramesh Singh provides an important starting point. It demonstrates that technological assistance cannot replace verification, professional responsibility or human judicial control. A fabricated precedent may be generated in seconds, but its consequences can extend far beyond the moment of generation.
The emerging legal principle should therefore be simple: AI may assist legal reasoning, but it cannot authenticate itself.
India’s challenge is not to resist technological change but to regulate its use in a manner consistent with the rule of law. The future of AI in the justice system should be built on a partnership between technological efficiency and human accountability. In an age where machines can generate persuasive legal language, the responsibility to determine what is actually law must remain firmly with humans.
References
- Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., 2026 INSC 668 (Supreme Court of India, July 2, 2026).
- Supreme Court of India, Landmark Judgment Summary: Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., 2026 INSC 668.
- Bharatiya Sakshya Adhiniyam, 2023, Act No. 47 of 2023.
- Bharatiya Nagarik Suraksha Sanhita, 2023, Act No. 46 of 2023.
- Justice K.S. Puttaswamy (Retd.) v. Union of India, (2017) 10 SCC 1.
- Ministry of Electronics and Information Technology, Government of India, Digital Personal Data Protection Rules, 2025 (notified Nov. 14, 2025).
- Ministry of Electronics and Information Technology, Government of India, Digital Personal Data Protection Act, 2023.
