⚖️ Manufacturing Justice: The Supreme Court on AI Use in Courts

Thesis

Technology may accelerate justice, but it can never author it; the integrity of adjudication rests on human reasoning that no machine can outsource.

Opening Hook

"A citation that never existed can still convict a man who does." The Supreme Court's comparison of AI hallucinations to methyl isocyanate reframes a technical glitch as an existential threat to the rule of law.

Body Structure

• Part I: The analogy — Bhopal 1984 as a parable of invisible, delayed catastrophe.
• Part II: The crisis — hallucinated citations from trial courts to tribunals.
• Part III: The response — "misconduct not error," draft rules, BCI norms.
• Part IV: The principle — human primacy and rigorous verification.

Counterargument

"AI can democratise access to justice." Concede the gain — then show it is realised only within guardrails that keep human judgment at the centre.

Conclusion

The invisible must be made visible through regulation. Justice seen to be done is justice grounded in truth, not fabrication.

Thesis

AI is simultaneously an engine of capacity and a source of error; in public institutions, its promise is inseparable from its peril.

Opening Hook

"The most dangerous falsehoods are the ones spoken with perfect confidence." Generative AI is fluent precisely where it is fabricating.

Body Structure

• The dual nature of AI — from Babbage's engine to large language models.
• Sectoral applications: judiciary, administration, e-governance, welfare targeting.
• Risks: bias, opacity, and the accountability deficit when machines "decide."
• Regulatory design: human-in-the-loop, transparency, independent audit.

Conclusion

Governance by humans, assisted by machines — never the reverse — is the settlement a democracy must insist upon.

Thesis

Efficiency purchased at the cost of integrity is a false economy; institutions endure only when speed serves trust.

Opening Hook

"A backlog of five crore cases is a crisis; a judgment built on a lie is a catastrophe." India's courts face both temptations at once.

Body Structure

• The efficiency argument: pendency, the eCourts journey, assistive AI.
• The integrity counter: the NCLT hallucination case; verification burden.
• The trade-off: where to draw the assist-versus-decide line.
• Regulatory design: risk-tiered verification, capacity building, calibrated incentives.

Conclusion

The wise institution automates the routine and reserves judgment for the human — protecting both throughput and trust.

Thesis

Accountability cannot be delegated to an algorithm; when AI fails, responsibility must still rest with an identifiable human.

Opening Hook

"When the machine errs, who apologises to the wronged?" The question exposes the accountability gap at the heart of automated decision-making.

Body Structure

• From automation to autonomous systems — the shifting locus of responsibility.
• Legal frameworks: the "misconduct" standard, the BCI directive, draft rules.
• Ethical dimensions: automation bias, opacity, the duty to give reasons.
• Institutional design: human primacy, independent audit, an AI register.

Conclusion

The buck stops with the human, not the machine — accountability is the price of using powerful tools.

Thesis

The gravest risks are the invisible ones; systemic threats to justice must be named and regulated before they detonate.

Opening Hook

"In Bhopal, the gas was odourless until it was lethal." The Court's analogy warns that AI's harm to law will be felt only when it is already done.

Body Structure

• The analogy explored — invisibility, delay, scale.
• Bhopal (1984) as a cautionary tale of regulatory failure.
• Contemporary crisis: hallucinations, automation bias, regulatory gaps.
• The response: zero tolerance, transparency, international cooperation.

Conclusion

Only vigilance turns the invisible visible — and turns a warning into a safeguard.

The analogy references the 1984 Bhopal gas tragedy, where a methyl isocyanate leak caused mass casualties. By comparing AI-generated fake judgments to that gas, the Court captured three qualities of the threat: it is invisible (fabricated citations look authentic), insidious (it spreads silently through pleadings and orders), and catastrophic by the time it is noticed (a wrongful decision may already stand).

Its significance is rhetorical and doctrinal. Rhetorically, it elevates a seemingly technical error to an existential risk for the rule of law. Doctrinally, it justifies a zero-tolerance standard — that any judgment tainted by hallucinated material is "no decision in the eyes of law." It is a memorable framing that communicates urgency to judges, lawyers and the public alike.

The Supreme Court set aside the orders of the NCLT and the NCLAT in the Essel Infraprojects insolvency matter, after the NCLT relied on fictitious AI-generated citations and the NCLAT failed to catch the error. The ruling came from the Bench of Justices P.S. Narasimha and Alok Aradhe.

The Supreme Court AI Committee is chaired by Justice P.S. Narasimha — not the CJI — and was reconstituted by CJI Surya Kant in December 2025. Its mandate is to steer the adoption, development and deployment of AI across the higher and subordinate judiciary; it authored the draft "Regulations for Use of AI in Courts, 2026." Keeping the chair distinct from the CJI is a common factual slip worth avoiding.

I would apply a simple test: does the AI output touch the substantive rights of parties or the exercise of judgment? If yes, it is impermissible; if it merely supports the process, it is assistive. On the permissible side sit translation (PANINI), transcription (ASR-SHRUTI), legal research (LegRAA) and document or case management — tasks that improve access and efficiency without deciding anything.

On the impermissible side sit adjudication, sentencing, bail eligibility, credibility assessment and risk scoring — the very functions the draft rules prohibit, because they involve discretion, liberty and dignity. Crucially, even assistive outputs must be independently verified; a research suggestion is a lead to check, never an authority to cite. The draft rules operationalise this line, and Clause 8's "human primacy" ensures accountability always rests with the human officer.

"Error" implies an inadvertent, good-faith mistake attracting lesser consequences; "misconduct" implies a professional or ethical breach that invites disciplinary action. By choosing the latter, the Court held that citing fabricated precedents is not a pardonable slip but a failure of the basic duty to verify — one that strikes at adjudicatory integrity and the rule of law.

The distinction matters because it elevates the accountability standard for both judges and advocates. For lawyers, it opens the door to proceedings under Section 35 of the Advocates Act, 1961, and underpins the SC's direction to the Bar Council of India to frame norms. It sends a deterrent signal: verification is a non-negotiable professional obligation in the AI era, not a courtesy.

Automation bias is the well-documented human tendency to over-trust machine-generated outputs, especially when they are fluent and confident. In a judicial setting, a judge or clerk may accept an AI-suggested citation or summary without the scrutiny they would apply to a junior's draft — effectively ceding a slice of independent reasoning to the tool.

This is dangerous precisely because AI hallucinations are persuasive: fabricated cases arrive with plausible names, numbers and holdings. Left unchecked, automation bias erodes the core of adjudication — the independent application of mind. The counter-measures are procedural and cultural: mandatory verification of every AI-derived authority, AI-literacy training, and a workplace norm that treats AI output as a starting point to be tested, never an endpoint to be trusted.

My first duty is to the court and the integrity of the proceeding. I would privately and promptly alert the colleague, giving them the chance to correct the record by withdrawing the citation and informing the court — the honourable and least damaging course.

If the colleague refuses and the false citation risks misleading the court, my obligation to the administration of justice overrides collegiality: I would bring it to the court's notice and, if warranted, report the matter to the Bar Council under Section 35 of the Advocates Act. The Supreme Court has characterised such conduct as "professional misconduct." Throughout, I would act factually and without malice — the goal is to protect the litigant and the court, not to punish a peer.

For: such tools can rapidly surface leads across vast material, assist with translation and drafting, and help clear backlog — a real public good in an overburdened system. Used well, they widen access and save time for substantive reasoning.

Against: they hallucinate confidently, invite automation bias, are often opaque, and may carry biases from their training data — risks that are unacceptable in adjudication. My balanced position: permit them strictly as assistive research aids, subject to mandatory independent verification of every output, a firm prohibition on AI in adjudication, sentencing and bail, and compulsory AI-literacy training. In short, the tool may help the judge think, but it must never think for the judge.

I would combine correction with prevention. First, issue a practice circular mandating that all citations be verified against official repositories before filing, and require counsel to certify that AI-assisted drafts have been independently checked. Second, in individual cases, cross-verify suspect citations (case numbers, party names, holdings) and reject or return non-verifiable filings, recording reasons.

Third, refer persistent or wilful offenders to the State Bar Council for disciplinary consideration, consistent with the Supreme Court's misconduct standard. Fourth, invest in capacity — sensitisation workshops on AI literacy and automation bias for the local bar, and, where feasible, screening tools to flag anomalous citations. I would escalate systemic concerns to the High Court so that a uniform, fair approach applies across courts, protecting litigants without penalising honest error.