Authored By: Varsha Yadav
Llyod School of Law
Introduction
AI has penetrated into all aspects of daily Indian life. From approving the loan in one’s phone application to recommending the next episode to watch on Netflix, AI systems are making millions of decisions every day. However, the power and ubiquity of such technology give rise to an awkward question – who is watching over the system? In India, a country where over 1.4 billion people produce incredible amounts of digital data, the relationship between AI and privacy presents a legal minefield.
The Indian government has attempted to address these concerns. The Digital Personal Data Protection Act, 2023 (DPDP Act) became India’s first data protection law. Moreover, in November 2025, India introduced the India AI Governance Guidelines aimed at creating a risk-based approach to applying AI technology to various sectors of the economic. However, there are still many legal gaps in this area. Specifically, the DPDP Act failed to introduce regulation regarding the unique problems associated with AI (algorithmic biases, automated decision-making, or the right to explanation, among others). This paper discusses legal challenges at the intersection of AI and data privacy, provides an overview of existing regulations, and offers ways to deal with the problem.
Scale of AI in India and data collection.
As of the year, India is number one for the adoption of AI and the Prime Minister Narendra Modi commented during the Paris AI Action Summit in February 2025: ‘India leads in AI adoption and techno-legal solutions on data privacy’. The stakes are economic. Boston Consulting Group study, supported by IIT-A, reports that AI adoption can increase real annual GDP growth to as much as 1.4 per cent, and private investment in AI R&D amounts to nearly $642 million.10
This influx is underpinned by equally large data collection. Without substantial datasets, an AI system cannot operate. The backbone of modern AI, machine learning, is inextricably linked to the process of data collection and training sets as well as their subsequent utilization for accurate data processing. In India, data is collected for automated AI decision-making from various sources, including social media, transactions and biometric datasets such as Aadhaar. The National Data Governance Framework Policy of 2022 provides the means to share data for research in AI, while taking some measure of security and confidentiality.
The scale in India creates a fundamental tension between AI’s need to collect as much data as possible to optimize it and the data protection principle of collection limitation, which states that a sufficient but not excessive quantity of data is collected for only the necessary purpose.
Scale of AI in India and data collection.
As of the year, India is number one for the adoption of AI and the Prime Minister Narendra Modi commented during the Paris AI Action Summit in February 2025: ‘India leads in AI adoption and techno-legal solutions on data privacy
The stakes are economic. Boston Consulting Group study, supported by IIT-A, reports that AI adoption can increase real annual GDP growth to as much as 1.4 per cent, and private investment in AI R&D amounts to nearly $642 million.10
This influx is underpinned by equally large data collection. Without substantial datasets, an AI system cannot operate. The backbone of modern AI, machine learning, is inextricably linked to the process of data collection and training sets as well as their subsequent utilization for accurate data processing. In India, data is collected for automated AI decision-making from various sources, including social media, transactions and biometric datasets such as Aadhaar. The National Data Governance Framework Policy of 2022 provides the means to share data for research in AI, while taking some measure of security and confidentiality.
The scale in India creates a fundamental tension between AI’s need to collect as much data as possible to optimize it and the data protection principle of collection limitation, which states that a sufficient but not excessive quantity of data is collected for only the necessary purpose.
DPDP Act 2023-A promising frame with flaws:
Key aspects of the Act and the limitations to its application:
The DPDP Act 2023 is India’s first piece of legislation for data protection. It promotes an understanding of data protection centre around consent, i.e., “consent is the rule and the primary form of informational autonomy”. As per section 4 of the DPDP Act, no processing of the data of a data principal may occur without valid consent or without any legitimate use.
The Table of permitted bases (legitimate use) as per section 7 of the DPDP Act has been discussed under 9 circumstances where the use of data would be considered legitimate:
1.Specific purpose associated with voluntary disclosure
2.Processing of data by the State or any instrumentality thereof for the issue of any license, benefit or subsidy.
3.Performing any public duty imposed upon the State or any instrumentality thereof.
4.Compliance of legal obligation imposed upon the State or any instrumentality thereof in India.
5.Compliance of any order passed by any court or Tribunal in India.
6.Emergency life threatening situations / necessity for immediate care.
7.Treatment during any epidemic and health emergencies.
8.In cases of national emergency, security or public safety issues, disaster and other emergencies.
9.Employees related purpose
Owners will need either consent, or one of these legitimate grounds to process data for training AI. However, the Act is very silent about AI-specific issues, and therefore regulation is seriously lacking.
No Automated Decision-Making Safeguards
Perhaps the most glaring omission is that of fully automated decision-making. Although it has requirements for consent and notice that make it valid, the DPDP Act fails to provide a legislative basis for the exclusion of purely automated decision-making. Similar to the right under Article 22 of the EU GDPR for individuals to not be subject to decisions made on purely automated processing without meaningful human intervention or to contest automated decisions with human review, India does not have robust protections in place for algorithmic accountability.
The issue arises out of what legal scholar Tasmai Mandal identifies as a “regulatory paradox” The consent regime in the DPDP Act takes into account occasional consent at onboarding users give their consent and that’s it-but algorithmic analysis occurs constantly throughout their interaction, predicting and determining behaviors over time. Consent is then given not merely for an initial processing activity but for an entire decision-making environment the inputs to which are largely unknown,25and downstream implications can’t be reasonably anticipated.
The Act doesn’t guarantee explainability for AI, nor provide a mechanism to challenge automated decisions.26Individuals whose job applications are rejected or who are denied loans due to algorithms remain without recourse.
Public Data Exception and Potential Abuse
3(c)(ii) of DPDP Act provides exception for personal data that is made available by the data principal or any person under obligation to do so. Companies could use this provision for processing the datasets on large scale and without the need for consent. However, the Act does not provide clear guidelines on cases where this publicly available data has been taken down or has been made private and whether it still counts as the personal data to which the Act would apply. The above creates uncertainty to data fiduciaries as well as the data principals. As Artificial Intelligence models more often ‘scrape’ data available on public platforms for its training, the difference between ‘use’ of the data for beneficial purposes and privacy violation is becoming ever blurrier.
Algorithmic Bias and Discrimination – An Unattended Problem
Entering AI Algorithms
One major concern with AI-automated decision-making systems is the existence of algorithmic bias that can result in unjust treatment and discrimination. Because AI algorithms are trained on data collected from history, the bias present in these historical data sets can affect the results obtained, particularly in aspects of gender, caste, and socioeconomic status. Unregulated AI tools for recruitment, credit scoring, facial recognition, etc can replicate and solidify previous discriminative trends, especially impacting vulnerable groups.
This problem is more exacerbated given India’s immense diversity. With thousands of castes, countless major languages, huge disparities in income and economic status, varying levels of digital literacy among the population, algorithms trained on unrepresentative data can yield systematically discriminatory results. A recruitment tool that uses an algorithm trained mostly on data sets for urban and English-speaking participants can result in unfairly discriminating against capable individuals from non-English and rural backgrounds. A credit scoring system
that relies on training data from only a subset of demographic groups can end up rejecting a loan for individuals with similar credit ratings as those from groups for which more data was available.
The regulatory vacuum
While the EU’s GDPR has set precedents for AI decision transparency, Indian legislation has no express regulations relating to fairness of algorithms. This lack of explicit requirements to conduct fairness audits, check for bias and ensure diverse datasets poses systemic risks of discrimination within AI-powered making systems.
Though, the India AI Governance Guidelines were released in November 2025 and proposed a risk-based, evidence-led methodology to regulate AI risks and provided for safety measures for preventing risks on individual or societal interests; they do not have any statutory mechanism like appeals and independent audits and are principle-based guidelines instead of statutory laws. Thus, these guidelines depend on existing laws such as the IT Act and DPDP Act which themselves don’t contain explicit provisions relating to AI fairness.
As a result of such regulatory vacuum, affected individuals are not able to have recourse for compelling that AI systems to be fair in testing or even challenge an algorithm if they feel it has led to biased decision.
The Black Box Problem: Transparency and Explainability
Why AI is a Black Box
One of the biggest issues facing AI and automated decision -making is lack of transparency. A lot of AI systems, especially those employing deep learning, behave like “black boxes” where they are capable of producing decisions but are not capable of explaining those decisions in a
way that humans can understand. Because they operate on very large, complex algorithms, their developers cannot necessarily even track how particular inputs are mapped to particular outputs.
The lack of transparency is especially problematic in critical sectors. In healthcare an AI could potentially propose a course of treatment or make a diagnosis but would not be able to explain why doctors or patients should follow that proposal or accept the diagnosis. In the criminal justice system, facial recognition programs may identify potential suspects in cases, but it is impossible to transparently understand or appeal any false positives or other misidentification in the system. In the finance industry automated credit decisions can and do result in loan denials that cannot be reasoned through or challenged by the applicant.
Legal Implications of capacity
The DPDP Act does not compel an explanation of the AI decision making process, raising significant questions for the prospects of procedural justice: how will an individual be able to challenge a decision they do not comprehend; how will a regulator verify compliance if they cannot investigate how a decision was made; and how will courts be able to evaluate whether rights have been infringed when they cannot determine how the infringement occurred? The India AI Governance Guidelines acknowledge the potential harms that AI systems may pose to individuals and society but these principles are just that-principles-and do not require specific measures around transparency. Regulators in the various sectors will need to ensure
enforcement within the existing mandates, but typically existing sector-specific legislation does not account for AI explainability requirement. This is unlike trends elsewhere, the EU AI Act defines AI system by risk level and imposes transparency requirements, particularly on high-risk applications. Similarly, the US has drafted Algorithmic Accountability Act provisions that call for impact assessments of automated systems.
Accountability and Liability: Whose Accountability?
Problem of Attribution
One crucial aspect related to decision-making based on AI technology concerns attributing liability in case something goes wrong. In situations where an AI-based technology makes a wrong choice, whether it is about rejecting the job application, loan rejection, wrong patient diagnosis, or any other decision leading to some losses, the question is who should be held accountable for such an outcome: the one who developed the algorithm; the organization that utilized it; the entity whose data was used in training; or the person who authorized the usage of AI technology?
The DPDP Bill does not stipulate liability related to harm inflicted by AI decision-making. There are no special provisions concerning this problem in the Information Technology Act, 2000.
Proposed Solutions and Associated Issues
Some legal specialists suggest implementing a “human in the loop” solution, meaning that human decision-making will oversee the operation of AI in fields where the use of such technologies is more delicate. Such an approach means that important decisions made by AI algorithms will be thoroughly checked and, thus, will include some human assessment and not purely automatic actions.
Another option would involve creating a particular liability regime for AI developers and users which would entail them being legally responsible for mistakes made by algorithms and instances of discrimination. It can be achieved through introducing compulsory insurance, implementing strict liability regimes for some applications, or amending the burden of proof rule.
There are no new statutory liability mechanisms in the India AI Governance Guidelines; instead, existing legislation is utilized to provide solutions for potential issues associated with AI technologies
Bias and Discrimination in Algorithms: The Unsolved Problem
How Bias Can Creep into AI Technology
One of the biggest challenges with making decisions through automation by AI is that there will be algorithmic bias and discrimination. The AI is designed and taught using historical data which is likely to be influenced by gender, caste, and socio-economic disparities when left unchecked, the use of automated AI recruitment processes, loan scoring mechanisms, and facial recognition technology can result in perpetuating discrimination.
In India, the problem of bias is compounded by the very diversity in the country. There are thousands of castes in India, many languages, wide economic differences, and varied levels
of digital literacy. Using non-representative data for training AI can result in systematic biases. An example of this is where an AI recruitment algorithm trained on data mainly based on urban and english speaking candidates will discriminate against other qualified candidates who may not share those characteristics. The same goes for algorithms used in credit scoring systems.
Regulatory Vacuum
Contrary to the GDPR in the European Union, which requires transparency in AI decision making, India’s regulation makes no specific reference to algorithmic fairness. Without a mandate for fairness auditing and bias detection, and without diversity guidelines, there is an increased possibility of systemic bias being built into decision-making through AI algorithms
The India AI Governance Guidelines, issued in November 2025, take a risk-based, evidence driven approach and specify safeguards to ensure the safety of individuals and society. But the India AI Governance Guidelines are principle-based and provide no new statutory processes such as independent audits and appeals. Instead, they refer to other laws such as the IT Act and DPDP Act, which again do not have any specific requirements for algorithmic fairness.
It follows from the regulatory vacuum that affected individuals have no way of asking for a fairness audit of AI algorithms and challenging results based on bias.
The Black Box Problem: Transparency and Explainability
Why Is AI Opaque?
The biggest issue that exists when talking about AI is the lack of transparency. Some AI algorithms, such as deep learning algorithms, are black boxes; thus, there is no explanation provided for why a certain input led to a specific output. Furthermore, due to the complexity of the algorithm, not even the creators can sometimes figure out what made the computer do something in a certain way.
In many industries where it matters most, AI’s opacity causes the biggest problems. For example, in the healthcare industry, some AI algorithms provide recommendations on how to deal with a certain problem; however, doctors cannot know why. In law enforcement, facial recognition technology may find a suspect whose identification cannot be verified because the algorithm does not provide any information about the process itself. Finally, in finance, automated loan decisions cannot provide clear reasons for why someone was denied a loan
New Cases and Legal Precedents
ANI vs Open AI
ANI vs Open AI is an important legal case in India, wherein the Delhi High Court examined the issues of copyright infringement related to the creation of content by using AI technology. The case is a reflection of the new legal challenges arising from the use of AI technologies in copyright infringement cases.
International Legal Precedents
The trend of filing lawsuits against AI technologies is growing internationally. Microsoft, GitHub, and OpenAI are some organizations which are facing various legal challenges relating to data misuse in AI technologies. There are more than 8,500 authors, artists, and media houses that have filed lawsuits against AI firms in the matter of theft of works for AI training.
Although legal precedents have been set in India to resolve legal disputes concerning AI, there are no effective laws that can help in controlling damage caused by AI technologies.
Paradoxes of Consent: Dark Patterns and Consent Manufacture
How Consent Becomes Irrelevant
The consent-based DPDP Act faces another paradoxical dilemma: consent often comes as manufactured consent rather than consent itself. Interface design, defaults, bundling, functional dependencies, and computational inference lead to broader processing of personal data than anticipated at the point of collection.
Dark patterns refer to interface design that manipulates users to give away more data than intended or even consent. Dark patterns remain largely unaddressed by law in India . Users face take-it-or-leave-it terms where denying consent equates to losing access to critical services. Such an approach does not entail actual consent.
Temporal Misalignment
According to Tasmai Mandal’s paper, a temporal discrepancy forms the essence of consent based legitimacy. At the point of onboarding, autonomy exists episodically in the form of clicking “agree.” However, subsequent algorithmic processing takes place over an extended period of time and entails the continuous acquisition of information.
Such a disparity implies that there is no way for consent given in the beginning to cover all future data processing activities. Users simply do not have any idea about something they did not foresee. However, the truth is that artificial intelligence systems are always coming up with new uses and inferences they could make, and which users had never considered.
Reform Proposals
Among other proposed reform steps outlined in the article are:
– Anti-dark pattern consent regulations to prevent manipulation through UI design – Transparency regarding profiling and inferences about users
– Review and control
Digital Personal Data Protection Act, No. 15 of 2023, Acts of Parliament, 2023 (India). Information Technology Act, No. 21 of 2000, Acts of Parliament, 2000 (India). Cases
ANI v. OpenAI, W.P.(C) No. 11984/2023, Delhi High Court (2023).
Journal Articles
Tasmai Mandal, Consent and Algorithmic Decision-Making under the DPDP Act, 2023: A Regulatory Paradox, 3 NAT’L J. CRIM. L. & SOC. JUST. 1 (Mar. 4, 2026).
Legal Challenges of Artificial Intelligence in India’s Cyber Law Framework, 11 INT’L J. FORENSIC SCI. & MGMT. 31347 (Nov. 23, 2024).
A Critical Analysis of India’s DPDP Act, 2023, 22 LIBERATION 1 (July 2025).
AI AND DATA PROTECTION: CHALLENGES IN AUTOMATED DECISION-MAKING, 5 INDIAN J. INT’L ST. POL. & RES. 1 (Feb. 27, 2025).
Governing AI: A Narrative Review of Algorithmic Accountability and…, 4 LEGALIS 1 (Apr. 29, 2025).
Government Documents & Reports
Press Information Bureau, India AI Governance Guidelines Do Not Allow Unrestricted Deployment of High-Risk AI Systems, GOVT. OF INDIA (Nov. 19, 2025), https://www.pib.gov.in/PressReleasePage.aspx?PRID=2206769.
Press Information Bureau, India AI Governance Guidelines Do Not Allow Unrestricted Deployment of High-Risk AI Systems, GOVT. OF INDIA (Nov. 4, 2025), https://static.pib.gov.in/WriteReadData/specificdocs/documents/2025/nov/doc202511568560 1.pdf.
Vidhi Centre for Legal Policy, Comments on the Draft Digital Personal Data Protection Rules, 2025, VIDHI LEGAL POLICY (May 1
