Authored By: Mokgaetji R. Brunette Mamokabi
UNISA
Introduction
Administrative, civil, and criminal justice paradigms have been drastically changed by the quick adoption of automated decision-making systems (ADMS) and Artificial Intelligence (AI) in public and commercial sector[1]. Unchecked use of automated decision-making presents unique legal issues in South Africa, where constitutional doctrine places a strong emphasis on transformative justice, procedural fairness, and non-discrimination. The “black box” problem, which occurs when a machine learning algorithm’s internal decision-making logic is hidden from end users, impacted parties, and even system creators, is at the center of this conflict[2]. The Protection of Personal Information Act 4 of 2013(POPIA) and Section 33 of the Constitution of the Republic of South Africa, 1996, which ensure just administrative action, are directly at odds with this opacity. In the event that an individual is denied bail, employment, housing and/or social benefits due to an algorithmic result, traditional administrative remedies become significantly compromised.
This article examines the legal implications of algorithmic bias and opacity in accordance to South African Administrative and Constitutional law. It assesses the effectiveness of international comparative frameworks, like the Artificial Intelligence Act of the European Union, in enforcing algorithmic accountability and suggests a comprehensive legal reform based on algorithmic due process and mandatory impact assessments within the South African Jurisprudence.
The Structure of Algorithmic Bias and Opacity
To assess the legal risks in association with ADMS, one must first dispel the myth that mathematical models are intrinsically objective. Algorithmic systems are artefacts of human engineering and historical data; accordingly, they consistently adopt, exacerbate, and entrench prevailing social disparities. Algorithmic bias typically manifests through three primary channels namely: historical bias, representation bias, and proxy variables. When training data represents deeply ingrained socioeconomic gaps, historical bias takes place. Predictive algorithms trained on legacy data run the risk of institutionalising historical injustices in South Africa, where historical spatial segregation and socioeconomic disparity still exist[3]. For instance, in historically underprivileged townships, overpolicing is frequently sustained by predictive policing technologies based on past arrest data. Because arrest records reflect enforcement patterns rather than raw crime rates, the algorithm treats living in a specific geographic area as a risk factor, creating a self-fulfilling feedback loop.
Insufficient demographic diversity in training datasets leads to representation bias, which makes the model unreliable or discriminatory against some populations. Commercial facial recognition systems, which show noticeable greater error rates when detecting women and people with darker skin tones, are well-known example[4].
Lastly, proxy variables- data points that closely correspond with protected qualities under Section 9(3) of the South African Constitution, like race, gender, and/or socioeconomic status- are widely used by algorithmic systems. An algorithm can recreate those characteristics using proxies like postal codes, income levels, and/or educational background even in the absence of clear demographic markers.
The legal issue posed by these mechanisms is compounded by three types of algorithmic opacity:
Trade Secret Opacity: Intellectual Property claims where proprietary software vendors withhold source code and algorithmic weights from judicial review.
Technical illiteracy: The disparity between complex machine learning architectures and the technical literacy of administrative officials, legal practitioners, and judges.
Inherent Unexplainability: The mathematical nature of the in depth learning of neural networks, where decision pathways are distributed across millions of parameters, rendering the rationale humanly uninterpretable.
Algorithmic Decision–Making under South African Law
Just Administrative Action and PAJA
Section 33(1) of the Constitution guarantees every individual the right to administrative action that is lawful, reasonable and procedurally fair. Section 33(2) guarantees the right to be given written reasons for administrative action that adversely affects rights. These guarantees are operationalised through the Promotion of Administrative Justice Act 3 of 2000(PAJA).
These fundamental protections are directly threatened by the use of opaque algorithms in official acts, such as social grant decisions, municipal licensing, or public sector employment:
Procedural Fairness (Section 3 of PAJA): Procedural fairness requires that parties that are affected receive adequate notice of the nature and purpose of the proposed administrative action and a meaningful opportunity to make representations. An opaque algorithmic determination deprives affected individuals of the factual and logical basis needed to challenge an adverse finding, rendering the right to make representation illusory.
The Duty to Provide Written Reasons (Section 5 of PAJA): In Koyabe v Minister of Home Affairs, the Constitutional Court upheld that giving reasons is essential to administrative justice, allowing impacted parties to choose whether to follow a ruling or request judicial review[5]. A mere numerical risk score or automated binary output generated by a closed algorithm does not constitute “written reasons” under administrative law. Without an interpretable explanation indicating how specific factual inputs produced the final product, the judgement fails the statutory requirement for reasoned administrative action.
Grounds for Judicial Review (Section 6 of PAJA): Under Section 6(2)(e)(iii) of PAJA, administrative action is reviewable if irrelevant considerations were taken into account or pertinent considerations were overlooked. When an organ of state relies solely on an opaque algorithm, proving which considerations were evaluated by the system becomes functionally impossible without access to source code, weights, and training parameters. Additionally, automated outputs run the danger of breaking Section 6(2)(f)(ii), which mandates that decisions be logically related to the administrator’s information and the purpose for which they were made.
South African Legal Safeguards
Constitution/Statutory Provision
Section 33 Constitution/ PAJA s 3 (procedural fairness and reasons)
Section 9 Constitution/ PEPUDA (prohibition of unfair discrimination)
POPIA Section 71(automated decision-making)
Vs Algorithmic Risks
Algorithmic Vulnerability
Opaque “black box” decisions deprive citizens of meaningful explanations and appeal rights
Proxy variables (e.g. , postal codes)mirror discriminatory patterns without intent
Exceptions for profiling and automated processing leave operational regulatory gaps
Indirect Unfair Discrimination under Section 9 and PEPUDA
Section 9(3) of the Constitution and the Promotion of Unfair Discrimination Act 4 of 2000(PEPUDA) prohibit direct and indirect unfair discrimination by both state and private entities[6].
Under Section 13 of PEPUDA, once a complaint establishes a prima facie case of discrimination did not take place, or that the conduct was fair[7]. In an algorithmic context, this burden-shifting framework creates a significant hurdle for defendants relying on proprietary tools. An entity cannot establish that an algorithmic output is “fair” if it cannot explain the internal logic that produced the biased result.
The Protection of Personal Information Act(POPIA)
Section 71 of the POPIA fully addresses automated decision-making. Section 71(1) establishes a general prohibition against subjecting a data subject to a decision that results in legal consequences or significantly affects them, if that decision is based solely on the automated processing of personal information intended to profile characteristics such as creditworthiness, reliability, or conduct[8].
However, Section 71(2) contains rather notable exceptions[9].
Comparative Regulatory Frameworks: The EU AI Act
To address these enforcement gaps, South African legal reform can draw valuable insights from international regulatory initiatives, most notably the European Union’s Artificial Intelligence Act (EU AI Act)[10].
The EU AI Act establishes a comprehensive, risk-based regulatory hierarchy:
Unacceptable Risk: Prohibits AI applications that pose a clear threat to safety or fundamental rights, such as government social scoring systems and real-time biometric identification in public spaces for law enforcement[11].
High-Risk Systems: Encompasses AI deployed in critical infrastructure, education, employment, migration, and the administration of justice. These systems are subject to strict ex-ante compliance mandates, including continuous risk management systems, rigorous data governance standards, technical documentation, automatic logging, transparency requirements, human oversight, obligations, and high levels of accuracy, robustness, and cybersecurity[12].
Limited/Minimal Risk: Subject primarily to basic transparency duties, such as notifying users that they are interacting with an AI system or that content is AI generated.
The EU AI Act represents a paradigm shift from reactive, post-hoc litigation to proactive structural regulation. By mandating pre-market conformity assessments, continuous post-market monitoring, and formal registration for high-risk systems, the EU framework provides a concrete model for how administrative law can address algorithmic bias before deployment occurs.
Strategic Recommendations for South African Legal Reform
To reconcile automated governance with South Africa’s constitutional guarantees and human rights obligations, the legal framework must evolve beyond traditional administrative machinery:
Codifying an Enforceable Right to Algorithmic Explanation: Regulations under PAJA and POPIA should be updated to establish that a legally sufficient explanation for an automated decision must disclose specific input variables, weightings, and counterfactual adjustments[13] [14].
Restricting Trade Secret Assertions in Public Law Matters: Where public organs deploy private software that affects constitutional rights, courts must restrict the scope of trade secret privilege[15] [16].
Mandating Pre-Deployment Algorithmic Impact Assessments(AIAs): Organs of state should be statutorily required to perform comprehensive AIAs prior procuring or deploying automated decision tools[17].
Addressing Automation Bias in Human Oversight: To prevent “human-in-the-loop” provisions from becoming procedural fictions, regulations under POPIA must require that human reviewers receive technical training and possess demonstrated authority to overrule algorithmic recommendations[18].
Conclusion
The integration of artificial intelligence into public governance and private decision-making presents a defining test for South African jurisprudence. While automated systems offer substantial efficiency gains, their unchecked deployment risks institutionalising historical biases and eroding constitutional guarantees of procedural fairness, just administrative action, and equal protection.
Protecting fundamental rights in the digital era requires updating administrative legal mechanisms to match modern technological realities. By enforcing algorithms transparency, requiring pre-deployment impact assessments, limiting proprietary evidentiary shields, and guaranteeing meaningful human oversight, South African law can harness technological innovation while upholding the rule of law and constitutional justice.
Bibliography
Primary Sources
South African Constitutional and Statutory Authorities
Constitution of the Republic of South Africa, 1996
Promotion of Administrative Justice Act 3 of 2000
Protection of Personal Information Act 4 of 2013
South African Case Law
Koyabe and Others v Minister for Home Affairs and Others 2010(4) SA 327(CC)
International Regulations
Regulation (EU) 2016/679 of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data[2016] OJ L119/1(General Data Protection Regulation)
Regulation (EU) 2024 /1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence [2024] OJ L1689/1(Artificial Intelligence Act)
Secondary Sources
Books
O’Neil C, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Crown Publishing Group 2016)
Journal Articles
Buolamwini J and Gebru T, ’Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification’ (2018) 81 Proc Machine Learning Res 1
Citron DK, ‘Technological Due Process’(2007) 34 Berkeley Tech LJ 189
Richardson R, Schultz JM and Crawford K, ‘Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice’ (2019) 94 NYU L Rev Online 15
Selbst AD and Barocas S, ‘The Intuitive Appeal of Explainable Machines’ (2019) 87 Geo Wash Rev 1085
Skitka LJ, Mosier KL and Burdick M. ‘Does Automation Bias Decision Making?’ (1999) 51 Int J Human-Computer Stud 991
Wachter S, Mittelstadt B and Russell C, ‘Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR’(2018) 31 Harv JL & Tech 841
Wexler R, ‘Life, Liberty and Trade Secrets: Intellectual Property in the Criminal Justice System’ (2018) 70 Stan L Rev 1343
[1] Cathy O’ Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Crown Publishing Group 2016) 21-26.
[2] Andrew D Selbst and Solon Barocas, The Intuitive Appeal of Explainable Machines’ (2019) 87 Geo Wash L Rev 1085, 1092.
[3] Rashida Richardson, Jason M Schultz and Kate Crawford, ‘Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems and Justice’ (2019) 94 NYU L Rev Online 15, 22-25.
[4] Joy Buolomwini and Timnit Gebru, ‘Gender Shades’.
[5] Koyabe v Minister of Home Affairs 2010(4) SA 327 (CC) paras 62-64.
[6] Constitution of the Republic of South Africa, 1996, s 9(3); Promotion of Equality and Prevention of Unfair Discrimination Act 4 of 2000.
[7] Promotion of Equality and Prevention of Unfair Discrimination Act 4 of 2000, s 13.
[8][8] Protection of Personal Information Act 4 of 2013, s 17(1).
[9] Ibid s 71(2).
[10] Regulation (EU) 2024/1689 of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence [2024] OJ L1689/1(Artificial Intelligence Act).
[11] Ibid arts 5-9.
[12] Ibid arts 10-15.
[13] Sandra Wachter, Brent Mittelstadt and Chris Russell, ‘Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR’ (2018) 31 Harv JL & Tech 841, 852.
[14] Promotion of Administrative Justice Act 3 of 2000, s 5; Protection of Personal Information Act of 2013, s 71.
[15] Rebecca Wexler, ‘Life, Liberty, and Trade secrets: Intellectual Property in the Criminal Justice System’ (2018) 70 Stan L Rev 1343, 1365.
[16] Constitution of the Republic of South Africa, 1996, s 33.
[17] Margot E Kaminski, ‘The Right to Explanation, Explained’ (2019) 34 Berkeley Tech LJ 189, 205.
[18] Linda J Skitka, Kathleen L Mosier and Mark Burdick,’ Does Automation Bias Decision Making? (1999) 51 Int J Human-Computer Stud 991, 993.
