Authored By: Saleena Shahid
University of Westminster
Introduction
Artificial intelligence refers to the development of machines which have been designed to simulate human intelligence and perform day to day tasks in a human-like manner. This essay will focus on a well-known and particularly influential type of AI known as Generative Artificial Intelligence, which has become increasingly integrated into various aspects of our society in order to facilitate technical, ethical, and legal developments. This integration has shown to have increased efficiency in industries such as healthcare and commerce, as the automation of everyday tasks and advancement of scientific research via widespread data collection and analysis, have helped to reduce reliance on manual labour and therefore streamline complex processes.
As such, these benefits have encouraged the business utilisation of Gen AI across various areas, as its ability to drive growth and reshape the way in which companies engage with customers has demonstrated to be key in enhancing overall company performance. However, despite these benefits, Gen AI poses serious risks which if left solely under the management of profit-driven businesses risk being overlooked and exacerbated. Such risks include: bias and discrimination and data privacy leaks, both of which are likely to occur as such businesses cannot be trusted to not misuse AI in their prioritisation of revenue over accountability and fairness. This essay will thus examine why Gen AI is too dangerous to be left to businesses to control, weigh the effectiveness of existing frameworks such as the EU AI act in regulating the technology, and propose recommendations to ensure the safe and ethical future use of AI systems.
Benefits of Gen AI
Before addressing the dangers of Gen AI it is important to take into consideration the benefits it has brought to both society and businesses. With regards to its positive impact on society, we will first look at its role in healthcare and how it was utilised in tackling the problems posed by the Covid 19 pandemic. The Covid 19 pandemic affected people’s lives in many ways that went far beyond physical health, bringing about lasting changes to the way in which societies functioned, with healthcare professionals being forced to work with limited resources and citizens being made to quarantine in an attempt to contain the virus. Whilst going into lockdown ultimately helped to prevent the virus from spreading, it had shown to have had a damaging effect on businesses with many companies having to shut down, whilst others underwent major changes that left job seekers facing a highly stagnant job market. These lockdowns also worsened existing inequalities, as remote workers were made to face poor working conditions while students who were already struggling at school resorted to dropping out entirely as a solution, thus emphasising the severity of the disruptions caused and the necessity of AI utilisation to bring them to an end.
One of the many ways in which AI had supported society during the pandemic was by helping to manage and predict the spread of the virus. Research teams around the world came together to create forecasting models that could track cases of the virus and aid decision-making, as can be seen through the collaboration between New York University’s Courant Institute of Mathematical Sciences with Facebook AI to develop localised prediction models for New Jersey. These models used public data to generate daily predictions which hospitals could then combine with their own data to better allocate resources such as: masks, ventilators, beds, and staff schedules in anticipation of an increase in patients. Another collaboration between Facebook AI and NYU’s Department of Radiology applied learning methods to clinical data like computer tomography scans and x-rays, to help hospitals predict the number of patients whose condition would worsen or get better in order to prepare for ICU admissions, transfers, or discharges. Another way in which AI contributed to managing the pandemic was through the use of wearable health devices. These devices allowed users to receive medical assessments at home, helping to reduce the need for in person health checks as healthcare providers used the data collected – respiration rates, heart rates, and resting heart rates to detect early signs of illness and isolate positive cases. Ultimately, the integration of AI into healthcare has proven to be key in improving public health through offering fast and more accurate medical support in times like the pandemic, showing the need for its future use to improve healthcare systems and prepare society for potential future challenges.
With regards to Gen AI’s positive impact on businesses, we will look at its ability to create new opportunities and enhance overall efficiency. Such technology possesses the ability to transform the way in which businesses operate through helping them gain valuable insights from data, automating processes to reduce their overall reliance on manual labour, and lastly aiding their decision-making to help them reach more informed conclusions. This is particularly beneficial for small businesses as they often have tight budgets and limited resources, therefore utilising Gen AI will help them to enhance the quality of customer service, and develop their marketing strategies through tools such as: virtual assistants and chatbots, all without having to worry about any additional costs. In particular, in the construction industry the utilisation of AI to automate repetitive tasks can help enhance accuracy during the planning and design stages to not only help produce results that stakeholders can easily interpret, but to also improve overall project stability as AI is less likely to produce errors in comparison to human workers.
Another way Gen AI can support business growth is by giving companies a competitive advantage, as by using this technology, businesses can understand their customers better through data insights, personalise their services, and make quicker and smarter decisions to keep up with market trends. Furthermore, by recommending products and services that customers are likely to want to purchase, AI can help to boost sales/increase the profit of businesses since consumer satisfaction is guaranteed as customer needs are being met more effectively. This is more important for small businesses than bigger ones, as by understanding customer needs they are able to create more targeted campaigns which will enable them to maximise their marketing efforts, and thus be on par with bigger businesses despite having limited resources. The business utilisation of Gen AI is therefore key to ensuring future success, as it will help to improve the efficiency of their work, cut down costs, and lastly help them to prepare and adapt to evolving market demands in order to outperform competition and ensure customer satisfaction.
Risks of Gen AI
On the other hand, it is important to consider the dangers of Gen AI and why it cannot be left to businesses to control. One risk is its potential to perpetuate bias and discrimination, thus raising concerns about accountability, transparency, and whether these principles should be integrated into the law. Examples of this include: Amazon’s recruitment tool which was shown to favour male candidates over female ones, as well as AI-driven job advertisements which tended to show women lower paying roles while higher level STEM positions were shown more frequently to men, reflecting gender biases present in the data the technology was trained on. Therefore, such a risk although severe on its own, can be exacerbated if Gen AI is left to businesses to manage as their profit-driven interests may prioritise efficiency and growth over the social and ethical considerations needed for responsible AI development. This particular concern has also been reflected through recent research which shows that 79% of senior IT leaders are worried about the security risks posed by AI systems, such as potential data privacy breaches, whilst 73% expressed concerns about these tools unintentionally reinforcing existing biases and inequalities. As professionals who directly oversee the implementation and management of AI systems, the concerns of senior IT leaders reflect a deep understanding of both the technical challenges and the ethical implications, providing critical insight into how these technologies could affect society at large and thus why it is too dangerous to be controlled by businesses.
Another serious risk posed by Gen AI is the likelihood for data privacy leaks, which can contribute to growing public distrust and undermine societal confidence in the technology. Even when data is collected and used within legal boundaries, AI systems are still expected to follow the principle of data minimisation so they should only handle information that is essential for their purpose. In practice, however, especially with complex systems like deep learning, it is often unclear what qualifies as essential which makes enforcement difficult. Regulations like Article 35 of the GDPR require companies to evaluate potential risks and justify their data usage, but since these evaluations are usually kept private to protect business interests and intellectual property, it becomes hard to determine how consistently or effectively these standards are being followed across the AI industry. Therefore, because oversight is often limited and businesses aren’t always obligated to be fully open about how they handle data the chances of personal information being exposed become higher, contributing to growing public concerns over whether AI can truly be relied on to keep their data safe. Such concerns have also arisen due to the fact that more than 70% of apps send personal data to third-party companies such as: Google and Facebook without the user’s full awareness, as can be seen through the Cambridge Analytica scandal, in which the personal data of millions of Facebook users was taken without permission and misused for political campaigns.
Current Frameworks
Therefore as awareness around the risks of AI increases, regulations such as the EU AI Act have been established in an effort to protect society from the potential harms. The framework follows a risk-based approach in which AI systems are classified into four risk categories: unacceptable risks – which involves manipulative systems that are banned, high risks – which includes systems that could harm health, safety, or rights and are thus heavily regulated, limited risks – which covers systems subject to fewer regulations, and lastly minimal risks – such as spam filters which face no specific obligations apart from AI literacy requirements. In addition to the four risk categories the Act also introduces a separate group for general-purpose AI models (GPAI), which apply additional strict requirements regardless of risk level, and are typically powerful, multi-functional models trained on large amounts of data like ChatGPT and Google Gemini. The Act applies to businesses operating within and outside of the EU, and aims to ensure that AI systems in the EU are transparent, fair, and environmentally friendly, however it must also be taken into consideration that part of the motivation behind the EU’s approach is not purely ethical. It is also driven by economic goals aiming to strengthen the EU’s global position in the AI race, as rather than distancing European AI development from market pressures, the approach is about staying competitive and tends to overlook the broader global consequences of rapidly accelerating AI advancements within the EU.
The Act itself also poses some limitations, as although the risk-based classification system establishes differing requirements based on the level of risk posed, it lacks a clear definition for what constitutes a high-risk system. Article 6 attempts to set out rules for classifying high-risk systems by referencing specific application areas listed in Annex III, however it simultaneously introduces exceptions that allow certain AI systems to be excluded from the high-risk category even if they fall within those areas. Additionally, rather than assessing the severity or likelihood of harm in a quantifiable way, the Act reduces the issue into whether a system does or does not have the potential to harm rights, thus allowing for differing interpretations among member states and providing corporations with a level of interpretive flexibility that could be used to avoid full compliance with the regulations. Profit-driven businesses in particular, can then take advantage of this flexibility to avoid accountability, hence undermining the goal of ensuring the safe and fair use of AI across the EU and rendering the framework for identifying high-risk systems both fragmented and inconsistent. Overall, this lack of clear regulation and accountability is key in demonstrating that Gen AI, given its potential for harm, is indeed too dangerous to be left solely in the hands of businesses to control.
Recommendations
Thus, in order to strengthen the EU AI Act recommendations for improvement are required. One such recommendation is to provide a clearer definition for high-risk systems, as the current ambiguity allows for interpretation and its potential exploitation by profit-driven businesses. Given the risks of Gen AI, further clarity is also key in ensuring that AI systems that are likely to cause harm are subjected to scrutiny and not integrated into society. Furthermore, alongside a clearer definition, the involvement of AI development companies in creating regulatory frameworks is also important. This is because these companies offer the technical expertise and practical understanding required to create rules that take into consideration the different ways in which AI operates across multiple sectors, which lawmakers alone may not be able to do as effectively. For example: the input of AI development companies have shown to be significant in sectors such as finance and healthcare, where trust and transparency are essential, as whilst generative AI can support decision-making it is ultimately human involvement that ensures ethical and accountable outcomes.
UK Approach
It is also worth considering how the UK has approached the topic of AI regulation. As part of its goal of becoming a global leader in AI, in 2023 the UK set out its vision for future governance through a White Paper focused on flexible regulation. The paper “A pro-innovation approach” highlights the need to build public trust in AI by addressing risks like: privacy breaches, security threats, and lastly harm to health, whilst also mentioning that overly strict rules could hold back progress. This message runs throughout the document and leans toward a more adaptive style of regulation in comparison to the EU’s more rigid legal framework, focused solely on uniform AI laws across member states. The UK’s approach however introduces a level of unpredictability, as businesses could face different regulations across sectors which could lead to confusion and uncertainty about which rules apply at any given time. Whilst this lack of consistency might be seen as a drawback of the approach, if the system is implemented well its flexibility to adapt to evolving technologies could offer a better regulatory framework compared to the EU’s rigid one, and thus allow the UK to remain competitive in the changing AI landscape.
Conclusion
To conclude, it is clear that Gen AI is far too dangerous to be left to businesses to control, as whilst AI has proven to be beneficial to both society and businesses, its benefits are ultimately undermined by the dangers associated with its utilisation. Therefore, improvements to regulatory frameworks need to be made with the input of AI development companies in order to ensure the safe and ethical future use of AI systems.
Bibliography:
Barfield W and Ugo Pagallo, Advanced Introduction to Law and Artificial Intelligence (Edward Elgar Publishing 2020)
Baxter K and Schlesinger Y, ‘Managing the Risks of Generative AI’ (Harvard Business Review6 June 2023) <https://hbr.org/2023/06/managing-the-risks-of-generative-ai> accessed 24 March 2025
Cath C, ‘Governing Artificial Intelligence: Ethical, Legal and Technical Opportunities and Challenges’ (2018) 376 Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
Clarke J and Darling C, ‘A Pro-Innovation Approach: UK Government Publishes White Paper on the Future of Governance and Regulation of Artificial Intelligence’ (Westlaw.com) <https://uk.westlaw.com/Document/IE0BAC9601CB511EE9B94BFC22BC1EB93/View/FullText.html> accessed 21 March 2025
Etienne H, ‘Solving Moral Dilemmas with AI: How It Helps Us Address the Social Implications of the Covid-19 Crisis and Enhance Human Responsibility to Tackle Meta-Dilemmas’ (2022) 14 Law, Innovation and Technology
Ka C and Colloton T, Generative AI in Higher Education (1st edn, Routledge 2024)
Kusche I, ‘Possible Harms of Artificial Intelligence and the EU AI Act: Fundamental Rights and Risk’ [2024] Journal of Risk Research
Lausberg I, AI in Business and Economics (Walter de Gruyter GmbH & Co KG 2024)
McCorquodale R and Tse M, ‘Artificial Intelligence Impacts: A Business and Human Rights Approach’ (Westlaw.comAugust 2024) <https://uk.westlaw.com/Document/I8F90A9F0763C11EBAA94B79BAF070BBB/View/FullText.html> accessed 18 March 2025
Mügge D, ‘EU AI Sovereignty: For Whom, to What End, and to Whose Benefit?’ (2024) 31 Journal of European Public Policy
Pavlidis G, ‘Unlocking the Black Box: Analysing the EU Artificial Intelligence Act’s Framework for Explainability in AI’ (2024) 16 Law, Innovation and Technology
Raposo VL, ‘Big Brother Knows That You Are Infected: Wearable Devices to Track Potential COVID-19 Infections’ (2021) 13 Law, Innovation and Technology
Wilson-Bushell A and Neal S, ‘Impact of the EU AI Act on the Creative Industries’ (Westlaw.com2 August 2024) <https://uk.westlaw.com/Document/ICAB6D300B2A211EFBE0DE4A699FAAD71/View/FullText.html> accessed 20 March 2025
