High-Risk Artificial Intelligence

Abstract

Artificial intelligence (AI) technologies are increasingly permeating and transforming all walks of life, unlocking new potential for effciency, automation, and innovation across most industry sectors (Yang et al. 2024). However, their implementation is not without signifcant challenges and risks. Integrating AI into information systems (IS), which are per se socio-technical systems, introduces risks that extend beyond purely technical concerns (Jussupow et al. 2021; Maedche et al. 2019; Pfeiffer et al. 2023; Wiener et al. 2023). Given the immense opportunities and signi?cant risks associated with AI, the topic has increasingly drawn attention from academia, industry, and legislators. As such, in April 2024, following three years of trilateral negotiations between the European Commission, the European Council, and the European Parliament, the European Union introduced a landmark regulation on AI. This so-called AI Act aims to establish the first comprehensive legal framework governing the use of AI technologies in the European Union (European Commission 2021; European Parliament 2024). A central component of the AI Act is its defnition of four risk classes for AI systems (see Fig. 1). With this risk classification, the AI Act puts particular emphasis on the regulation of high-risk AI systems (i.e., those AI systems that could impact and endanger the health, safety, or fundamental rights of individuals). These systems are subject to rigorous oversight, including mandatory internal conformity assessments by the providers and, in special cases, external reviews by notified bodies (European Parliament 2024; Hupont et al. 2023). Despite such regulatory measures being put forward, numerous questions remain open, especially regarding the integration of high-risk AI in IS.