The EU’s proposed AI Regulation

Conformity assessment (as described above) is intended to certify that the system in question meets these requirements:

1. Risk management systems must be established, implemented, documented, maintained and regularly updated. The risk management system must identify and analyse foreseeable risks associated with the AI and eliminate or reduce those risks to the extent possible and otherwise implement control measures in relation to those risks.

2. Data and data governance. High-risk AI systems which involve training models with data must use training, validation and testing data sets which are subject to appropriate data governance and management practices, are relevant, representative, free of errors and complete, and take into account the characteristics or elements that are particular to the specific geographical, behavioural or functional setting within which the AI system is intended to be used.

3. Technical documentation, containing as a minimum the information detailed in Annex IV, including a detailed description of the elements of the AI system and the process of its development, must be drawn up before the AI systems are placed on the market or put into service, and must be kept up-to-date.

4. Record keeping. High-risk AI systems must have logging capabilities ensuring traceability of the AI system’s functioning throughout its lifecycle, at a level appropriate to its intended purpose.

5. Transparency and provision of information to users. The operation of high-risk AI systems must be (i) sufficiently transparent to enable users to interpret the AI system’s output and use it appropriately; and (ii) accompanied by instructions for use, including any known and foreseeable circumstances that may lead to risks to health and safety or fundamental rights, human oversight measures, and the expected lifetime of the high-risk AI system. The information must be concise, complete, correct and clear, and must be relevant, accessible and comprehensible to users.

6. Human oversight. High-risk AI systems must be capable of being overseen by natural persons, with the aim of preventing or minimising risks to health, safety or fundamental rights. The provider is to identify and build (where possible) oversight measures into the AI system. The designated individual should fully understand the capacities and limitations of the AI system and be able to monitor its operation and output for signs of anomalies, dysfunctions and unexpected performance. Humans should be able to intervene and stop the system.

7. Accuracy, robustness and cybersecurity. High-risk AI systems must, in light of their intended purpose, be appropriately accurate, and the accuracy metrics must be declared in the accompanying instructions of use. The systems must also be appropriately robust and resilient to errors, faults or inconsistencies and resilient to third parties intending to exploit system vulnerabilities, including data poisoning and adversarial examples.

These high-risk requirements will be onerous to comply with. Potential issues include:

• The requirement that data sets be “representative” does not sit easily with GDPR requirements regarding sensitive personal data. Similarly, a provider of a high-risk AI system is allowed to process the GDPR special categories of personal data for the purposes of ensuring bias monitoring, detection and correction in those systems, subject to certain safeguards. But minimal detail is provided as to what those safeguards would encompass.

• The requirement that data used to train AI systems be “free of errors and complete” may well be unachievable in practice, given the scale of the data sets used in machine learning.

• The requirement, in certain circumstances, for a designated individual to be able to “fully understand” the operation of a complex AI system, sets a very high bar; this is unlikely to be an attractive role for anyone to take on.

• Traceability requirements may pose problems for certain deep learning AI systems, where it is difficult to clearly explain and trace how the system is functioning.