How Does the AI Act Impact Image and Video Forensics?

Reading time: 35 min

In this post, Martino Jerian, CEO and Founder of Amped Software, breaks down the complex AI Act and explores how it affects the work of forensic image and video analysts, including key compliance areas and the challenges it brings to AI-based technologies used in law enforcement and investigations.

In this blog post, I will provide a very generic overview of the AI Act and then provide an in-depth analysis of its impact on the work of image and video forensic analysts.

Let’s start with an important disclaimer: I am not a lawyer, and this blog post is not legal advice. I read the AI Act (multiple times, especially some parts) from the perspective of a person involved in these topics from a technical point of view. What is written below is my understanding and interpretation of a pretty long and complex law. I hope it will inspire our readers to study the subject further. For proper legal advice on it, I recommend you or your organization seek the advice of a qualified legal professional.

Why the AI Act Is Important for Forensics and Investigations

The Artificial Intelligence Act is a regulation aimed at creating a comprehensive legal framework for artificial intelligence in the European Union. It entered into force on August 2, 2024, with full application proceeding in steps until August 2, 2027. Among its main purposes is the protection of fundamental rights, including democracy and the rule of law, against the harmful effects of AI while at the same time supporting innovation. In fact, many aspects of the regulation are related to the work done by law enforcement. It is essential that both vendors of technologies used for forensic and investigative activities and their end users understand and properly apply these regulations.

While the AI Act is an EU-specific law, we can expect it to have a broad influence worldwide. The European privacy law GDPR (General Data Protection Regulation) had a profound impact all over the world. First, because it defined a sort of privacy framework that inspired similar regulations in many other countries (and even some US states). Second, because the EU market is so big most enterprises cannot afford to ignore it. It forced them in most cases to evolve their products and processes for all their users, not just those based in Europe. I expect something similar to happen with the AI Act. The AI Act was highly publicized as the first worldwide AI regulation. It seems clear that the EU rushed it a bit to keep its title of “innovator in regulation” and “regulator of innovation”.

Finally, it’s important to be aware of the fact that responsibilities don’t belong only to technology producers but are shared also with users and other stakeholders, and non-compliance fines and consequences are relevant. This is the reason why I decided to write such a detailed article on our blog. You, our users, are subject to the AI Act as we are and it’s important to be aware of the potential risks of using technologies that are not compliant with it.

The AI Act in a Nutshell

This is the best I could do to summarize the AI Act. Please understand that I am probably oversimplifying many aspects.

The AI Act is a EU law of about 150 pages, published on the Official Registry in July 2024, but most will be compulsory by 2026 and 2027. It applies to all the operators (providers, deployers, distributors, etc.) working in or with EU subjects, even if the legal entity is not European, similarly to what happened with the GDPR.

The AI Act does not apply to uses for national security, military, research, and partially for open-source software.

The penalties for non-compliance are high, up to 7% of the global turnover of an organization.

The AI Act defines the following categories:

• Prohibited AI practices: that are, well, prohibited.
• High-risk AI systems: they can be used, following some compliance requirements.
• Generative AI: actually defined as “certain AI systems”, they have transparency obligations.
• General Purpose AI: they regard generic and widely available AI systems that have to adhere to some rules.

The following uses are prohibited AI practices:

• Behavioral manipulation
• Social scoring
• Predictive policing (partially)
• Scraping of facial images for face recognition
• Emotion recognition in work and education
• Biometric categorization (with some specific purposes)
• Law enforcement use of real-time biometric identification in public (with some exceptions)

The following uses are high-risk AI:

• Safety components of certain products, such as vehicles
• Biometric identification (not verification)
• Biometric categorization (in general)
• Emotion recognition (in general)
• Critical infrastructures
• Education
• Employment and workers’ management
• Medical devices
• Access to services (insurance, banks, emergency, etc.)
• Law enforcement and border control
• Justice
• Elections and voting

These are the requirements for the use of high-risk AI systems:

• Fundamental rights impact assessment
• Registration in an EU database
• Risk and quality management
• Data governance (bias, training data, etc.)
• Transparency (documentation, logging, etc.)
• Human oversight
• Accuracy, robustness, and cybersecurity

Obligations for certain AI systems (essentially generative AI):

• People should be informed when interacting with a chatbot or another AI system
• AI-generated or manipulated content, such as deepfakes, should be labeled and detectable

Obligations for general-purpose AI models:

• Transparency: technical documentation, training data summaries, copyright and IP safeguards
• Additional requirements for high-impact models with systemic risk: model evaluations, risk assessments, adversarial testing, incident reporting

The Objectives of our Analysis

The main objective of this article is to understand together if some typical activities performed during the analysis of image and video evidence are subject to compliance requirements according to the AI Act.

In a seminal blog post I wrote on the subject, I divided the kind of processing into “enhancement” and “analysis”:

• Image enhancement: algorithms that process an input image into an output image
• Image analysis: algorithms that process an input image into something else, often a decision or a classification

I also differentiated the requirements whether the results would have been used for investigative purposes only (what some call “forensic intelligence”) or had to be used as evidence in court.

On the other end, the classification done on the AI Act is for some aspects very generic and for others very specific.

Our objective is then to identify some possible activities that can be carried out with AI and understand whether they can be classified as prohibited, high-risk, or non- high-risk.

More specifically, we will consider the following activities:

• Image and video search and content analysis
• Face recognition on recorded video
• License plate recognition on recorded video
• Image and video redaction
• Image and video authentication
• Image and video enhancement

Note that we are only considering the post-analysis of a specific fragment of recorded video or a selection of images. We are not addressing the continuous real-time analysis of footage, as that is normally considered “surveillance” and not “forensics”. This is a very important distinction, especially for activities such as face recognition and biometric analysis in general, as we will see later.

The AI Act makes it clear that the classification of some activities as high-risk or prohibited, can change over time, so we will have to monitor the law for changes.

In the meantime, we will also understand the main points of the AI Act together. We will also see very broadly what the compliance requirements for high-risk systems are. Finally, we will discuss the potential impact of the AI Act on the field of image and video forensics.

Important Definitions

Before we discuss something, we need to define it. Of the many general provisions and definitions in the AI Act, I copied and commented on those I deemed important for our scope. Here and in the following sections, the italics indicate text copied from the AI Act. While the bold and underlined text is mine to highlight important concepts for our analysis.

Art 1.1. The purpose of this Regulation is to improve the functioning of the internal market and promote the uptake of human-centric and trustworthy artificial intelligence (AI), while ensuring a high level of protection of health, safety, fundamental rights enshrined in the Charter, including democracy, the rule of law and environmental protection, against the harmful effects of AI systems in the Union and supporting innovation.

The beginning of the first article of the AI Act (relative to the subject matter of the law) is important because it gives a basic idea of the motivations behind it. Some of the foundational aspects are fundamental rights, such as democracy (one of the most often underlined aspects of EU policies) and the rule of law. This will reflect the fact that law enforcement applications make a good part of the regulation.

Art 2.3 (…) This Regulation does not apply to AI systems where and in so far they are placed on the market, put into service, or used with or without modification exclusively for military, defence or national security purposes, regardless of the type of entity carrying out those activities.

I find this aspect worth noting: investigative and forensic tools are often used also in the military field and for national security purposes. If they are exclusively used for these applications (and not for law enforcement), they are not subject to the AI Act. The line between public safety and national security is somewhat blurred. We can better differentiate between the two by considering the specific organizations in each country that focus on either aspect. Additionally, the work done by law enforcement can potentially lead to judiciary proceedings, while it doesn’t happen for national security organizations.

Let’s proceed with some definitions and comment on them.

(Art 3.1) ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments;

The definition of an AI system is very vague. So vague that, in my opinion, if we strictly follow only this definition, it’s unclear whether some of the commonly used AI tools are included. Moreover, it is possible that traditional model-based systems (without AI) may be included in it. However, it must be said that in most cases, at the technical level, it’s somewhat well-agreed what is AI and what is not.

(Art 3.3) ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge;

This makes it clear for example that Amped Software, or any other software vendor, is defined as “provider”.

(Art 3.4) ‘deployer’ means a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity;

This makes it clear that our end users are defined as “deployers” in the regulation.

(Art 3.8) ‘operator’ means a provider, product manufacturer, deployer, authorised representative, importer or distributor;

Operator is a word that includes all subjects that have something to do with the AI system.

(Art 3.45) ‘law enforcement authority’ means:

1. any public authority competent for the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, including the safeguarding against and the prevention of threats to public security; or
   
   
2. any other body or entity entrusted by Member State law to exercise public authority and public powers for the purposes of the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, including the safeguarding against and the prevention of threats to public security;

(Art 3.46) ‘law enforcement’ means activities carried out by law enforcement authorities or on their behalf for the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, including safeguarding against and preventing threats to public security;

These items are very important since they clearly define “law enforcement”! My understanding is that this definition also includes those who work in private practice (“…or on their behalf”) and not only government agencies.

(Art 3.60) ‘deep fake’ means AI-generated or manipulated image, audio or video content that resembles existing persons, objects, places, entities or events and would falsely appear to a person to be authentic or truthful;

I like this definition of deep fake a lot (even though I prefer to use the word “deepfake” without the space): anything created with AI that would falsely appear to be authentic or truthful. According to this definition, if it’s clear that it’s false or AI-generated, it’s not a deepfake. This causes the fact that it’s not possible to clearly and unequivocally define what is a deepfake only from the technical point of view, but some context is needed.

I think this makes sense: if we use a text-to-image tool to create “a drawing of a dinosaur riding a motorbike” I don’t call it a deepfake, as I wouldn’t call a fake a hand-drawn image of the same subject. But if I make a photorealistic image that can deceive the observer, like a political leader engaged in illegal or embarrassing activities, that’s a deepfake.

The definition of deepfake is consistent with the definition of “authentication” according to the SWGDE: “The process of substantiating that the data is an accurate representation of what it purports to be.”

Prohibited AI Systems

The first thing we need to do is to understand if any of the activities we have defined above are classified as “prohibited” according to the AI Act. Reading what is written in Art. 5, none of them are included, but two points are worth mentioning since they are related to our field anyway.

The first is the prohibition of scraping of facial images from the web or other sources for face recognition.

(Art. 5.1e) the placing on the market, the putting into service for this specific purpose, or the use of AI systems that create or expand facial recognition databases through the untargeted scraping of facial images from the internet or CCTV footage;

The second is the prohibition of real-time biometric identification in public. There has been a long negotiation on defining some exceptions to this, which are detailed in the regulation, but are out of the scope of our analysis since we are not interested in real-time use.

(Art. 5.1h) the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purposes of law enforcement, unless and in so far as such use is strictly necessary for one of the following objectives:

1. the targeted search for specific victims of abduction, trafficking in human beings or sexual exploitation of human beings, as well as the search for missing persons;
   
   
2. the prevention of a specific, substantial and imminent threat to the life or physical safety of natural persons or a genuine and present or genuine and foreseeable threat of a terrorist attack;
   
   
3. the localisation or identification of a person suspected of having committed a criminal offence, for the purpose of conducting a criminal investigation or prosecution or executing a criminal penalty for offences referred to in Annex II and punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least four years.

High-risk AI Systems

We need now to go one step further and identify if any of the activities of interest are classified as high-risk.

Art. 6 defines what activities are considered high-risk, and for our purposes further defines them in Annex III, whose content is expected to change over time (important!).

2. In addition to the high-risk AI systems referred to in paragraph 1, AI systems referred to in Annex III shall be considered to be high-risk.

I copied here only the sections of the Annex that are relevant to our analysis. I also highlighted the parts of interest that will be commented on later in this section.

High-risk AI systems pursuant to Article 6(2) are the AI systems listed in any of the following areas:

1. Biometrics, in so far as their use is permitted under relevant Union or national law:
   
   
   1. remote biometric identification systems.
      This shall not include AI systems intended to be used for biometric verification the sole purpose of which is to confirm that a specific natural person is the person he or she claims to be;
      
   2. AI systems intended to be used for biometric categorisation, according to sensitive or protected attributes or characteristics based on the inference of those attributes or characteristics;
      
      
   3. AI systems intended to be used for emotion recognition.

(…)

6. Law enforcement, in so far as their use is permitted under relevant Union or national law:

1. AI systems intended to be used by or on behalf of law enforcement authorities, or by Union institutions, bodies, offices or agencies in support of law enforcement authorities or on their behalf to assess the risk of a natural person becoming the victim of criminal offences;
   
   
2. AI systems intended to be used by or on behalf of law enforcement authorities or by Union institutions, bodies, offices or agencies in support of law enforcement authorities as polygraphs or similar tools;
   
   
3. AI systems intended to be used by or on behalf of law enforcement authorities, or by Union institutions, bodies, offices or agencies, in support of law enforcement authorities to evaluate the reliability of evidence in the course of the investigation or prosecution of criminal offences;
   
   
4. AI systems intended to be used by law enforcement authorities or on their behalf or by Union institutions, bodies, offices or agencies in support of law enforcement authorities for assessing the risk of a natural person offending or re-offending not solely on the basis of the profiling of natural persons as referred to in Article 3(4) of Directive (EU) 2016/680, or to assess personality traits and characteristics or past criminal behaviour of natural persons or groups;
   
   
5. AI systems intended to be used by or on behalf of law enforcement authorities or by Union institutions, bodies, offices or agencies in support of law enforcement authorities for the profiling of natural persons as referred to in Article 3(4) of Directive (EU) 2016/680 in the course of the detection, investigation or prosecution of criminal offences.

7. Migration, asylum and border control management, in so far as their use is permitted under relevant Union or national law:

1. AI systems intended to be used by or on behalf of competent public authorities or by Union institutions, bodies, offices or agencies as polygraphs or similar tools;
   
   
2. AI systems intended to be used by or on behalf of competent public authorities or by Union institutions, bodies, offices or agencies to assess a risk, including a security risk, a risk of irregular migration, or a health risk, posed by a natural person who intends to enter or who has entered into the territory of a Member State;
   
   
3. AI systems intended to be used by or on behalf of competent public authorities or by Union institutions, bodies, offices or agencies to assist competent public authorities for the examination of applications for asylum, visa or residence permits and for associated complaints with regard to the eligibility of the natural persons applying for a status, including related assessments of the reliability of evidence;
   
   
4. AI systems intended to be used by or on behalf of competent public authorities, or by Union institutions, bodies, offices or agencies, in the context of migration, asylum or border control management, for the purpose of detecting, recognising or identifying natural persons, with the exception of the verification of travel documents.

8. Administration of justice and democratic processes:

1. AI systems intended to be used by a judicial authority or on their behalf to assist a judicial authority in researching and interpreting facts and the law and in applying the law to a concrete set of facts, or to be used in a similar way in alternative dispute resolution;
   
   
2. AI systems intended to be used for influencing the outcome of an election or referendum or the voting behaviour of natural persons in the exercise of their vote in elections or referenda. This does not include AI systems to the output of which natural persons are not directly exposed, such as tools used to organise, optimise or structure political campaigns from an administrative or logistical point of view.

First of all, I think it’s important to consider the sentence “Law enforcement, in so far as their use is permitted under relevant Union or national law”. The AI Act has to coexist with other European and National laws. The fact that it allows a certain practice doesn’t supersede other laws or, for example, best practices of the forensics community.

At the very beginning, we clearly see the fact that biometrics are classified as high-risk.

a. remote biometric identification systems.

This means that face recognition, but potentially also voice recognition or more exotic activities like gait recognition, are considered high-risk. Let’s remember that this is for recorded video; real-time operations are listed in the prohibited AI practices.

Then, this is one of the most important, and, in my opinion, problematic, items on the list.

b. AI systems intended to be used by or on behalf of law enforcement authorities or by Union institutions, bodies, offices or agencies in support of law enforcement authorities as polygraphs or similar tools;

At first glance, it reads as a restriction only for polygraphs or similar tools. However, let’s remember that this is a law, and as such, we need to analyze it carefully. This sentence can also stand on its own without the last part, and then “as polygraphs or similar tools” becomes only an example.

If this is the case, it means that any AI system used by law enforcement is considered high-risk. I don’t think this is the correct interpretation, since putting a broad statement such as this, makes the rest of the list useless. I think it’s more probable for this item to include just polygraphs and similar tools. However, I will look for clarification and updates on it. The question would be then, what does it mean for a system to be similar to a polygraph?

The next point, on the other hand, is pretty evident. When applied to our case, it makes the use of AI for tasks such as image and video authentication (including deepfake detection) a high-risk activity.

c. AI systems intended to be used by or on behalf of law enforcement authorities, or by Union institutions, bodies, offices or agencies, in support of law enforcement authorities to evaluate the reliability of evidence in the course of the investigation or prosecution of criminal offences;

(6d) and, especially, (6e) seem to have some relation with our tasks (“detection, investigation or prosecution of criminal offences”). They both refer to profiling, however, and point to the relative law for its precise definition:

(4) ‘profiling’ means any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements;

While I thought that “profiling” could also include activities aimed at identification, such as face recognition, the definition is pretty precise and doesn’t include this aspect.

The next section is relative to migration, asylum and border control management. For subsection 7a we can apply the same considerations as 6a, relative to the example of polygraph.

a. AI systems intended to be used by or on behalf of competent public authorities or by Union institutions, bodies, offices or agencies as polygraphs or similar tools;

The other relevant point is 6d:

d. AI systems intended to be used by or on behalf of competent public authorities, or by Union institutions, bodies, offices or agencies, in the context of migration, asylum or border control management, for the purpose of detecting, recognising or identifying natural persons, with the exception of the verification of travel documents.

This is interesting because it sets a pretty wide statement on AI systems used for the identification and recognition of persons in the context of immigration and border control (except travel documents). Many forensic technologies are used for this purpose. It seems that they are in general not considered high-risk for law enforcement use, unless it’s related to border control (that is often a law enforcement activity anyways).

Finally, a last potentially relevant point is:

8. Administration of justice and democratic processes:

1. AI systems intended to be used by a judicial authority or on their behalf to assist a judicial authority in researching and interpreting facts and the law and in applying the law to a concrete set of facts, or to be used in a similar way in alternative dispute resolution;

I was wondering whether it could be interpreted so widely to include any technical activity in the forensic field. But it seems to be related to the idea of a so-called “Robo Judge” or “AI judge” that sifts through laws and sentences and reaches or suggests a verdict in court.

Finally, even if the activity is identified as high-risk, Art. 6 defines some derogations:

3. By derogation from paragraph 2, an AI system referred to in Annex III shall not be considered to be high-risk where it does not pose a significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making.

The first subparagraph shall apply where any of the following conditions is fulfilled:

a. the AI system is intended to perform a narrow procedural task;

b. the AI system is intended to improve the result of a previously completed human activity;

c. the AI system is intended to detect decision-making patterns or deviations from prior decision-making patterns and is not meant to replace or influence the previously completed human assessment, without proper human review; or

d. the AI system is intended to perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III.

Notwithstanding the first subparagraph, an AI system referred to in Annex III shall always be considered to be high-risk where the AI system performs profiling of natural persons.

4. A provider who considers that an AI system referred to in Annex III is not high-risk shall document its assessment before that system is placed on the market or put into service. Such provider shall be subject to the registration obligation set out in Article 49(2). Upon request of national competent authorities, the provider shall provide the documentation of the assessment.

These derogations, especially (3d), can probably be applied to most systems where the AI does just a small part of the work, together with non-AI systems, and just support the work done by an operator. We can think about video enhancement or image authentication: even though some part of the forensic work is done with AI, according to the AI Act, they are not seen as high-risk activities. Under point 4, it’s important to note that this derogation should be assessed and documented properly.

What Video Forensics Activities Are Considered High-risk?

Now that we have reviewed prohibited and high-risk practices, let’s analyze the specific video forensics activities identified at the beginning of the article.

Image and Video Search and Content Analysis

This includes a very broad range of applications, for example:

• Create a video summary of all motion events in a video, trimming the parts where nothing happens
• Find all the videos where a person with a blue t-shirt is visible
• Extract all faces present in a video
• Find all pictures where there is some weapon or drugs
• Find all explicit pictures or videos
• Identify all the vehicles in a video that were traveling in the wrong direction
• Identify events out of the ordinary in a video

According to the previous section, search and content analysis are not high-risk activities in general. There may be an exception when they are used in the context of migration, asylum, or border control management to detect, recognize, or identify natural persons (Annex III, Art. 8a). But this can likely be derogated since it is just a preparatory task (Art. 6.3d).
Note that this is valid for a recorded video (real-time analysis may be more problematic) and does not include the cases where the automatic analysis is done for profiling.

Face Recognition on Recorded Video

According to our analysis, face recognition on recorded video is a high-risk activity, together with any other remote biometric identification system on recorded video.
Let’s remember not only that real-time biometrics are among the prohibited AI practices (Art. 5.1h), but also that the creation of facial recognition databases scraping from the internet or CCTV (Art. 5.1e) is prohibited.

License Plate Recognition

According to our analysis, license plate recognition is not a high-risk activity.
According to Lorch et al., who published an excellent paper on the topic in 2022, they expect this kind of systems to be considered high-risk:

It is not explicitly stated whether AI systems for source identification or photographic comparison, including license plate recognition, are considered high-risk. However, the Commission is entitled to expand the list of high-risk applications. AI systems can be added to the list when they pose a similar or even higher risk to the health, safety, or fundamental rights than the applications already listed. Since source identification
and photographic comparison aim at identifying a suspect, potentially leading to investigative measures against the suspect, we expect these applications to also be considered high-risk.

However, with the current version of the law, it’s pretty clear that license plate recognition is not included in high-risk activities.

Image and Video Redaction

Image, video (and also audio) redaction is quite a good candidate for automation with AI.
Redaction is normally performed by blurring, pixelating, or putting a solid shape on some detail we want to protect from view. This often includes faces, license plates, and text written on screens.
According to our analysis, redaction is not a high-risk activity.

Image and Video Authentication

Authentication is usually performed by combining multiple types of analysis. For example, this may include analyzing image format, metadata, content, pixel statistics, and so on. Deepfakes, also defined in the AI Act, are an important and growing problem.
The most popular way to detect deepfakes is to use AI-based detectors. While these detectors are very powerful, we suggest complementing and verifying their output with traditional model-based algorithms.

According to our analysis, authentication may be considered a high-risk activity since it’s used to evaluate the reliability of evidence in the course of the investigation or prosecution of criminal offences (Annex III, Art. 6c). On the other hand, AI-based authentication is usually just a small part of an analyst’s toolbox that can use it also to verify their assumptions. It may be intended to improve the result of a previously completed human activity (Art. 6.3b) and to perform a preparatory task (Art. 6.3d), the analyst uses to draw their conclusion. Therefore, derogations may apply.

Lorch et al. are pretty much aligned with our analysis:

Annex III explicitly states law enforcement application scenarios that the Commission classifies as high-risk. These include AI systems for evaluating the reliability of evidence in the course of investigation or prosecution of criminal offenses, and AI systems for deep fake detection. Therefore, we conclude that AI-based image authentication counts as high-risk application.

Image and Video Enhancement

In the prohibited or high-risk AI sections, there is no mention of image and video enhancement for law enforcement purposes. Therefore, according to our analysis, enhancement is not a high-risk activity according to the AI Act.

There may be an exception when enhancement is used in the context of migration, asylum or border control management, to detect, recognise or identify natural persons (Annex III, Art. 8a). But likely it can be derogated since it is just a preparatory task (Art. 6.3d).

As explained multiple times, my personal position, shared by many in our field, is that it’s a very risky activity. So I am worried by the fact that this aspect has not been directly addressed. On the other hand, I can see that it is a pretty technical topic, probably best discussed in more specific laws and best practices than a very broad and generic regulation like the AI Act.

Requirements for High-risk AI Systems

Now that we have identified what activities are potentially considered high-risk, what are the requirements for compliance? Please note, again, that here I am doing just a high-level overview and commenting on things that have caught my attention in relation to multimedia forensics activities. Some parts of it are somewhat easy, and others seem pretty challenging. I expect over time that compliance processes will be defined and understood better. Definitely, a professional expert on the topic will be needed in most cases.

Article 8 – Compliance with the requirement

This article essentially says that the following requirements should be fulfilled.

Article 9 – Risk management system

This article defines how to implement a risk assessment document.

Article 10 – Data and data governance

2. Training, validation and testing data sets shall be subject to data governance and management practices appropriate for the intended purpose of the high-risk AI system. Those practices shall concern in particular:

This is one of the most important and difficult parts. Datasets used for training, validation, and testing AI systems should follow some guidelines. I expected it to be important only for training. However, if we want to give an actual verifiable number for the system’s accuracy, validation and testing are also important.

The definitions of training, validation and testing data come from Art. 3:

(29) ‘training data’ means data used for training an AI system through fitting its learnable parameters;
(30) ‘validation data’ means data used for providing an evaluation of the trained AI system and for tuning its non-learnable parameters and its learning process in order, inter alia, to prevent underfitting or overfitting;
(32) ‘testing data’ means data used for providing an independent evaluation of the AI system in order to confirm the expected performance of that system before its placing on the market or putting into service;

b. data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection;

The origin of the data should be clearly defined, and data collection should be clear concerning the privacy aspect. I think this may be a big challenge for dataset creation. If the dataset was originally collected for another purpose than training an AI system, likely we have a problem.

f. examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations;

It’s important to evaluate possible biases introduced by the dataset that can affect the fundamental rights of persons. Of course, an AI system that is used for forensic purposes can seriously affect the security and freedom of a person.

g. appropriate measures to detect, prevent and mitigate possible biases identified according to point f.;

This indicates the need for some bias mitigation measures.

3. Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose. They shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons in relation to whom the high-risk AI system is intended to be used. Those characteristics of the data sets may be met at the level of individual data sets or at the level of a combination thereof.

We should use databases representative of the population to avoid biases. A common example is the bad performance of some current face recognition systems on people of specific ethnicities.

Article 11 – Technical documentation

This article essentially defines the need to explain in depth what the system is expected to do and how it works, with specifics detailed in Annex IV.

Article 12 – Record-keeping

3. For high-risk AI systems referred to in point 1 (a), of Annex III, the logging capabilities shall provide, at a minimum:

a. recording of the period of each use of the system (start date and time and end date and time of each use);

b. the reference database against which input data has been checked by the system;

c. the input data for which the search has led to a match;

d. the identification of the natural persons involved in the verification of the results, as referred to in Article 14(5).

I think this is sensible, but it could conflict with the security and confidentiality needed in some scenarios.

Another point I was wondering about was whether it was ok for desktop software to log on the deployer systems and not centrally (since technically the end user could modify the logs, unless a technical protection like a digital signature is applied).

I somewhat found an answer in Art. 16: (e) when under their control, keep the logs automatically generated by their high-risk AI systems as referred to in Article 19. So, having logs on the deployer system and not centrally is considered a viable option.

Article 13 – Transparency and provision of information to deployers

1. High-risk AI systems shall be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured with a view to achieving compliance with the relevant obligations of the provider and deployer set out in Section 3.

3. The instructions for use shall contain at least the following information:

b. the characteristics, capabilities and limitations of performance of the high-risk AI system, including:

1. its intended purpose;
   
   
2. the level of accuracy, including its metrics, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity;
   
   
3. any known or foreseeable circumstance, related to the use of the high-risk AI system in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, which may lead to risks to the health and safety or fundamental rights referred to in Article 9(2);
   
   
4. where applicable, the technical capabilities and characteristics of the high-risk AI system to provide information that is relevant to explain its output;
   
   
5. when appropriate, its performance regarding specific persons or groups of persons on which the system is intended to be used;
   
   
6. when appropriate, specifications for the input data, or any other relevant information in terms of the training, validation and testing data sets used, taking into account the intended purpose of the high-risk AI system;
   
   
7. where applicable, information to enable deployers to interpret the output of the high-risk AI system and use it appropriately;

This article is extremely important for our field and describes the need for transparency, explainability, and interpretability. Unfortunately, we all know that in many situations, AI is very weak from this point of view. This article acknowledges this in a way by adding “where applicable” or “when appropriate” in several points.

Article 14 – Human oversight

1. High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.

4. For the purpose of implementing paragraphs 1, 2 and 3, the high-risk AI system shall be provided to the deployer in such a way that natural persons to whom human oversight is assigned are enabled, as appropriate and proportionate:

a. to properly understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions and unexpected performance;

b. to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;

c. to correctly interpret the high-risk AI system’s output, taking into account, for example, the interpretation tools and methods available;

d. to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system;

e. to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.

This article essentially says that the AI system should be a decision support system. Ultimately, the responsibility and choice must stand in the hands of a human, who must be allowed at any time to stop the system safely.

Article 15 – Accuracy, robustness and cybersecurity

1. High-risk AI systems shall be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle.

3. The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.

While accuracy was already mentioned in Art. 13, it’s stressed again here, together with robustness. Cybersecurity seems to be included here just as a second thought. However, it is nevertheless a very important aspect that falls under more specific regulations. In this case, it is likely connected to adversarial learning, where an attacker may get control of or pollute part of the training dataset, or even the weights of a trained model.

Obligations for AI Image Generation Tools

Interestingly, there is an article that touches our field tangentially but has an important impact on it. In fact, article 50 speaks about foundational models and big providers of text-to-image and text-to-video tools (and their users).

Article 50
Transparency obligations for providers and deployers of certain AI systems

2. Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated. Providers shall ensure their technical solutions are effective, interoperable, robust and reliable as far as this is technically feasible, taking into account the specificities and limitations of various types of content, the costs of implementation and the generally acknowledged state of the art, as may be reflected in relevant technical standards. This obligation shall not apply to the extent the AI systems perform an assistive function for standard editing or do not substantially alter the input data provided by the deployer or the semantics thereof, or where authorised by law to detect, prevent, investigate or prosecute criminal offences.

There is A LOT to unfold here.

The first part “Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated.” can be summarized with the obligation of image generation tools to digitally sign or watermark any media that has been artificially generated or manipulated. Bingo! We don’t need deepfake detection anymore. Except for the fact that 1) this is a European law, 2) not everybody respects the law and 3) the existence of open-source tools makes this effectively unenforceable everywhere. Sure, we will likely solve part of the big problem, since we expect major players to comply. But this won’t block any attacker with a minimum of interest or technical skills.

On the other hand, this clarifies that whenever an image is manipulated, this must be clearly detectable. What is not clear is the definition of “manipulation”. Below they write do not substantially alter the input data provided by the deployer or the semantics thereof. Using AI super resolution to zoom on a face adding details that weren’t present in the original image is a substantial alteration or not? A lot of food for thought.

Note that the articles don’t say not to use AI, just to make it evident when it has been used.

These are the situations where the obligation does not apply:

1. Assistive function for standard editing: if the AI is just finding the optimal parameters, but the processing is done in traditional ways. One example could be identifying with AI the shape of a blur, but then applying a standard Wiener filter or automatically detecting faces to be redacted, and then pixelizing them.
2. Do not substantially alter the input data provided by the deployer or the semantics thereof: I think we could discuss ages on this point, whether AI changes the input data SUBSTANTIALLY, as it’s very much context-dependent. As an example, the image below comes from our paper “Does Deep Learning-Based Super-Resolution Help Humans With Face Recognition?”. The photo on the left is an original hi-resolution image of Tommy Lee Jones, the one on the right is a low-resolution version enlarged with AI. While the differences may be subtle, doing a facial comparison on the image processed with AI will likely cause very dangerous errors. The shape of the eyes of Tommy Lee Jones is very peculiar, and changes drastically in the AI-processed image.
   
   
   
3. Or where authorised by law to detect, prevent, investigate or prosecute criminal offences: essentially when these tools are authorized for LE, no need to put a disclaimer. This is a bit counterintuitive, we need MORE rigor to investigate crime, not less! However, I think this has been written with in mind something like the creation of fake accounts to be used on social media during investigations. In a hypothetical scenario where you have bots chatting with suspects in the hope of obtaining a confession, it would not be ideal to reveal that the friendly lady the criminal was chatting with does not actually exist. Furthermore, her photos may have been generated using Midjourney, while the text was created with a ChatGPT-like application. This example illustrates the potential issues with transparency in such situations.

Finally, point 4 of the same article clearly states that it’s not just an obligation of the vendors to follow transparency requirements, but also of the deployers.

4. Deployers of an AI system that generates or manipulates image, audio or video content constituting a deep fake, shall disclose that the content has been artificially generated or manipulated. This obligation shall not apply where the use is authorised by law to detect, prevent, investigate or prosecute criminal offence. Where the content forms part of an evidently artistic, creative, satirical, fictional or analogous work or programme, the transparency obligations set out in this paragraph are limited to disclosure of the existence of such generated or manipulated content in an appropriate manner that does not hamper the display or enjoyment of the work.

The Existing Amped Position on AI and the AI Act

I’ve been working on the impact of AI on forensics for a few years, writing blog posts and papers. I’ve been giving countless presentations and participated in meetings on the topic. I have also shared insights through various LinkedIn posts.

The table below summarizes our position on AI for forensics.

It’s a bit difficult to do a direct comparison with the distinctions we made because the AI Act approach is very broad (and vague) in some aspects and very specific in others.

From a very general point of view, most of the safeguards I mentioned are in line with the AI Act requirements. I think that generally deciding what is prohibited, high-risk, or not, does not belong to the technical aspect, but is essentially a social and political choice. Once we decide what is risky, general countermeasures are relatively obvious.

In general, let’s assume that an analysis system is not prohibited but considered high-risk, and let’s compare it with the safeguards I wrote about in my original blog post.

Only for Decision Support

Decision support system. The decision should help the analyst, not replace him. A system should help the user focus his attention on the most probable targets, but the user should always have the last word. For example, a facial identification system may help the analyst identify the most probable matches for a suspect, but the actual analysis and decision should be made by a human with a proper explainable analysis. For example, a face identification match given by an AI algorithm should never be used as forensic evidence but may be used to help the investigator identify the most likely matching faces.

– Martino Jerian, CEO and Founder of Amped Software

In the AI Act, the concept of human oversight is mostly addressed in Art. 14. Key aspects of this article include:

1. High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.

a. to properly understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions and unexpected performance;

b. to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;

d. to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system;

Known Reliability

The system should give some indication about the reliability of the result in general cases and/or specific situations. This should help the analysts understand how much they can rely on the result given. What’s the usual reliability of the system? Does it work correctly 60% or 99.99% of the time? What’s the confidence level in a specific analysis?

– Martino Jerian, CEO and Founder of Amped Software

This is mainly considered in Article 13 which focuses on Transparency and provision of information to deployers. Additionally,  Article 15 emphasizes the importance of Accuracy, robustness and cybersecurity.

Article 13 – Transparency and provision of information to deployers

1. High-risk AI systems shall be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured with a view to achieving compliance with the relevant obligations of the provider and deployer set out in Section 3.

3. The instructions for use shall contain at least the following information:

b. the characteristics, capabilities and limitations of performance of the high-risk AI system, including:

1. its intended purpose;
   
   
2. the level of accuracy, including its metrics, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity;
   
   
3. any known or foreseeable circumstance, related to the use of the high-risk AI system in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, which may lead to risks to the health and safety or fundamental rights referred to in Article 9(2);
   
   
4. where applicable, the technical capabilities and characteristics of the high-risk AI system to provide information that is relevant to explain its output;
   
   
5. when appropriate, its performance regarding specific persons or groups of persons on which the system is intended to be used;
   
   
6. when appropriate, specifications for the input data, or any other relevant information in terms of the training, validation and testing data sets used, taking into account the intended purpose of the high-risk AI system;
   
   
7. where applicable, information to enable deployers to interpret the output of the high-risk AI system and use it appropriately;

Article 15 – Accuracy, robustness and cybersecurity

1. High-risk AI systems shall be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle.

3. The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.

User Bias Mitigation

Limit human bias. Analysts should be aware of the possible bias induced by the system over the human user and take the proper steps to mitigate it. If the analyst receives a strong match by an AI system for what regards an identification, they will likely be unconsciously biased towards a positive match. Similarly, maybe the correct matching face could have been mistakenly discarded by the AI system and thus condition the user to ignore it. In this regard, it’s important to educate users on the limits of technology and how it can condition the opinion of the operator doing the analysis, and adopt bias mitigation techniques, such as linear sequential unmasking.

– Martino Jerian, CEO and Founder of Amped Software

This part can be found in Article 4: AI Literacy and Article 9: Risk management system.

Interestingly the first is a general article and not only for high-risk systems. I don’t think the AI Act stresses enough the risks of bias by these systems. Maybe this issue is addressed in the risk assessment for the deployers?

Article 4
AI literacy

Providers and deployers of AI systems shall take measures to ensure, to their best extent, a sufficient level of AI literacy of their staff and other persons dealing with the operation and use of AI systems on their behalf, taking into account their technical knowledge, experience, education and training and the context the AI systems are to be used in, and considering the persons or groups of persons on whom the AI systems are to be used.

Article 9
Risk management system

5. […] With a view to eliminating or reducing risks related to the use of the high-risk AI system, due consideration shall be given to the technical knowledge, experience, education, the training to be expected by the deployer, and the presumable context in which the system is intended to be used.

Data Governance

Data governance is a huge additional part we didn’t consider. I believe they want to stress the responsibility of providers in implementing bias mitigation measures. This includes establishing appropriate guidelines for datasets, which is effectively the most complicated part of AI. It’s not just for training data, but also validation and testing. Of course, it’s also the responsibility of deployers to mitigate the bias during the usage, but, if the system itself has significant issues in this regard, this limits what can be done.

Article 10 – Data and data governance

2. Training, validation and testing data sets shall be subject to data governance and management practices appropriate for the intended purpose of the high-risk AI system. Those practices shall concern in particular:

b. data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection;

c. relevant data-preparation processing operations, such as annotation, labelling, cleaning, updating, enrichment and aggregation;

d. the formulation of assumptions, in particular with respect to the information that the data are supposed to measure and represent;

e. an assessment of the availability, quantity and suitability of the data sets that are needed;

f. examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations;

g. appropriate measures to detect, prevent and mitigate possible biases identified according to point (f);

3. Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose. They shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons in relation to whom the high-risk AI system is intended to be used. Those characteristics of the data sets may be met at the level of individual data sets or at the level of a combination thereof.

In summary, our existing position on AI is different but in line with the AI Act. The AI Act strongly emphasizes the importance and transparency of datasets used for training, validation, and testing. While it is more specific in some aspects, it remains vague in others. This approach aims to partially address the AI black box issue through improved data engineering. For example, when training our DeepPlate system (which is for investigative use and in any case not a high-risk AI system according to our analysis) we made extensive use of data augmentation and informed assumptions on the structure of license plates.

Conclusions

As we have seen, the AI Act includes many aspects that are relevant to law enforcement. However, it doesn’t provide too many details for professionals working in digital and multimedia forensics.

According to our analysis

• these practices are considered high-risk when performed with AI:
  • Image and video authentication (not just deepfake detection, but also the detection with AI of “traditional” forgeries)
  • Face recognition on the post-analysis recorded video (in real-time it is prohibited, with some exceptions)
    
    
• these practices are not considered high-risk when performed with AI:
  • Image and video search and summary
  • License plate recognition
  • Image and video redaction
  • Image and video enhancement (but the image must be marked as AI-generated)

Let’s remember that the list of high-risk applications is expected to change over time, and many details will have to be clarified.

One big doubt that I have is image and video authentication: from our analysis, it seems to be considered a high-risk practice. For example, many companies are claiming to do automatic deepfake detection nowadays. What happens if a deepfake detection system, designed for general use (and then not supposed to be a high-risk activity), is employed in a law enforcement context, to evaluate the reliability of the evidence? Is there a responsibility on the side of the user or also of the vendor?

In general, I think that the AI Act is a great start. Over time, however, things will have to be specified more in some aspects, both on what systems are high-risk or not and how to address compliance. I think we need stricter rules on the use of AI to enhance image and video evidence for forensic purposes. At least Art. 50 gives us an assist in the fact that they should be transparently documented as if they were completely AI-generated.

Let’s not forget that the AI Act is one of the many laws forensic labs and courts must adhere to. They are also subject to other regulations and should follow best practices that could be way more restrictive on certain aspects. For example, in a recent case in Washington state, in the US, videos enhanced with AI have been rejected, setting an important precedent.

Is the AI Act a risk for innovation in Europe? Definitely! Even if we put safeguards on many aspects of AI, many other countries won’t and we risk running with our legs tied together.

However, the AI Act is fully coherent with the European Union’s fundamental values of human rights, democracy, and security for its citizens, touted many times. In the context of Europe, the AI Act makes a lot of sense, and I am sure it will be a guide and inspiration for many (but not all) other countries.