In recent days, following the media attention on the Kenosha Shooting trial (“STATE OF WISCONSIN – VS – Kyle H. Rittenhouse”) in which Amped FIVE has been used for evidence analysis, a previous article on this blog about the use of Artificial Intelligence on image and video forensics has been misunderstood and instrumentalized. The applicability of image interpolation and image enhancement at large as evidence in court has been discussed, sometimes without the needed in-depth knowledge of the field. In this article, we will clarify some very important concepts related to forensic video analysis at large. Amped FIVE implements forensically safe algorithms to enhance video evidence.
1) Amped FIVE does NOT use Artificial Intelligence
Amped FIVE has been designed specifically for evidentiary use. It does not use Artificial Intelligence: image and video enhancement in Amped FIVE are implemented in a forensic workflow based on carefully selected algorithms that guarantee reliability, repeatability, and reproducibility. Thanks to this, Amped FIVE has become widely accepted as the standard tool for forensic image and video analysis, being used in 100 countries worldwide.
2) Interpolation DOES NOT TAMPER with the image
We need interpolation to show things as they are. Interpolation is not only used to “zoom on” an image, but it is an essential part of the creation and display of a digital photo or video. Interpolation does not add image information, but improves visualization of image data 1,2,3. Questioning the general acceptability of interpolation means questioning the acceptability of images and videos as evidence.
3) Image enhancements performed by a competent analyst with the right tools are INSTRUMENTAL FOR COURT USE
An analyst with the right tools, technical preparation, and workflow can enhance the image in a way that can help the trier of fact and be accepted in court. Image enhancement is a fundamental part of forensic video analysis and it’s the duty of the forensic video analyst to properly enhance images and videos to give a more accurate representation of the scene, compensating, when possible, the imperfections introduced by the image generation process.
We hope this new take on the argument will help to better comprehend the topic and will clarify some of the misinterpretations of the original post. And if you want to learn more on these topics, please keep reading!
YOU DON’T KNOW WHAT YOU DON’T KNOW
The beauty and damnation of image and video forensics is the fact that it’s something that at a visual and intuitive level seems very understandable to the average viewer, but actually underlies pretty complex mathematical concepts. While it seems easy, the risk of getting something wrong is high, especially if you are not aware of the limitations of technology and the shortcuts done by our human brain during interpretation. This, in fact, has been the focus of our Video Evidence Pitfalls series earlier this year.
FORENSIC SCIENCE, NOT FORENSIC SHOW
We rarely mention real cases on our blogs and socials. As a company, we focus on providing the best tools and support for forensic video analysis, which is why we don’t perform expert witness consultancy.
Recently, there’s been this high profile, and very politicized, case in the United States. We received several requests for comment but we don’t usually render opinions on open trials, so we decided to stay silent until the jury verdict was delivered. We thought that it wouldn’t have been appropriate to exploit the situation while the case was open and that whatever we said, it could have been used by either party to support their side.
In fact, even without saying anything, a previous article on our blog was distorted and misinterpreted. So, let me put this straight.
AMPED FIVE DOES NOT USE ARTIFICIAL INTELLIGENCE FOR ENHANCEMENT
People went to our website, blog, and YouTube videos and widely commented on them. While many got our message right, there were some who didn’t seem to understand. I like to think that when there’s a misunderstanding, it’s because the source needs to be clearer. After all, as a forensic video expert, my style has always been keener on providing the best possible technical explanations instead of bending analysis on behalf of marketing and corporate communication. For this reason, I decided to release a single tweet from my personal account with the hope of clarifying the fact that Artificial Intelligence (AI) techniques are not present in Amped FIVE.
The confusion was due to the fact that in this blog post, I did an analysis about the use of AI for different kinds of forensic applications. Given the scientific approach of our company, in our blog, we often speak about industry-wide topics, and not only about what’s inside our products. I thought this was a very important topic to study and explain, and to take a clear position on it. I was speaking in general terms, and my skepticism on AI was made clear multiple times, even on older blog posts.
In this specific article, it was NOT written that Amped FIVE uses AI to enhance images, just that AI shouldn’t be used (in general) to enhance images for evidentiary purposes. Amped FIVE was not mentioned at all, nor any other of our products. However, some individuals interpreted that post like a declaration that Amped FIVE uses AI, and as such, it should be used only for investigations, not for working on evidence. Why should I have created forensic software not to be used for evidence? This would not have made any sense.
No part of our blog, let alone the user manual, states that Amped FIVE uses AI or that it should not be used on evidence and I’m personally very worried that this misinterpretation was published on various websites and even used during the trial.
AMPED FIVE IS DESIGNED FOR EVIDENTIARY USE
As explained multiple times on this blog, since the beginning, Amped FIVE has been designed with the single and clear purpose of supporting the forensic analyst on the complete workflow of image and video analysis, with a reliable, repeatable, and reproducible workflow, based on solid scientific foundations and forensic best practices.
This focus allowed us to become the standard tool for forensic image and video analysis in many countries worldwide. Both private and government forensic labs in one hundred countries are using our software because, despite different legal contexts, the language of science is one and only.
The problem is that, when it comes to the court and the general public, there’s always a risk of a misunderstanding and instrumentalization of technology power and limits.
Forensic video analysis is usually defined as “the scientific examination, comparison and/or evaluation of video in legal matters”.
When we talk about forensic science, “science” should be the foundation, and “forensics” should build over it, not the other way around. When the opposite happens, there’s a risk of science being bent to the needs of a legal proceeding.
Those of you who have been following us for a while, know that our company motto is “justice through science”, and our company goals and culture have been built around this fundamental principle. Between someone going overboard with processing and others arguing the possibility to do any kind of enhancement, we believe we should strive for a proper balance.
IMAGE INTERPOLATION IS EVERYWHERE
In the above-mentioned trial, the process of “image interpolation” has been questioned, especially for what regards the “bicubic interpolation” algorithm, fundamentally for a misunderstanding of how it works.
What is interpolation in a nutshell? Digital images have a given size in pixels. Whenever you need to display an image on a medium that has a different resolution in pixels (either lower or higher), you need to estimate the values of the pixels in positions that were not present in the original image. Additionally, you need interpolation for every kind of geometric transformation applied to an image, for example, when you are resizing it to zoom.
Bicubic is a standard algorithm that is implemented virtually everywhere, and it’s one of the most used ones. It’s not an AI-based algorithm that can add or remove objects, but it’s only a mathematical combination of nearby pixels.
Interpolation is everywhere and mostly out of our control. Where can we find it, for example?
- Taking a picture or video with any digital camera (during the demosaicing and image acquisition process).
- During video recording, every time digital stabilization is used.
- Playing a video in Windows Media Player, VLC, or any other software (interpolation is part of the decoding process, for example, as a consequence of chroma subsampling).
- Playing a video from the internet in your browser, either as a thumbnail or in full screen.
- Often during the rendering of the image on a TV, a computer monitor, a projector, or any other screen.
If we think that interpolation is tampering with the evidence, then basically we should exclude any form of image and video evidence from any trial.
WE NEED INTERPOLATION TO SHOW THINGS AS THEY ARE
Videos captured from CCTV and other devices are often saved with the wrong aspect ratio or are affected by lens distortion. To show the actual proportions and shapes of an object, we need to correct them. Whatever process you use to adjust them, this involves interpolation. There is absolutely no way around that. Should we stick to curved walls and confuse SUVs with limousines because of this?
What if we stick to nearest-neighbor interpolation? Essentially, it means replicating the pixels and creating big square blocks. Is this more reliable than bicubic?
Remember that when we have a limited number of pixels in the part of interest of a photo, it is because of a process called sampling, which essentially drops information from the real world.
Sampling is almost always followed by a compression phase (and a lot of other processing) that essentially removes even more data and creates artifacts as a side effect. During post-processing, even not interpolating at all, or sticking to the nearest neighbor algorithm, doesn’t guarantee that what we see in the picture at the pixel level is an accurate representation of the scene.
We always recommend comparing the results obtained with different interpolation algorithms and with the original image. If we have to analyze something made by a very limited number of pixels, especially in a single frame, there may not be enough information to distinguish details whether there is interpolation or not.
Furthermore, the human visual system does a lot of tricks that are out of our own control. An experienced analyst knows that and how to limit these effects in order to state what a video does or doesn’t show.
PROPER IMAGE ENHANCEMENT GIVES YOU A MORE ACCURATE REPRESENTATION
An image or video is an imperfect representation of the scene captured. With forensic image and video enhancement and restoration techniques, when done properly, we can provide a more faithful representation of the scene by reducing some defects introduced during the image generation process and amplifying details of interest.
Overly cautious analysts, or those who don’t have the adequate tools, experience, and preparation, may tell you not to enhance the image at all because they are afraid of getting questioned about it. The safest thing to do for them is just to play the video, if possible, and refrain from every possible attack. However, I think that the duty of a forensic video analyst is to do all that’s possible to help the trier of fact get the truth. Image science does not take a “side” in a case.
ENHANCEMENT WORKS, WHEN THERE IS SOMETHING TO ENHANCE
We can attenuate the defects of an image and amplify the information of interest, but we can only show better what’s already there. We can’t and we must not add new information to the image (as can potentially happen with AI techniques). My typical example is a white license plate made of 3 pixels; we’ll never be able to get anything from there, and whatever you could “believe” to read would be completely unreliable. The success of enhancement depends on the following factors:
- The technical characteristics of the image or video
- The purpose of the analysis (understanding the dynamics of an event is generally easier than identifying a person, for example)
- The technical preparation of the analyst
- The tools available for the analyst
To get some objective data, at the Amped User Day 2021, we asked our users how often they were having different levels of success with enhancement. The pie chart below summarizes the findings.
As it can be seen, according to our users, in almost 6 cases out of 10, the enhancement provides some form of benefit. When there’s no improvement, it could be either because of the lack of actual information in the image or because they weren’t able to get it. Often this situation can be improved with training or seeking the help of someone with more experience.
THE FOUR PRINCIPLES OF FORENSIC IMAGE AND VIDEO PROCESSING
There are just a few basic concepts to keep in mind, to properly process images and videos in a forensic context, around which we design our tools and teach our training.
- Have access to the original piece of evidence with a proper chain of custody and integrity verification (for example, with its hash code).
- Process and analyze the evidence with procedures that are reliable (as much as possible free from bias and errors) and accepted by the scientific community.
- Produce an output image or video.
- Produce a report that allows repeatability (by the same analyst) and reproducibility (by other analysts).
The same analyst, or an independent one with the appropriate preparation and tools, should be able to independently obtain an equivalent output image (3) from the input image (1), following the process (2) described in the report (4). All the procedures done and reported should be individually correct and applied in the right order.
It’s clear that you should arrive at the stand with all the evidence ready just to be played. There shouldn’t be the need to process or make calculations “on the fly”, and those that are asked to be done could be subject to acceptance or rejection. Even still, all your work should respect the same guidelines of reliability, repeatability, and reproducibility. Essentially it means accurately recording all the steps.
OUR ONLY GOAL: JUSTICE THROUGH SCIENCE
Anything can happen in a trial, things can go down a bad path, and prosecution or defense can be a mess, do a great job, or simply not play their cards well. Regardless of the circumstances, the duty of an expert witness is to help in the best possible way the trier of fact with objective and reliable techniques based on science, not on opinions and tricks. Our software is designed from scratch to help the analyst be transparent, objective, and unbiased. We are open about what we use and what we don’t use. If Amped FIVE has been used in 100 countries worldwide for more than thirteen years and has been widely accepted in court, it’s for a reason.
NOTE: This article has been later updated for clarity thanks to users’ feedback.
References
[1] Birkfellner, W. (2016). Applied Medical Image Processing: A Basic Course. United States: CRC Press. Page 215.
[2] Vaseghi, S. V. (2007). Multimedia Signal Processing: Theory and Applications in Speech, Music and Communications. Germany: Wiley. Page 166.
[3] Ledesma, S. A. (2015). A proposed framework for forensic image enhancement. University of Colorado at Denver. Page 45.
