The Untold Secrets of Forensic Video Enhancement: Myth versus Science

So you are probably reading this blog post because you have been watching too much CSI (Enhance! Uncrop! Flop!) or because you have tried Amped FIVE on your crappy DVR video and weren’t able to obtain the miracles that you can see on our samples page.

The point is that we don’t do miracles, we are just scientists. We work with number sequences that incidentally happen to represent some visual information called image or a signal called video.

Some people may even wonder if the images on our website’s samples page are fake and we artificially introduced defects on good images in order to show how the software works. We assure you that what you see are results of real pictures or video, most of them have been taken by myself to reproduce typical situations you will find working in forensic cases and with CCTV videos. Of course we are not going to put on the web confidential and personal data belonging to real cases that we receive from customers or that we are asked to analyze. We are a professional company, we keep those cases strictly confidential, so they never would appear as samples.

clip_image002


 

NOTHING CAN BE CREATED FROM NOTHING

We view more than a few hundred cases every year from customers, but even with the continuous improvement of image processing algorithms, very often we simply look at the video/image and realize that there is nothing to do. The bottom line is this: we are analyzing the picture to get some information (identifying a face, reading a license plate…); if this information is present in the image but not visible because of some defect, within certain limits and with the proper scientific procedure, we are able to recover it and make it visible. If the information is not there in the original footage, we cannot (and we must not, since we are in a forensic context and not the photo editing or art world) add it or recreate it.

More than half of the cases that we receive are quickly analyzed and then discarded because the enhancement is not feasible or will not bring any useful result. Without the necessary experience it is difficult to quickly evaluate the feasibility of a case and very often, potential customers evaluate it giving a completely wrong assessment. Sometimes a license plate is looking like it’s almost readable, so very little should be needed to make it readable, but actually it’s not possible to get anything. Other times situations that look hopeless can give amazing results. As a forensic video analyst working everyday on multiple cases, it’s very important to acquire the expertise to evaluate the feasibility in a fast manner in order not to waste precious time on activities which will not bring any results.

Many times, the fast workflow in Amped FIVE is a huge benefit if the image or video is a dead end. Most CCTV systems out there are optimized for minimal storage space – even though storage is very cheap today. Because of this, you have a very limited amount of time to recover data and highly compressed data. Your window can be as little as 12 hours, a real world maximum of one week, and an average of 48 hours. That means that if you can’t get a result that is actionable from the data initially recovered, you have a very limited amount of time to recover more data. That could be footage from another business across the street, next door, or city cameras at an intersection. The more time you spend “photoshopping” a dead end, the less time you have to get more data. Once it is gone, it is gone. Granted a few guys know the secret sauce of byte level recovery, but that is a rare skill set and very time consuming and expensive if you’re paying an expert to do it, and probably not worth it if your perpetrator just stole some $100 sneakers.


 

UNREALISTIC EXPECTATIONS

Often times we get inexperienced customers demanding that “if this works on this case, we will buy it” as their purchasing model. Can a random sequence of buttons be pushed to get a result? Maybe in a slot machine or other casino game, but not with forensic software. This is obviously not the right way to look at things, since you can’t get something from nothing. While we put many evaluation tools in Amped FIVE to demonstrate how it works (sample projects, tutorials, videos, etc.), none of that can take the place of training. Training is something that you need as a professional if you want to do a professional job. This demand for a result without training, understanding, or professional development isn’t realistic. When we hear this, we think: “What is your agency’s firearms policy? Do you give any new hire off the street a gun and tell them to sort it out for themselves?” Mmm… Probably not. Your agency mandates professional safety training, marksmanship qualification, testing, ongoing practice, etc. With forensic tools, the responsibility is the same since public safety is also at stake. Either not properly handling evidence, not knowing how and when to recover evidence, or using improper tools and techniques can either get the wrong guy arrested or the right guy out of jail for technical reasons. Both scenarios are bad. Real bad. Bottom line, get training either from us or from a professional organization like LEVA.


 

WHAT IS THE MINIMUM QUALITY OF THE CAMERA TO GET A SUCCESSFUL ENHANCEMENT?

This is one of the most common questions newcomers ask us. “Our evidence comes from someone who has all HD cameras so it should work”.

The reality is that we cannot give objective “minimum quality requirements” where we are sure to get something useful. The feasibility of the enhancement is a combination of many different factors.

1. The purpose of the enhancement (What I need to get?): for example to understand the kind or color of a car it is generally sufficient less quality than that needed to read its license plate.

2. The resolution of the area of interest (How many pixels I have?): in general we don’t care about the overall video resolution, but only how many pixels compose the target of the enhancement; it’s better to have a low resolution video with a license plate very close to the camera than an HD stream where the car we are interested in is very far.

3. Compression level (How “real” is the information in the pixels?): very often the excessive compression (both spatial, between pixels of the same frame, and temporal, between different frames) of surveillance videos is the main problem which does not allow us to apply with success image processing algorithms. Sometimes we theoretically have a very high resolution, but then all the pixels are grouped in blocks with the same value (loss of detail) or have strong compression artifacts (false details) or prediction errors, which make the actual resolution much lower. Furthermore, the nonlinearities introduced by the compression will impede the proper execution of image restoration algorithms since they are extraneous to the mathematical model. It’s counterintuitive, but the truth is that the image enhancement process was more successful in the VHS age, where tapes could be acquired without digital compression.

4. Number of frames (How many images I have?): the number of frames, the relative position of the subject in the different frames and the frame rate of the video are other very important aspects: the possibility of getting a good result is very much higher in cases where we have more frames to put together to increase the quality rather than with just a single snapshot.

5. General quality issues (How good are these pixels?): the presence and combination of noise, blur, brightness and color issues is another important factor. Among these, the most common problem is the saturation which happens on completely white areas (typical on license plates captured in night videos). Even here it is counterintuitive. Often in a dark video the main problem is the noise, that can be effectively reduced if you have enough frames, while a saturated sensor will not have any useful signal: no noise and no information.

Different combinations of these factors will cause different feasibility of the enhancement. For example:

  • From a single frame in which the license plate is made of 3 white pixels, it will never be possible to obtain anything.
  • From a very dark video, affected by strong noise, often it is possible to restore good detail if we have enough frames available.
  • Even with very strong blur, if the resolution is enough and the compression not too strong, it is possible to get a clean image.

SOME EXAMPLES

Let’s see a few samples:

CASE 1: NO WAY!

clip_image003

In this image the license plate is completely saturated (all pixels are white having a value of 255). Here it is not possible to get anything, there is no information at all.

CASE 2: NO WAY!

clip_image005

In this image the vertical resolution of the license plate is around 8 pixels (and there is also a strong compression) and it is thus too low to get any useful detail.

CASE 3: SUCCESS!

clip_image007One frame of the original video

clip_image009 One frame after intensity adjustment

clip_image011 Result of frame integration

In this case we are working on a very dark video: after adjusting the exposure and integrating more frames the license plate becomes readable. The image has low resolution and it’s very dark, but having many frames available makes it possible to keep what’s good in them while discarding the noise to obtain a successful enhancement.

CASE 4: SUCCESS!

clip_image013

clip_image015

In this case we have a license plate affected by motion blur. Even if we have a single frame and the blur is very strong, the resolution is good enough and the compression level not too bad, so we can make the license plate readable.


 

A RULE OF THUMB

For what regards the license plate enhancement, which is for sure one of the most common requests, according to our experience we can say that in a typical CCTV video if the vertical resolution is less than 12-15 pixels we cannot obtain any meaningful improvement. With very sharp images where no big defect is present, we can even go under 10 pixels, but these are the exception and not the norm. And, to put it clear, it is not always said that a 15 pixels tall license plate is useful, all the above factors must be considered.


 

EXPLAINED FOR THE AVERAGE JOE

Various CSI and investigative fiction have created unrealistic expectations for the general public but also for the unspecialized law enforcement operators. Very often it is difficult to explain to a judge, a prosecutor or a potential client that we cannot zoom in on that image to get everything like in the movies. Unfortunately from a few pixels it’s not possible to get anything, since data and information is not there at all. We can make the image more appealing to our eyes, but we cannot get new information. In order to explain the concept to a non-technical audience I usually use this practical example: let’s suppose we have a license plate which is 3 pixels tall. Since the character is higher than wider, every letter may have a 2×3 pixels resolution. Let’s try to draw by hand a letter or a number on a piece of paper with only 6 points on a 2×3 grid. Even without considering the limitations of the digital data and typical defects of video surveillance, we cannot in general draw a letter or a number in a way which is distinguishable by the others.

Let’s suppose this grid below represents pixels. Can you represent some readable number and letter by just filling the squares? And this is not even the worst case.

clip_image016


 

WHY USE A FORENSIC SOFTWARE?

So, many times you will not get what you need, why bother with a forensic software?

1. Sometimes you will still get great results. We cannot guarantee always amazing results, but luck is not always bad. Training is also a key element and improves your chances.

2. Enhancement is only a minimal part of the work of a forensic video analyst: acquisition, conversion, triage, photogrammetry, comparison… Amped FIVE is like a “Swiss Army Knife” of analysis software and supports many functions you run into every day.

3. Even if what usually attracts new users to our products are the exceptional samples on our website, in the long run the main benefit of working with Amped FIVE is the easy, fast and scientific workflow which allows you to work on cases simply and effectively. Even if you don’t get any result, you are sure to have tried everything, working in a scientific manner and have come to a conclusion (whether positive or negative) quickly so you have more critical time for the investigation.

Our most loyal customers are the expert analysts which have been working in this field for years, maybe even those who used other products. They know what life was like before and now they are… Amped up!