Forensic Image and Video Analysis: The complete workflow.

Amped Software is a pretty young company, but we have had the opportunity to work on almost 400 cases. We are mostly focused on software development, but from time to time, we are asked to work as expert witnesses for some major cases or we are asked by our customers to help them by evaluating (more or less informally) their material and providing some consultation.  These external cases have allowed us to see that there are steps that are commonly missed and others that are rarely taken into account in practical forensic video analysis.

Today, speaking to an agency about a potential project, I realized just how broad and complex the complete workflow process is for our (and our users) job.

If we think about all the tasks related to analysis, it can be really overwhelming.  To stay on point, I usually list on paper all the possible steps we need to take to do a really complete analysis.  This way, I can stay organized and minimize the possibility of skipping or missing steps.  Also, if I do have to go to court, I have my outline that serves the basis of my presentation.

We always need to remember that our job doesn’t start and end with viewing and enhancing a video.  It’s more complex: we must ID the data, decode it properly, document the process, compare it with other material, and then go to court.  Since digital data is really just a collection of bits, we outline our process around working with these bits and answering key questions relating to what we need to do with the bits.

Step 1. Retrieve the data: get the bits!

This step may be most important and can be very complex and critical for evidence integrity.  Unfortunately, this is generally not done by us, or by those who know to take the care to document the steps.  Worst case (which seems to be the most common for police investigators), the original evidence is gathered by a patrol officer who is the first on the scene.  A low-res copy of an original is output to a disk or thumb-drive, and the original DVR is left to be copied over by the next day’s footage.  This happens far too frequently, and gets everyone off to a pretty bad start.

Another way we may get the data is if it comes from our colleagues on the digital forensics side of things. If they are meticulous and take the steps necessary to document their process, we can be in a good position to start.  It seems that often we start working on a video burned on DVD or on some images we receive by email, however we need to be conscious and document the previous steps, which may be, for example:

• Analyze a disk image for complete and deleted files
• Export a video from a DVR
• Copy a file from an hard disk
• Capture a video from an analog device (e.g. VHS)

Even a simple operation like a copy should be done in the most scientifically relevant way by verifying hash codes and so on.

For police investigators working in the real world case of evidence gathered by a non-scientific responder, with a little better communication, training, and diplomacy; the situation can be improved.  Let them know what you need them to gather, document, and what instructions to leave with the video source (like don’t copy over the DVR drive!!!) At the very least have them know to call you if they have any questions.

Step 2. Decoding the data: what are these bits?

Now that we have the bits, we must get something from them to analyze. If we have some plain video files, it is not a big problem, given we have the proper codecs. Since there are hundreds of proprietary video files out in the wild, that in itself may turn into a challenge.  To determine which approach we take, we need to know what we are looking at.  This step generally defines the challenges that we may be facing:

• We have a disk image and we need to reconstruct images and videos, even the ones that may have been deleted (this is common in child pornography cases, for example).
• We have a dump of a DVR drive and don’t know how the data is encoded.
• We have an export of surveillance footage, but the video is in a proprietary format (a very common situation).  Far too often the video player given by the system producer is full of bugs, unusable, incompatible with modern versions of Windows and videos can’t be properly exported.

With this step, we may need to do some research to really define what we need to complete the job.  Do we need to focus first on data recovery?  Do we need to find a better decoder?  Do we need to find a better player for the video or means to export it properly?

Sometimes, this is an easy step, other times it is far more difficult and time consuming.  Remember, this is a scientific pursuit and science sometimes can’t be rushed.

Step 3. Finding the useful data: where are the right bits?

At this step we should be able to view the videos or the images, but we need to find the right ones! Two examples of what we may face may be:

• Finding images of interest in a large database
• Looking for an event of interest in hours and hours of video

This step can be helped with communication from other team member working on this case.  For most cases, the basic thing we need to understand is: what happened, and when did it happen? And of course, technology may help too, with technologies such as video content analysis and face recognition.

Step 4. Finding the source of data: where do these bits come from?

Depending on the situation we may need to understand how the original files have been generated. With some generalization this may be called image ballistics. Understanding the type of file and the source can help us understand several things in a case. Some analysis that may be done could be:

• Identify the type of source (digital camera, scanned image, computer generated…)
• Identify the camera model used for taking the picture
• Identify the specific device that has taken the picture

We need to document the source so that we can maintain the integrity of our evidence.  This will help us if we have to go to court later.

Step 5. Verify the integrity of the data: has someone tampered with the bits?

At this point we may be interested in understanding if we can trust the data we have gathered.  Is there a probability that someone altered it?  This can be done on various levels.

• Verify if the file the file has been manipulated, for example altering the metadata
• Verify if the image has been manipulated, for example converting the format, resizing or cropping it
• Verify if the content has been manipulated, for example removing or adding a subject

Tampering is becoming more commonplace.  It can be done innocently (like converting formats from original to a low-res media file) or purposeful “photoshopping” to manipulate facts.  In this digital age, it is something that we need to address.  This is why anyone who has ever spoken to anybody here at Amped will attest to something that we always emphasize: Work with an original if possible.

Step 6. Estimate the quality of the data: do we have enough bits?

At this point we can see something in the image, but we must understand if the quality is enough for our purposes. For example, if we see a car, are able to read the license plate?  Or if we have a face, do we have enough pixels for a reliable identification?

• Does the image effectively contains the information we need (e.g. the license plate has enough pixels)?
• If not, can the information be recovered or viewed better with image enhancement or image restoration techniques?
• What are the specific defects in the image? Can they be recovered?

Technical knowledge and experience is very important to estimate quickly if we have enough quality or not. It is not always easy to estimate the minimum quality to get useful results.  A shortcut with things like faces and license plates is zooming in and counting pixels.  If you only have six or eight pixels to draw all the characters in a license plate, the probability of success is pretty low here.

Step 7. Enhance the data: get out the good bits!

Once we have identified the problems affecting the images or videos, given the right tools (Amped FIVE, obviously), we can enhance and restore the data. This step is actually pretty vast, and can involve processes like:

• Image enhancement techniques: emphasize (or reduce) some features of interest of the image (contrast enhancement, histogram equalization, sharpening…)
• Image restoration techniques: understand the mathematical model of a known disturb and try to invert the model to recover the image without the defect (deblurring, Fourier filtering, frame integration…)

For us hardcore video enhancement people, this is the most fun part of the process.  It does involve a bit trial and error, but it is at least the fun part.  While no one can guarantee the great results found with Hollywood magic on the CSI shows, we can often see some amazing results.

One thing that is really important to remember in this step is documenting the enhancement process.  Amped Five Professional and On Demand do this automatically, so you have documentation to take to court.  General purpose software does not usually provide enough tools for this issue.

Step 8. Analyze and compare the data: what do the bits represent?

This is where you see what you have gained.  The enhancement step would be useless if there’s no improvement to the content of the image so we can understand and classify it. In this step you can do the following:

• Compare a face in two different images
• Compare a face with a known subject
• Read the license of a vehicle
• Identify the place where a picture is take
• Measure the height of a subject
• Find the corresponding fingerprints in a database

If you don’t get the results you need, you can go back and repeat the steps until you do; or you determine that you can’t get what you need with the data provided.  Again, this is a scientific process and should be left without emotion.  I know it is too easy to get bent out of shape over a ton of work without results; but that is sometimes the cards that are dealt.

Step 9. Validation: did I get the right bits?

Validation isn’t just focused on the quality of the result.  It is also about the quality of the process used to gain the result.  We must always maintain that the techniques used must be valid both from the scientific point of view and follow a procedural set of standards accepted by the courts that have jurisdiction over what we are doing.  This is extremely important for the verification of image integrity and for documenting the enhancement workflow.

A few things to consider are:

• State of the art techniques must be validated by peer review and accepted by the scientific communities.
• The results must be scientific and repeatable.
• A detailed audit trail must be kept to explain how we go from the original image to the enhanced one.

This may be seen (and actually is) as manipulation of evidence, and thus we must be able to justify it properly from the scientific point of view.  In this case, documentation is key.

Step 10. Presentation: I’ll show you the right bits!

Getting the results it’s not enough. You must explain them to the court and the jury: you must be able to make them understand and accept the techniques you used. Scientists, engineers, attorneys and normal people speak different languages. It is really important to organize any facts you present with the idea that you must explain the terms you use in your context, be clear and open.

A good defense attorney/prosecutor will try to trip you up with the “has this image been photoshopped?” question and questions about certainty, possibility, etc. That is their job, and frankly would you respect them if they didn’t ask tough questions? An attorney isn’t a scientist.  The courtroom is a stage and the attorneys are actors.  Their questioning tends to be less focused on science, more focused on emotion.  In court, emotions are often charged and an attack on your process can be presented as a personal attack with the idea to get you to deviate from facts.  That is the game-plan for attorneys when they can’t debate the facts.

The key to overcoming this type of questioning is to document your workflow and stick to the science. Remember I mentioned organization?  Again, I mention that this is scientific and a scientific process; but it has to be explained to non-scientific people.   Can you present the facts of the case and explain the science in plain language?  If you are organized, clear, and concise this will help.  At the very minimum, you need to show:

• The original.
• Where it came from.
• How you got it.
• What steps were taken to get the result.
• How that result relates to the case.
• What scientific methods were used to validate the process.

It’s not an easy task, as very often complex matters are oversimplified to be understood by laymen and at the end of the day the work of the expert witness can amount to nothing.

The bottom line

You may not use all of these steps in all cases.  Often, you may work as a team and only concentrate on a couple of them.  Again, the point to drag away from this whole outline of workflow is organization and methodology of your process.

Depending on what we are working with, we may not formally define these steps so clearly.  Critical and systematic thinking are really the foundation for what we do.  When documenting the workflow, I am reminded daily of the old Gen. Dwight Eisenhower maxim: “those who fail to plan, plan to fail”.

Amped Five introduction video

We’ve just uploaded to YouTube two new videos, which show a quick introduction on how to work with Amped Five in some common cases. These videos are in Italian, and the English version should be added soon.

Part 1.

Topics covered:

• restoring a dark license plate with frame integration and stabilization
• deblurring a license plate
• deblurring a face
• enhancing a face behind a glass window

Part 2.

Topics covered:

• measuring a person’s height from an image
• extracting a latent fingerprint from a banknote
• correcting the perspective of a license plate

What’s the problem with this license plate?

I find myself analyzing new surveillance videos almost every day, and in most of the cases I can either solve the problem very quickly or understand (even more quickly) that there is no information to recover in the video.
In special cases, where something very specific and strange happened; or the problem is very complex it can take a lot of time.

As always… Pareto principle: you solve 80% of the cases in 20% of the time, and, well, 20% of the cases takes 80% of the time.

In my own work probably the right numbers are 95% to 5%, but the idea holds.

With my experience on several thousands cases, I can estimate whether an image or video contains some information and it’s worth processing, or not, very quickly. I usually do a couple of tests, just to be sure of my first glance and sometimes I get surprised by the power of the algorithms; anyway, usually you can tell quite fast if you will get something from a video.

The big picture

Let’s work on the most common case: enhancing a license plate. Of course, I won’t show you images from real cases so I’ll generate some samples for you.

Usually the main problems why we can’t see a license plate (or anything else useful) are:

• low resolution
• lossy compression
• optical blur
• motion blur
• low brightness / contrast
• noise

These, in order, are the most common cases. In most situations you have all of them combined together. One single of these problems, if strong enough, may cause your license plate to be unreadable.

Let’s see in detail all this issues and what we can do to correct them.

These are the problems related more strictly to the image quality, please note that there are also other problems that maybe be corrected more easily: for example a bad perspective or the interlacing are immediate to solve, if the overall quality of the image is not too bad.

A good quality license plate

Low resolution

The idea is pretty simple: the resolution determines the number of points you have in an image and if you don’t have enough points for drawing a digit, you won’t be able to see it.

How to recognize it: zoom in the license plate and count, for example, how tall is it in pixels. If the license plate is, for example 3 pixels tall, just think how can you draw a number with just three points (maybe two because it will also have the plate border).

How to solve it: if the problem is very relevant there is not very much to do. In intermediate situation a simple digital zoom (the filter Resize in Amped FIVE) can be helpful; although it does not actually add information to the image, it will make characters and details better recognizable to the human eye. If you have multiple frames applying techniques such as frame integration (Frame Averaging) or super resolution (currently in development) can be effective in some situations.

Low resolution

With digital zoom

Lossy compression

Lossy compression is the main drawback of digital surveillance systems if compared to the old good VCR. Very often it happens that the resolution is actually pretty good but the compression makes the digits unreadable. Videos are compressed in order to save bandwidth in transmission and space in the storage systems.

How to recognize it: The typical example is when the picture looks like you can see the license plate if you zoom, then you zoom and… you can’t still read the license plate.

The problem appears in two ways:

• you have enough pixels, but many of them appear to have the same value (tile effect), so in practice you have a much lower actual usable resolution that the native resolution of the image file format
• you have artifacts, that means details that do not belong to the image but are introduced by the compression (typical blocking artifacts of JPEG files)

How to solve it: With a deblocking filter (Deblocking in FIVE) you can noticeably improve the quality of the image from a visual point of view.  It is important to take care not to apply too strong a filter that will also cut out useful detail. Also here, on more frames/ frame integration / averaging can be very useful.

Compressed image

With deblocking

Optical Blur

The optical blur happens when an image is not perfectly in focus. Usually CCTV optics are configured to have a very big depth of field, everything is more or less in focus, but it’s difficult to have something perfectly in focus.

The basic idea is that every pixel of an ideal image in focus is being transformed into a circle with a radius depending on the strength of the blur.

How to recognize it: The border and the digits of the license plate are (guess what) blurred. Basically if you just see a white area it is often blur together with a lack of contrast.

How to solve it: If you have enough resolution, optical deblurring filters can be very effective to recover even license plates that are not visible at all. Amped FIVE provides the Optical Deblurring and Blind Deconvolution filters: the first works very well on big blurs (even some tenths of pixels), while the second works more effectively on little blurs (less than 10 pixels usually).

A big problem is that the deblurring won’t work very well in images that present a strong compression, since it introduces some non-linearities that the algorithm is not able to manage.

Image out of focus

Deblurred image

Motion Blur

The motion blur is the effect that appears on a subject that moves to fast in relation to the shutter speed. Even if the effect looks much different than optical blur, the nature of the problem is very similar: every point becomes a line (instead of a circle like to optical blur).

How to recognize it: Usually it’s pretty noticeable since you can distinguish a kind of trace that follows the points in the images.

How to solve it: If the kind of motion is linear and uniform, usually we can take some reference point that we know should be a point (like the angle of a license plate or a shimmer), but that in the image clearly appears like a segment. We can estimate the features of the blur through the length and the angle of this segment. In FIVE you can use the Motion Deblurring filter.

If the resolution is not too bad and the compression is not exaggerated we may be able to correct even very strong blurs.

Moving car

Deblurred image

Low brightness / contrast

Different light conditions or characteristics of the cameras may lead to images that are either too dark (for example in night shots), too bright (when we have a light that points in our direction) or simply lacks contrast (pixel values are too similar).

How to recognize it: You can look at the image histogram (in FIVE Tools / Frame Info panel). If the histogram is very shifted to the left, the image is too dark (most of the pixels will have values close to zero). If the histogram is shifted to the right the image is too bright (values close to 255). If the histogram is very thin the problem is in the contrast (too many pixels have the same value).

How to solve it: There are many tools, the use of which can vary from case to case.  Two good starting points are that you can automatically equalize the histogram (Histogram Equalization in FIVE) or manually adjust the brightness (filter Curves or Contrast /Brightness). In order to improve the appearance of the details you can use on of the various available sharpening filters (Sharpen group).

An area of the image is saturated when all the pixels have values of zero or 255. If the image is saturated (or, in general, all the pixels have the same value), there is nothing to do, since all useful information is gone.

Low contrast

With histogram equalization

Noise

The noise is a random variation of the pixels in an image and it appears like a grain that is usually very evident in pictures taken with low light. The noise becomes very visible usually when we brighten a dark video.

How to recognize it: There are random spots on the image that modify the correct value of the pixel. Many times it’s difficult to precisely separate noise and compression artifacts.

How to solve it: If we have only a single image we can improve the overall quality with one of the many smoothing filters (in FIVE in the group Denoise). However, on a single frame this is a risky approach because many times the filters can take away the details together with the noise. If we have multiple frames of the same license plate in the same position, a technique that gives amazing results is frame integration (Frame Averaging filter). If the license plate is not in the same position it must be aligned with a stabilization filter (Stabilization or Perspective Registration).

One noisy frame

After integration

The bottom line

Real cases are always more complicated than these examples, since most (if not all) of these problems will be present at the same time. It’s very important to keep these basic problems in mind in order to always start with the proper filters and algorithms to correct them.  It is an even more important concept to understand to determine when an image does not contain the minimum amount of information for feasible analysis. You can see some more real-world examples on Amped FIVE samples page.

Amped Five philosophy

In this post I’ll briefly explain the main idea that is behind Amped Five and how it will improve your work as a forensic video analyst.

Every time we show Amped Five to a person who is not exactly an expert on video analysis, but who has a keen interest, or works with video evidence (such a general law enforcement inspector or detective) he’s impressed with the samples and how effective Amped Five filters are. But, actually, what makes us even more proud is when real experts appreciate our work. They may be aware of other products in the market and, while there are in Five many features that are not present in other products, what amazes them is the workflow.

Yes, the workflow.

Maybe at the first instance they’ll be disappointed that Five is not behaving exactly like the Photoshop version or NLE video editor that they are used to, but once a few key concepts are understood, everything is much easier and faster.  Above all, the user feels in complete control.

The main idea is that absolutely everything you can do on an image or a video is done using a general object called a filter. A filter can be as simple as a luminance brightness adjustment or the selection of special frames of interest in a video, or the tool to measure an unknown height in an image.

The first filter is always a loader, so basically it loads some data from a file. Then you add another filter that accepts the loader as input and produces a processed image or video as output. That’s all. Filters can be moved, copied, pasted. Once you have a stack of filters you can modify any filter in any position (or modify its position) and directly, in real time, even while the video is playing, to see how the output changes.

What are the main advantages of this workflow:

• It is very easily test different combinations of filters until the desired result is reached
• The user is always in command and can feel the control of every single step of the processing
• Work is done very, very, very fast

If a picture is worth a thousand words, how many words is a video worth? Click on the video below to find out.  See the effectiveness of the processing in one of our YouTube screencasts.