Forensic Image and Video Analysis: The complete workflow.

Amped Software is a pretty young company, but we have had the opportunity to work on almost 400 cases. We are mostly focused on software development, but from time to time, we are asked to work as expert witnesses for some major cases or we are asked by our customers to help them by evaluating (more or less informally) their material and providing some consultation.  These external cases have allowed us to see that there are steps that are commonly missed and others that are rarely taken into account in practical forensic video analysis.

Today, speaking to an agency about a potential project, I realized just how broad and complex the complete workflow process is for our (and our users) job.

If we think about all the tasks related to analysis, it can be really overwhelming.  To stay on point, I usually list on paper all the possible steps we need to take to do a really complete analysis.  This way, I can stay organized and minimize the possibility of skipping or missing steps.  Also, if I do have to go to court, I have my outline that serves the basis of my presentation.

We always need to remember that our job doesn’t start and end with viewing and enhancing a video.  It’s more complex: we must ID the data, decode it properly, document the process, compare it with other material, and then go to court.  Since digital data is really just a collection of bits, we outline our process around working with these bits and answering key questions relating to what we need to do with the bits.

Step 1. Retrieve the data: get the bits!

This step may be most important and can be very complex and critical for evidence integrity.  Unfortunately, this is generally not done by us, or by those who know to take the care to document the steps.  Worst case (which seems to be the most common for police investigators), the original evidence is gathered by a patrol officer who is the first on the scene.  A low-res copy of an original is output to a disk or thumb-drive, and the original DVR is left to be copied over by the next day’s footage.  This happens far too frequently, and gets everyone off to a pretty bad start.

Another way we may get the data is if it comes from our colleagues on the digital forensics side of things. If they are meticulous and take the steps necessary to document their process, we can be in a good position to start.  It seems that often we start working on a video burned on DVD or on some images we receive by email, however we need to be conscious and document the previous steps, which may be, for example:

  • Analyze a disk image for complete and deleted files
  • Export a video from a DVR
  • Copy a file from an hard disk
  • Capture a video from an analog device (e.g. VHS)

Even a simple operation like a copy should be done in the most scientifically relevant way by verifying hash codes and so on.

For police investigators working in the real world case of evidence gathered by a non-scientific responder, with a little better communication, training, and diplomacy; the situation can be improved.  Let them know what you need them to gather, document, and what instructions to leave with the video source (like don’t copy over the DVR drive!!!) At the very least have them know to call you if they have any questions.

Step 2. Decoding the data: what are these bits?

Now that we have the bits, we must get something from them to analyze. If we have some plain video files, it is not a big problem, given we have the proper codecs. Since there are hundreds of proprietary video files out in the wild, that in itself may turn into a challenge.  To determine which approach we take, we need to know what we are looking at.  This step generally defines the challenges that we may be facing:

  • We have a disk image and we need to reconstruct images and videos, even the ones that may have been deleted (this is common in child pornography cases, for example).
  • We have a dump of a DVR drive and don’t know how the data is encoded.
  • We have an export of surveillance footage, but the video is in a proprietary format (a very common situation).  Far too often the video player given by the system producer is full of bugs, unusable, incompatible with modern versions of Windows and videos can’t be properly exported.

With this step, we may need to do some research to really define what we need to complete the job.  Do we need to focus first on data recovery?  Do we need to find a better decoder?  Do we need to find a better player for the video or means to export it properly?

Sometimes, this is an easy step, other times it is far more difficult and time consuming.  Remember, this is a scientific pursuit and science sometimes can’t be rushed.

Step 3. Finding the useful data: where are the right bits?

At this step we should be able to view the videos or the images, but we need to find the right ones! Two examples of what we may face may be:

  • Finding images of interest in a large database
  • Looking for an event of interest in hours and hours of video

This step can be helped with communication from other team member working on this case.  For most cases, the basic thing we need to understand is: what happened, and when did it happen? And of course, technology may help too, with technologies such as video content analysis and face recognition.

Step 4. Finding the source of data: where do these bits come from?

Depending on the situation we may need to understand how the original files have been generated. With some generalization this may be called image ballistics. Understanding the type of file and the source can help us understand several things in a case. Some analysis that may be done could be:

  • Identify the type of source (digital camera, scanned image, computer generated…)
  • Identify the camera model used for taking the picture
  • Identify the specific device that has taken the picture

We need to document the source so that we can maintain the integrity of our evidence.  This will help us if we have to go to court later.

Step 5. Verify the integrity of the data: has someone tampered with the bits?

At this point we may be interested in understanding if we can trust the data we have gathered.  Is there a probability that someone altered it?  This can be done on various levels.

  • Verify if the file the file has been manipulated, for example altering the metadata
  • Verify if the image has been manipulated, for example converting the format, resizing or cropping it
  • Verify if the content has been manipulated, for example removing or adding a subject

Tampering is becoming more commonplace.  It can be done innocently (like converting formats from original to a low-res media file) or purposeful “photoshopping” to manipulate facts.  In this digital age, it is something that we need to address.  This is why anyone who has ever spoken to anybody here at Amped will attest to something that we always emphasize: Work with an original if possible.

Step 6. Estimate the quality of the data: do we have enough bits?

At this point we can see something in the image, but we must understand if the quality is enough for our purposes. For example, if we see a car, are able to read the license plate?  Or if we have a face, do we have enough pixels for a reliable identification?

  • Does the image effectively contains the information we need (e.g. the license plate has enough pixels)?
  • If not, can the information be recovered or viewed better with image enhancement or image restoration techniques?
  • What are the specific defects in the image? Can they be recovered?

Technical knowledge and experience is very important to estimate quickly if we have enough quality or not. It is not always easy to estimate the minimum quality to get useful results.  A shortcut with things like faces and license plates is zooming in and counting pixels.  If you only have six or eight pixels to draw all the characters in a license plate, the probability of success is pretty low here.

Step 7. Enhance the data: get out the good bits!

Once we have identified the problems affecting the images or videos, given the right tools (Amped FIVE, obviously), we can enhance and restore the data. This step is actually pretty vast, and can involve processes like:

  • Image enhancement techniques: emphasize (or reduce) some features of interest of the image (contrast enhancement, histogram equalization, sharpening…)
  • Image restoration techniques: understand the mathematical model of a known disturb and try to invert the model to recover the image without the defect (deblurring, Fourier filtering, frame integration…)

For us hardcore video enhancement people, this is the most fun part of the process.  It does involve a bit trial and error, but it is at least the fun part.  While no one can guarantee the great results found with Hollywood magic on the CSI shows, we can often see some amazing results.

One thing that is really important to remember in this step is documenting the enhancement process.  Amped Five Professional and On Demand do this automatically, so you have documentation to take to court.  General purpose software does not usually provide enough tools for this issue.

Step 8. Analyze and compare the data: what do the bits represent?

This is where you see what you have gained.  The enhancement step would be useless if there’s no improvement to the content of the image so we can understand and classify it. In this step you can do the following:

  • Compare a face in two different images
  • Compare a face with a known subject
  • Read the license of a vehicle
  • Identify the place where a picture is take
  • Measure the height of a subject
  • Find the corresponding fingerprints in a database

If you don’t get the results you need, you can go back and repeat the steps until you do; or you determine that you can’t get what you need with the data provided.  Again, this is a scientific process and should be left without emotion.  I know it is too easy to get bent out of shape over a ton of work without results; but that is sometimes the cards that are dealt.

Step 9. Validation: did I get the right bits?

Validation isn’t just focused on the quality of the result.  It is also about the quality of the process used to gain the result.  We must always maintain that the techniques used must be valid both from the scientific point of view and follow a procedural set of standards accepted by the courts that have jurisdiction over what we are doing.  This is extremely important for the verification of image integrity and for documenting the enhancement workflow.

A few things to consider are:

  • State of the art techniques must be validated by peer review and accepted by the scientific communities.
  • The results must be scientific and repeatable.
  • A detailed audit trail must be kept to explain how we go from the original image to the enhanced one.

This may be seen (and actually is) as manipulation of evidence, and thus we must be able to justify it properly from the scientific point of view.  In this case, documentation is key.

Step 10. Presentation: I’ll show you the right bits!

Getting the results it’s not enough. You must explain them to the court and the jury: you must be able to make them understand and accept the techniques you used. Scientists, engineers, attorneys and normal people speak different languages. It is really important to organize any facts you present with the idea that you must explain the terms you use in your context, be clear and open.

A good defense attorney/prosecutor will try to trip you up with the “has this image been photoshopped?” question and questions about certainty, possibility, etc. That is their job, and frankly would you respect them if they didn’t ask tough questions? An attorney isn’t a scientist.  The courtroom is a stage and the attorneys are actors.  Their questioning tends to be less focused on science, more focused on emotion.  In court, emotions are often charged and an attack on your process can be presented as a personal attack with the idea to get you to deviate from facts.  That is the game-plan for attorneys when they can’t debate the facts.

The key to overcoming this type of questioning is to document your workflow and stick to the science. Remember I mentioned organization?  Again, I mention that this is scientific and a scientific process; but it has to be explained to non-scientific people.   Can you present the facts of the case and explain the science in plain language?  If you are organized, clear, and concise this will help.  At the very minimum, you need to show:

  • The original.
  • Where it came from.
  • How you got it.
  • What steps were taken to get the result.
  • How that result relates to the case.
  • What scientific methods were used to validate the process.

It’s not an easy task, as very often complex matters are oversimplified to be understood by laymen and at the end of the day the work of the expert witness can amount to nothing.

The bottom line

You may not use all of these steps in all cases.  Often, you may work as a team and only concentrate on a couple of them.  Again, the point to drag away from this whole outline of workflow is organization and methodology of your process.

Depending on what we are working with, we may not formally define these steps so clearly.  Critical and systematic thinking are really the foundation for what we do.  When documenting the workflow, I am reminded daily of the old Gen. Dwight Eisenhower maxim: “those who fail to plan, plan to fail”.