The advantages of using a specialized forensic image processing software

Despite being successful also for many other applications, Amped FIVE has been specifically designed for forensic specialists and law enforcement groups working mainly on video footage  from video-surveillance.

What’s the traditional approach for working with surevillance video?

1. A crime happens
2. If there are CCTV cameras in the neighborhood go there and pick up the videos of interest along with the proper player
3. If the video has some good quality frames of for the purpose of person or vehicle identification, we save it as image for further investigations

1. Work in one program for both images and videos.  The same interface and you can always work with the original.  No extra rendering or conversion steps and loss of resolution.
2. Have all the needed filters both for common and less common cases = More crimes solved
3. Employ a coherent workflow specifically designed for forensic needs and documenting evidence for the courtroom.  You can be confident that your work is valid from the scientific point of view
4. Save time and money: A single software program with all the needed tools allows to easily save 90% of the time in common cases. Even if the quality is too bad to get anything, you have lost minutes, not hours or days, and you are sure to have tried all the possibilities.
5. Bottom line: Solve more cases.  When promotion time rolls around, who looks at the numbers? You know who!

Amped Five introduction video

We’ve just uploaded to YouTube two new videos, which show a quick introduction on how to work with Amped Five in some common cases. These videos are in Italian, and the English version should be added soon.

Part 1.

Topics covered:

• restoring a dark license plate with frame integration and stabilization
• deblurring a license plate
• deblurring a face
• enhancing a face behind a glass window

Part 2.

Topics covered:

• measuring a person’s height from an image
• extracting a latent fingerprint from a banknote
• correcting the perspective of a license plate

What’s the problem with this license plate?

I find myself analyzing new surveillance videos almost every day, and in most of the cases I can either solve the problem very quickly or understand (even more quickly) that there is no information to recover in the video.
In special cases, where something very specific and strange happened; or the problem is very complex it can take a lot of time.

As always… Pareto principle: you solve 80% of the cases in 20% of the time, and, well, 20% of the cases takes 80% of the time.

In my own work probably the right numbers are 95% to 5%, but the idea holds.

With my experience on several thousands cases, I can estimate whether an image or video contains some information and it’s worth processing, or not, very quickly. I usually do a couple of tests, just to be sure of my first glance and sometimes I get surprised by the power of the algorithms; anyway, usually you can tell quite fast if you will get something from a video.

The big picture

Let’s work on the most common case: enhancing a license plate. Of course, I won’t show you images from real cases so I’ll generate some samples for you.

Usually the main problems why we can’t see a license plate (or anything else useful) are:

• low resolution
• lossy compression
• optical blur
• motion blur
• low brightness / contrast
• noise

These, in order, are the most common cases. In most situations you have all of them combined together. One single of these problems, if strong enough, may cause your license plate to be unreadable.

Let’s see in detail all this issues and what we can do to correct them.

These are the problems related more strictly to the image quality, please note that there are also other problems that maybe be corrected more easily: for example a bad perspective or the interlacing are immediate to solve, if the overall quality of the image is not too bad.

A good quality license plate

Low resolution

The idea is pretty simple: the resolution determines the number of points you have in an image and if you don’t have enough points for drawing a digit, you won’t be able to see it.

How to recognize it: zoom in the license plate and count, for example, how tall is it in pixels. If the license plate is, for example 3 pixels tall, just think how can you draw a number with just three points (maybe two because it will also have the plate border).

How to solve it: if the problem is very relevant there is not very much to do. In intermediate situation a simple digital zoom (the filter Resize in Amped FIVE) can be helpful; although it does not actually add information to the image, it will make characters and details better recognizable to the human eye. If you have multiple frames applying techniques such as frame integration (Frame Averaging) or super resolution (currently in development) can be effective in some situations.

Low resolution

With digital zoom

Lossy compression

Lossy compression is the main drawback of digital surveillance systems if compared to the old good VCR. Very often it happens that the resolution is actually pretty good but the compression makes the digits unreadable. Videos are compressed in order to save bandwidth in transmission and space in the storage systems.

How to recognize it: The typical example is when the picture looks like you can see the license plate if you zoom, then you zoom and… you can’t still read the license plate.

The problem appears in two ways:

• you have enough pixels, but many of them appear to have the same value (tile effect), so in practice you have a much lower actual usable resolution that the native resolution of the image file format
• you have artifacts, that means details that do not belong to the image but are introduced by the compression (typical blocking artifacts of JPEG files)

How to solve it: With a deblocking filter (Deblocking in FIVE) you can noticeably improve the quality of the image from a visual point of view.  It is important to take care not to apply too strong a filter that will also cut out useful detail. Also here, on more frames/ frame integration / averaging can be very useful.

Compressed image

With deblocking

Optical Blur

The optical blur happens when an image is not perfectly in focus. Usually CCTV optics are configured to have a very big depth of field, everything is more or less in focus, but it’s difficult to have something perfectly in focus.

The basic idea is that every pixel of an ideal image in focus is being transformed into a circle with a radius depending on the strength of the blur.

How to recognize it: The border and the digits of the license plate are (guess what) blurred. Basically if you just see a white area it is often blur together with a lack of contrast.

How to solve it: If you have enough resolution, optical deblurring filters can be very effective to recover even license plates that are not visible at all. Amped FIVE provides the Optical Deblurring and Blind Deconvolution filters: the first works very well on big blurs (even some tenths of pixels), while the second works more effectively on little blurs (less than 10 pixels usually).

A big problem is that the deblurring won’t work very well in images that present a strong compression, since it introduces some non-linearities that the algorithm is not able to manage.

Image out of focus

Deblurred image

Motion Blur

The motion blur is the effect that appears on a subject that moves to fast in relation to the shutter speed. Even if the effect looks much different than optical blur, the nature of the problem is very similar: every point becomes a line (instead of a circle like to optical blur).

How to recognize it: Usually it’s pretty noticeable since you can distinguish a kind of trace that follows the points in the images.

How to solve it: If the kind of motion is linear and uniform, usually we can take some reference point that we know should be a point (like the angle of a license plate or a shimmer), but that in the image clearly appears like a segment. We can estimate the features of the blur through the length and the angle of this segment. In FIVE you can use the Motion Deblurring filter.

If the resolution is not too bad and the compression is not exaggerated we may be able to correct even very strong blurs.

Moving car

Deblurred image

Low brightness / contrast

Different light conditions or characteristics of the cameras may lead to images that are either too dark (for example in night shots), too bright (when we have a light that points in our direction) or simply lacks contrast (pixel values are too similar).

How to recognize it: You can look at the image histogram (in FIVE Tools / Frame Info panel). If the histogram is very shifted to the left, the image is too dark (most of the pixels will have values close to zero). If the histogram is shifted to the right the image is too bright (values close to 255). If the histogram is very thin the problem is in the contrast (too many pixels have the same value).

How to solve it: There are many tools, the use of which can vary from case to case.  Two good starting points are that you can automatically equalize the histogram (Histogram Equalization in FIVE) or manually adjust the brightness (filter Curves or Contrast /Brightness). In order to improve the appearance of the details you can use on of the various available sharpening filters (Sharpen group).

An area of the image is saturated when all the pixels have values of zero or 255. If the image is saturated (or, in general, all the pixels have the same value), there is nothing to do, since all useful information is gone.

Low contrast

With histogram equalization

Noise

The noise is a random variation of the pixels in an image and it appears like a grain that is usually very evident in pictures taken with low light. The noise becomes very visible usually when we brighten a dark video.

How to recognize it: There are random spots on the image that modify the correct value of the pixel. Many times it’s difficult to precisely separate noise and compression artifacts.

How to solve it: If we have only a single image we can improve the overall quality with one of the many smoothing filters (in FIVE in the group Denoise). However, on a single frame this is a risky approach because many times the filters can take away the details together with the noise. If we have multiple frames of the same license plate in the same position, a technique that gives amazing results is frame integration (Frame Averaging filter). If the license plate is not in the same position it must be aligned with a stabilization filter (Stabilization or Perspective Registration).

One noisy frame

After integration

The bottom line

Real cases are always more complicated than these examples, since most (if not all) of these problems will be present at the same time. It’s very important to keep these basic problems in mind in order to always start with the proper filters and algorithms to correct them.  It is an even more important concept to understand to determine when an image does not contain the minimum amount of information for feasible analysis. You can see some more real-world examples on Amped FIVE samples page.

Extracting a license plate from a dark video [How to]

This is the first video tutorial of a series. It has been ripped and then subtitled directly without any editing, so it shows Amped Five work flow on real time.

The video is showing a very dark and shaking sequence of a license plate. The steps illustrated are:

1. loading the video (Video Loader filter)
2. increasing the brightness and contrast (Curves filter)
3. stabilizing the video (Stabilization filter)
4. integrating the enhanced and stabilized frames (Frame Averaging filter)

Processing a video in a proprietary format with CamStudio [How to]

One of the biggest problem during investigations on CCTV footage is the fact that most of the systems records in some proprietary format. Although usually encoding algorithms employs the standard mpeg-like algorithms, you won’t be able to view them on common media players. Some are providing a DirectShow or VideoForWindows codec, but most of them allows to view the recorded material on some custom media player. If you want to try to process the video with Amped Five (or some other editing software, I hope not), you’ll need to capture directly what’s happening on screen and encode it back with a more standard codec. Some of these players have some options to save the frames of the video in image files or to export it in the avi format, but quite often you have to save manually all the single frames one by one (what will you do for hours of video?) and the avi file exported will be very badly compressed, further damaging footage that proabably was having a bad quality.

Here it comes in help a beautiful free tool, called CamStudio. You can download the program from here. A very important thing to note is that you need also to recompress the captured video (your hard drive won’t be big enough if you nedd to record more than few minutes) with the minimum loss of quality. For this purpose the CamsStudio author makes available also a lossless codec to install on the system, that you can download here. In the last days the website wasn’t accessible because of traffic limits, but you can still access it by SourceForge.

CamStudio is quite straightforward to use, but I can help you providing the following steps.

1. Open the video you need to capture with its proprietary player. Usually these players will open to fit the image without interpolation at the beginning. Be sure not to resize the window to avoid interpolation artifacts or, if possible, verify that the pixel size set being 1:1 on the screen.
2. Disable all eventual built-in filters, such as deblocking, brightness correction, etc, in order to save the video without any processing.
3. Start CamStudio, select from menu Region and then Fixed Region, then push the button Select. Drag a rectangle around the proprietary player. You don’t need to be very precise and you can include even the window borders, since you can also crop out later the unwanted regions.
4. Select from menu Options the entry Cursor Options and then select Hide Cursor.
5. Select from menu Options the entry Video Options. Select as compressor CamStudio Lossless Codec v1.0. Then set the Quality to 100.
6. At this point you have to set the frame rate. If you set the maximum, the software will try to save as fast as possible the different frames (70 fps on my 2 years old laptop). You’ll be sure not to loose any frame, but the frames will be probably replicated many times before they effectively change (and the saved file will be bigger). You can make a couple of tests to set the optimal frame rate, or if you know it from the player or from the system producer you can set it exactly. If you have replicated frames you can remove it later with Five, but this will be explained later.
7. Start capturing pressing the big red button, start your player and, when the part of the video of interest is finished stop the recording. Then a dialog to save your file will appear. Save it wherever you want.
8. At this point, you can see the encoded video in Windows Media Player; note that this is possible because you installed on the system the CamStudio Lossless Codec. If you try to open with another program which does not rely on system codecs, such as VLC you won’t be able lo load it.
9. Now you can import the video in Amped Five. Start a Video Loader from the Filters panel. Usually you can simply load the video and process it but in some cases (I suppose in files encoded with the older version of CamStudio Lossless Codec), you will need to set the Decoder to DirectShow (for a reason similar to the fact that the video is loaded in Windows Media Palyer but not in VLC), if you played already enough with Five you should now that when you open a video with DirectShow you need to transcode to work on it, so check the Transcode option. The other settings should be fine, since de default codec is set to avi – JPEG lossless. This is another lossless codec to convert to, suggested to avoid again any loss of quality. But, I repeat, usually yhe transcoding is not needed.
10. If you want you can crop out only a desired part of the video (for example if you set a too big area in point 3) with the filter Crop in the Edit category.
11. If at the point 6 you set a too high frame rate you can avoid multiple replicas of the same frame putting a Range Selector (from Select Frames). Just set the first and the last frames in the From position and To position field, check the underlying box and select the approriate step to retrieve only one frame every n (depending on how faster has been your capturing with respect to the actual frame rate). I should tell you that we are currently implementing a filter that should automatically do this, removing identical frames in a smarter and automatic way… but this is not ready yet.
12. Process and enhance!