I find myself analyzing new surveillance videos almost every day. In most of the cases I can either solve the problem (of a license plate for instance) very quickly or understand (even more quickly) that there is no information to recover in the video. In special cases, where something very specific and strange happened; or the problem is very complex it can take a lot of time.
As always, Pareto principle: you solve 80% of the cases in 20% of the time, and, well, 20% of the cases takes 80% of the time.
In my own work probably the right numbers are 95% to 5%, but the idea holds.
With my experience on several thousands cases, I can estimate whether an image or video contains some information and it’s worth processing, or not, very quickly. I usually do a couple of tests, just to be sure of my first glance. Sometimes I get surprised by the power of the algorithms. Anyway, usually you can tell quite fast if you will get something from a video.
The Big Picture
Let’s work on the most common case: enhancing a license plate. Of course, I won’t show you images from real cases so I’ll generate some samples for you.
Usually the main problems why we can’t see a license plate (or anything else useful) are:
- low resolution
- lossy compression
- optical blur
- motion blur
- low brightness / contrast
These, in order, are the most common cases. In most situations you have all of them combined together. One single of these problems, if strong enough, may cause your license plate to be unreadable.
Let’s see in detail all this issues and what we can do to correct them.
These are the problems related more strictly to the image quality. Please note that there are also other problems that maybe be corrected more easily. For example, a bad perspective or the interlacing are immediate to solve, if the overall quality of the image is not too bad.
The idea is pretty simple: the resolution determines the number of points you have in an image. If you don’t have enough points for drawing a digit, you won’t be able to see it.
How to recognize it: zoom in the license plate and count, for example, how tall is it in pixels. If the license plate is, for example 3 pixels tall, just think how can you draw a number with just three points (maybe two because it will also have the plate border).
How to solve it: if the problem is very relevant there is not very much to do. In intermediate situation a simple digital zoom (the filter Resize in Amped FIVE) can be helpful. Although it does not actually add information to the image, it will make characters and details better recognizable to the human eye. If you have multiple frames applying techniques such as frame integration (Frame Averaging) or super resolution (currently in development) can be effective in some situations.
Lossy compression is the main drawback of digital surveillance systems if compared to the old good VCR. Very often it happens that the resolution is actually pretty good but the compression makes the digits unreadable. Videos are compressed in order to save bandwidth in transmission and space in the storage systems.
How to recognize it: The typical example is when the picture looks like you can see the license plate if you zoom, then you zoom and… you can’t still read the license plate.
The problem appears in two ways:
- you have enough pixels, but many of them appear to have the same value (tile effect), so in practice you have a much lower actual usable resolution that the native resolution of the image file format
- you have artifacts, that means details that do not belong to the image but are introduced by the compression (typical blocking artifacts of JPEG files)
How to solve it: With a deblocking filter (Deblocking in FIVE) you can noticeably improve the quality of the image from a visual point of view. It is important to take care not to apply too strong a filter that will also cut out useful detail. Also here, on more frames/ frame integration / averaging can be very useful.
The optical blur happens when an image is not perfectly in focus. Usually CCTV optics are configured to have a very big depth of field, everything is more or less in focus. But it’s difficult to have something perfectly in focus.
The basic idea is that every pixel of an ideal image in focus is being transformed into a circle with a radius depending on the strength of the blur.
How to recognize it: The border and the digits of the license plate are (guess what) blurred. Basically if you just see a white area it is often blur together with a lack of contrast.
How to solve it: If you have enough resolution, optical deblurring filters can be very effective to recover even license plates that are not visible at all. Amped FIVE provides the Optical Deblurring and Blind Deconvolution filters. The first works very well on big blurs (even some tenths of pixels). While the second works more effectively on little blurs (less than 10 pixels usually).
A big problem is that the deblurring won’t work very well in images that present a strong compression, since it introduces some non-linearities that the algorithm is not able to manage.
The motion blur is the effect that appears on a subject that moves to fast in relation to the shutter speed. Even if the effect looks much different than optical blur, the nature of the problem is very similar. Every point becomes a line (instead of a circle like to optical blur).
How to recognize it: Usually it’s pretty noticeable since you can distinguish a kind of trace that follows the points in the images.
How to solve it: If the kind of motion is linear and uniform, usually we can take some reference point that we know should be a point (like the angle of a license plate or a shimmer). But that in the image clearly appears like a segment. We can estimate the features of the blur through the length and the angle of this segment. In FIVE you can use the Motion Deblurring filter.
If the resolution is not too bad and the compression is not exaggerated we may be able to correct even very strong blurs.
Low Brightness / Contrast
Different light conditions or characteristics of the cameras may lead to images that are either too dark (for example in night shots), too bright (when we have a light that points in our direction) or simply lack contrast (pixel values are too similar).
How to recognize it: You can look at the image histogram (in FIVE Tools / Frame Info panel). If the histogram is very shifted to the left, the image is too dark (most of the pixels will have values close to zero). However, if the histogram is shifted to the right the image is too bright (values close to 255). If the histogram is very thin the problem is in the contrast (too many pixels have the same value).
How to solve it: There are many tools, the use of which can vary from case to case. Two good starting points are that you can automatically equalize the histogram (Histogram Equalization in FIVE) or manually adjust the brightness (filter Curves or Contrast /Brightness). In order to improve the appearance of the details you can use on of the various available sharpening filters (Sharpen group).
An area of the image is saturated when all the pixels have values of zero or 255. If the image is saturated (or, in general, all the pixels have the same value), there is nothing to do, since all useful information is gone.
The noise is a random variation of the pixels in an image. It appears like a grain that is usually very evident in pictures taken with low light. The noise becomes very visible usually when we brighten a dark video.
How to recognize it: There are random spots on the image that modify the correct value of the pixel. Many times it’s difficult to precisely separate noise and compression artifacts.
How to solve it: If we have only a single image we can improve the overall quality with one of the many smoothing filters (in FIVE in the group Denoise). However, on a single frame this is a risky approach because many times the filters can take away the details together with the noise. If we have multiple frames of the same license plate in the same position, a technique that gives amazing results is frame integration (Frame Averaging filter). If the license plate is not in the same position it must be aligned with a stabilization filter (Stabilization or Perspective Registration).
The Bottom Line
Real cases are always more complicated than these examples, since most (if not all) of these problems will be present at the same time. It’s very important to keep these basic problems in mind in order to always start with the proper filters and algorithms to correct them. It is an even more important concept to understand to determine when an image does not contain the minimum amount of information for feasible analysis. You can see some more real-world examples on Amped FIVE samples page.