Dear friends, welcome to a new video pitfall post! Among the many visual clues, the shape, size, and aspect ratio of objects are quite important ones. However, we’ll see that in surveillance videos, they can quite often be misleading. Hold tight and keep reading!
Issue: Can you trust the proportions of objects in a video?
You get an urgent call. “They’ve just kidnapped a girl and fled in a car. We have a video from the pub next door!”. While you rush to the place, you’re already drafting the plan. You’ll look at the video and immediately start a search for the car based on the license plate… or at least the model!
Once you reach the place, they pass you a USB drive with a video file in it. You plug it in your laptop and play the video. Here’s the car.
Not only is the license plate maybe 1 pixel high, but the proportions of the car are also very strange.
With a picture like this, you don’t even know whether you’re looking for a limousine, an SUV, or a utility car.
Explanation: There’s a lot from the scene to your screen!
The image creation pipeline of a typical surveillance system involves a camera, which often mounts a fisheye or grand-angular lens (so as to capture the widest possible scene), connected to a DVR that records the video. Even today, many surveillance systems use analog cameras, and the video gets digitized inside the DVR. Now, in this process, several things can go wrong and change the shape and size of objects.
Lens distortion:
The camera lens will often cause optical distortion, which is more pronounced for fisheye and grand-angular lenses. We’ll dedicate the next blog post of this series to this specific defect, let’s hide it for now.
Field-based images (half-lines are missing):
Some cameras send an interleaved video to the DVR. It means that, instead of sending full frames, only the odd lines of a frame are sent at the time “t” and the even lines at time “t+1” (these “half-frames” are called “fields”). A clever DVR should fill in the missing rows (e.g. by interpolation) and store the reconstructed full frames. However, many DVR producers choose to simply store the fields as they arrive, so they save both processing time and storage space. The problem is: when you play the video, each frame has half the lines it should have! We informally call this phenomenon “stretched video”, and you have a valid example in the picture of the car at the top of this post. The sole good thing about this defect is that it’s quite easy to spot since proportions get very odd.
Display aspect ratio (informally called “non-square pixels”):
In the analog-to-digital conversion process, there is often a clash between the rectangular aspect ratio of the video signal and the requirement to store the sampled signal as squares (the pixels) used by our computer screens. This can cause the video to be displayed with a wrong aspect ratio. For example, it is common to find PAL (UK, Europe, Australia etc) videos with a pixel matrix of 720×576 that are expected to be displayed at a 4:3 Display Aspect Ratio. As a consequence, we must increase the width slightly to account for the analog sampling process. For NTSC Analog Videos, we may need to decrease the width. Compared to the previous one, this issue is somewhat subtler: it causes less deformation, but for this reason, it’s also harder to spot at a first glance.
Solution: Check and fix when you can
Probably, you’re now tempted to not trust any video whatsoever. But wait, we’re here to help! Let’s work on our kidnapping case, step by step.
Step A: Correct field based images
Okay, it’s quite evident that we’re probably in a “stretched video” case. The first thing to try is thus a line doubling filter, which will simply double the height of each frame by interpolating or replicating rows. Here’s how it looks in our Amped Replay.
Step B: Correct display aspect ratio
Already better, isn’t it? Now, let’s take a look at the new resolution of the video (in the screenshot above, I’ve activated the Resize filter just to show the frame size). It reads 704×480, which is rather strange (704/480 = 1.46: what an odd aspect ratio). We can thus try to force the aspect ratio to match one of the two common ones, 4:3 or 16:9. Choosing the right one depends on visual inspection and knowledge about the CCTV system which produced the video. In our case, setting to 4:3 seems to get the work done nicely and brings the resolution to 640×480, which is the well known “480p” resolution.
So our car has improved quite a bit already!
At least, we’ll rather focus on stopping SUVs instead of limousines.
As shown in this example, the issues we’ve presented get mixed up quite often, so you may need to be careful and fix several of them in a single video.
