Color Fidelity: It’s Hard to Objectively Evaluate Colors on Video

Welcome to this week’s Video Evidence Pitfalls post, dear friends! Today we’re dealing with the reliability of colors in CCTV footage (and with any kind of digital images in general). Colors are an essential part of human perception, but can often be misleading in surveillance footage. Keep reading to find out why!

Issue: Colors Are Often Unreliable in CCTV Footage

As part of an ATM fraud robbery investigation, you are reviewing footage from two cameras in the same area. This is the image of the suspect as he walks away from the ATM.

He’s suspected of having visited another ATM of a bank nearby. So you review the footage of that camera as well, but… you don’t find anyone with a blue jacket and beanie hat. After syncing the clocks and checking frame by frame, you realize that it could be him:

But you also realize you will have a hard time convincing anyone about it. Why are the colors completely different?

Explanation: Colors Depend Heavily on Camera, Settings, and Compression

There are several reasons that can make colors in surveillance video unreliable. To keep things simple, we can isolate four stages in the image formation chain: scene, camera, compression, and playback.

Let’s start with the scene, which means, lighting conditions. Even a virtually perfect camera would capture colors as they “happen to be” in the scene. If the scene is illuminated by colored light, this will obviously affect the color of objects. The most obvious example is sodium street lighting, which turns everything orange.

Not only – the color of an object as “seen” by the camera may be affected by reflections from other objects in the scene. On top of that, sunlight changes during the day and is obviously impacted by weather conditions. All of this makes it extremely hard to be able to recreate the same lighting conditions, even if you can go back to the scene!

Once entered into the camera, the light is filtered to retain only the visible spectrum. If infra-red light reaches the sensor, it would make colors completely misleading, as we’ve explained in a previous post in this series. For example, trust me when I say that the guy below wasn’t proudly wearing a total-red outfit…

After reaching the sensor, the image is formed and processed. The camera may automatically adjust the brightness, the contrast, and it may even apply white balancing. White balancing algorithms aim at “adjusting” colors so to render them more accurately. While these kinds of algorithms perform well when a reference can be provided (e.g., you indicate to the camera something that you know to be white in the scene), they can make things worse when running in “fully unattended” mode, as they do in surveillance cameras.

Then, compression comes. Since the human visual system is more sensitive to shapes than to colors, most image and video compression algorithms heavily penalize colors. Most surveillance footage is available in the YUV 4:2:0 color space, which means that 3/4 of the color information is dropped through the so-called “chroma subsampling”. Then, the remaining color information is also compressed more aggressively than the luminance. It’s not surprising, then, that color artifacts are ubiquitous in surveillance videos, as shown in the gallery below.

Last but not least, we have the playback stage. Proprietary players will often apply some filtering in an attempt to make their footage more “pleasing to the eye”. Deblocking and adding contrast to counteract the compression is a common one – but some also change the saturation of colors. So keep in mind that if you use a propriety player to extract an image, the colors can be different to what was actually recorded. And how can we not mention monitors: a poor monitor can make colors appear washed out, and even the placing of a monitor can affect what you are seeing.

Solution: Fix What You Can, Don’t Overweight the Rest

As usual with video forensics, when some information is gone, there’s nothing you can do to bring it back. Therefore, if colors are important for your case you should, first of all, assess the situation: are you dealing with an infrared video? Then colors cannot be assessed at all. Are you aware of colored lighting in the scene? Then assessing the real color of objects will be challenging, unless you find a reliable reference. If you have a reference, you can apply image processing filters such as Amped FIVE‘s Automatic Color Equalization, White Balance, or Temperature Tint, and use the reference to cross-check the obtained result.

As a general rule, we recommend to be very cautious in determining the “true” color of objects from CCTV videos, especially if the footage comes from different cameras, let alone different surveillance systems. It could be possible to compare the color of two objects filmed by the same camera, though, provided lighting conditions did not change and objects pass in the same position (e.g., comparing two persons walking on the same sidewalk a few seconds apart).

And remember: keeping in mind all the potential issues is the best way to protect yourself from big mistakes, and some innocent person from getting into trouble!

That’s all for today! We hope you’ve found this issue of the Video Evidence Pitfalls series interesting and useful! Stay tuned and don’t miss the next ones. You can also follow us on LinkedInYouTubeTwitter, and Facebook: we’ll post a link to every new tip so you won’t miss any!