In this “How To” blog post, we examine the new Coding Tree Units (CTU) filter released in the latest revision of Amped FIVE. Evaluating CTUs, used in the encoding of H265/HEVC Video, is often a necessary process in Forensic Video Analysis.
Hi everyone, and welcome to this short blog post where we will look at using the new Coding Tree Units filter in Amped FIVE. Coding Tree Units (CTUs) are the building blocks that form H265/HEVC video. Interpreting this new encoder directly within Amped FIVE will help you when dealing with some of the challenges introduced by this particular encoding type.
If you are new to interpreting encoding, then it may be worth starting with Macroblocks. These form the frames used in the H264/AVC video coding standard. Having an understanding of Macroblocks will help you as we move forward.
We will also mention another filter, Block Difference, and here is a short look at that.
Finally, a post from our Formats and Conversion series dives deep into the murky world of video compression. In particular, it is important to read the Video Compression section.
High Efficiency Video Coding (HEVC), or H265, is a natural progression from H264/AVC. Designed to overcome some of the constraints in terms of resolution and compression ability, it is now being utilized in many video surveillance solutions and video recording devices such as cell phones.
However, it brings new challenges to Forensic Video Analysts and that is where we will start.
Compression Interpretation
Why does a Forensic Video Analyst need to understand how the image was formed? Surely people can just look at the picture and make their own opinion on it?
This is what separates an expert, in the eyes of a court, from a layperson.
When a video or image is to be used within an investigation, or presented to a judge or jury, it needs to have two things. Integrity and authenticity. These vital conditions must be present to correctly evaluate reliability. Can a specific object or image representation be relied upon?
With that in mind, think back to an old photograph and consider some of the issues that could be present, such as blur. Regardless of all the different issues in the image, it was all formed as one.
Digitization brought with it the battle of data size, and subsequently the requirement to compress. Without compression, the data sizes quickly become unmanageable. The key was to discard information without it becoming too noticeable. One of the methods used was to block each 8×8 set of pixels and deal with them as a single unit to code. This method of spatial compression is used in JPEG images.
Let us bypass the issue of JPEG2000, which uses wavelet compression, and look at a simple Motion JPEG Video. Each frame is a single picture made up of pixels, and those pixels are grouped in 8×8 blocks. The higher the compression, the lower the high-frequency detail. This is fairly simple to make sense of, and the entire image would commonly have the same structure.
Evaluating the reliability of the imagery is therefore straightforward.
The desire for more pixels, however, means more blocks. This means more data and larger file sizes.
This is where things get interesting. If you have read all of the previously mentioned posts, or already have an understanding of temporal compression, you will know that this can reduce data size considerably. Link that with the ability to control the spatial compression at a block level and you have a very efficient compression method. The most common compressor within the surveillance world? H264/AVC.
For a Forensic Video Analyst though, this adds a very big consideration. In our original standard old photograph, we may have had field-of-view, blur, or lighting considerations to evaluate. We must now add differences in spatial compression and then the ability to skip blocks or use blocks as references to only code the changes. You now have double the amount of issues that could affect reliability.
From Macroblocks To Coding Tree Units
With H264/AVC, the block sizes for coding were 8×8 and 16×16. These larger ones were the macroblocks. Technology doesn’t stand still though, and the desire for even more pixels pushed the next standard.
If the groups of pixels were not restricted by small block sizes, compression could be more efficient. How about having the ability to code 64×64 pixels as a single unit?
Welcome then to Coding Tree Units in H265/HEVC.
If we had certain considerations for JPEG, and these increase for H264, surely we must have more with H265? Correct, and when dealing with this as evidence, some of the new issues will have an impact.
The CCTV system owners’ desire for the largest field of view utilizing the highest pixel resolution (perhaps 4K) may result in a loss of high-frequency detail. The ability to restore and enhance will therefore be reduced as the information required was not captured to start with.
You may find larger variability in frame rate caused by the amount of data being transmitted. You may have frames dropped, or even issues with linked audio.
The most common issue, though, is the ability to easily interpret how a specific part of the image was formed, and thereby answer any questions surrounding its reliability. With increased technicality, comes more opportunity for small errors. These may be hidden when looking at the full frame as a whole. But look closer, and the accuracy of smaller objects could be questioned.
At Amped Software, we felt that a new filter was needed to observe this new coding technology so users were able to interpret the structure efficiently within their investigations.
Before looking at the new filter, let us understand what a Coding Tree Unit is.
What is a Coding Tree Unit?
The video encoder divides the image into blocks of 64×64 pixels. These are the CTUs. Decisions can then be made by the encoder on how to deal with the 4096 pixels within each unit. These decisions will depend on the parameters of the encoder, the complexity of the information received, and the changes to it compared to other reference frames.
It’s called a tree as there are sub-division levels within the CTU.
Within each CTU, there can be further Coding Units (CU). 32×32,16×16 or 8×8 pixels.
As it’s possible that objects can move inside a unit, they can also contain prediction information. Units that only have internal variation are referenced as Prediction Units (PU).
Here we can see an example of a CTU.
We can also visualize this example as a single 64×64 unit.
Within our Coding Tree Unit we have:
- Four 32×32 CU’s
- Three 16×16 CUs
- Two 8×8 CU’s. The further partitioning of these for Prediction Units is not shown in our example above.
As you can imagine, with all the coding and prediction, there is a lot of information. Currently, the filter will display the sub-division and the color of the Prediction Units.
For an in-depth look at this encoder, we recommend this practical reference book.
The Coding Tree Units Filter in Amped FIVE
Finding the Filter
The filter resides within the Verify category in the Filters window.
Remember, you can always use the Quick Search at the top of the window.
Filter Settings
The filter has two tabs: Settings and Legend.
Legend
We will look at the Legend first as this will help when we configure the Settings. It is very similar to that presented within Macroblocks, but there are fewer coding states.
Along with the colors to identify the temporal compression method, you will also see a color chart for spatial compression. This aids in visualizing the associated quantization parameter.
Settings
We now move to the Settings tab, where you can select to visualize different pieces of information.
The first checkbox overlays the CTU Subdivisions.
In this portion of a video frame, we can now see that there are four full 64×64 CUs near the top. Elsewhere, we have other sizes. You may also notice that some blocks are not subdivided equally. CTUs allow for block partitioning which enables many different variations and then higher flexibility in the compression of that specific part.
Using the settings we can now show the CU structure broken down into Prediction Units.
In this frame, the CUs with no color overlay are “skipped”. This means the encoder could copy them from the reference frame since their luminance has not changed enough to trigger any further coding. Those with a green overlay have been encoded using prediction based on a previous reference frame. Finally, those with a red overlay have been newly encoded.
Activating the next checkbox allows us to visualize the Motion Vector associated with that encoded block.
Each arrow represents the position of the origin of that Unit, with the arrowhead detailing the data’s new position.
If the video being evaluated contains B frames, therefore allowing Bi-directional prediction, then any associated Motion Vectors can also be displayed by checking the subsequent boxes.
Let us now analyze the quantization, and in particular, let’s look at the 4 full 64×64 CUs we highlighted earlier. By changing the analysis type and visualizing the values, we can see that although the top two were predicted using a previous frame, the right one has slightly higher spatial compression.
H265/HEVC has a quantization range of between 0 and 51, with 51 having the highest spatial compression.
Considerations
One of the many benefits of coding evaluation is observing how the frames are structured.
Here is a small part of a H265/HEVC P Frame.
Note the green horizontal banding, denoting predicted CUs.
In this next example, rather than using the Coding Tree Units filter, let us use Block Difference to hide the blocks that have no 8×8 change since the previous frame.
Again, notice the banding.
This is common with HEVC encoding when covering areas with little movement. The encoder’s ability to scan and reduce predicted blocks is important to observe and consider for several reasons.
At the start of this post, we mentioned a simple Motion JPEG video, where the only compression was within the 8×8 blocks, and each frame contained new information.
To restore and enhance a static object, we could use many consecutive frames and conceivably extract a much more reliable representation.
Now consider the temporal compression, the spatial compression, and the scan banding we see above. The recording system always captures the data, and then compares it to the one in preceding frames. If the difference is negligible, it simply copies pixels. But of course, if something changed in the scene, that would not be lost; the encoder would use a different encoding mode to capture such change within the limits set.
Therefore, a static object over several hundred frames may only have its data captured 3-4 times. And if that data sits inside one or two CTUs, it could be highly spatially compressed.
Consequently, you may find that the movement of an object helps with encoding technology like this. If there is enough detail captured, using filters such as Perspective Stabilization can restore moving objects better than static ones. This is because the encoder is not discarding information regarded as not required.
On the flip side, we are finding that many HEVC recordings are more susceptible to losing high-frequency or low-contrast data in moving objects when used with less-than-optimal lighting conditions.
We have a license plate 56 pixels high but all the high-frequency detail is lost. There is nothing to be restored, even with frame integration techniques.
Summary
Within Amped Support, we often encounter the belief that more pixels equate to more information. We are sure many of you will have heard, “It’s good footage, it’s from a modern 4K system”. The reality though is that there are so many competing influences on image quality. When compression is used, one slight imbalance can cause irreparable damage.
Evaluating the compression and verifying how the frame was formed is an initial stage in identifying opportunities to establish reliability. Consequently, it aids in further restoration or enhancement.
It is possible also, that this “closer look” at the pixels will assist in explaining why a task is not possible or why a question cannot be answered. This may be very relevant in video/image comparative analysis, as the compression may be the reason why a certain mark or object cannot be visualized. Look back at the license plate above. It’s not that the plate did not have letters and numbers. The data has not been captured due to the competing influences.
It is vital, however, to ensure that you are working on data that has integrity and authenticity. If the compression of a video has changed since it was first created, the ability to use compression information to assist in answering those questions is vastly reduced.
One of the biggest challenges within the forensic video world is mishandling, transcoding, and the belief that a video is just a video.
The coding technology of H265/HEVC is a great example that this cannot be further from the truth. It is volatile digital forensic data and should be treated as such.
Courts are being asked to rely on small parts of video data on a daily basis. It can only be considered reliable once we are able to verify its accuracy.
We hope you have learned a little something about H265/HEVC and take the time to use the filter on any compatible video files that you examine to aid in your learning.
The information here is the tip of the iceberg. There is a lot of great information out there on this codec and many different uses. How about the HEIC Image format that is formed from tiled HEVC video frames?
How about its ability to be used in identifying double compression/transcoding? Well, keep an eye out for updates to Amped Authenticate very soon.
Until then, take care.
