The Video Formats and Conversion blog series continues with an in-depth look into the history, challenges, and solutions of proprietary data. Often originating from video surveillance systems, this volatile source of evidence must be handled correctly to avoid errors in analysis or presentation.
Welcome back to this blog series on Video Formats and Conversion. In this post, we will dive into the murky waters of the proprietary data originating from video surveillance systems and CCTV.
Within “Proprietary Data: Navigating the Maze of Video Formats”, we will look at:
- Historical context: The rise of digital video in the early 2000s led to a surge in proprietary formats as manufacturers bypassed licensing and standards to create their own ecosystems, causing long-term issues in video compatibility and decoding.
- Early digital video challenges: Early digital video systems lacked export options for digital data, often outputting to analog formats, creating compatibility challenges for forensic analysis and video quality.
- Proprietary data complexity: Proprietary data from surveillance systems is often poorly documented, making it difficult to interpret and decode video footage, particularly due to unique indexing methods and non-standard formatting.
- Indexing and video data: DVR and NVR systems often use unique methods to index video data. Acquiring video without the proper index can result in missing date and time information, further complicating forensic analysis.
- Proprietary players: Each surveillance system often requires its proprietary video player, which can vary greatly in functionality. These players are not designed for forensic analysis and may introduce errors such as hidden processing and poor playback quality.
- Lossless extraction: Proper handling of proprietary video formats requires lossless data extraction. This includes extracting the video, audio, and metadata (like timestamps) and ensuring the format remains unaltered during conversion.
- Proprietary codecs: Many proprietary formats are based on standard codecs like JPEG or H264 but have been modified, leading to decoding errors with conventional tools.
- HDD acquisition: Hard disk drives (HDDs) from CCTV systems often lack recognizable partition formats, making them appear as unallocated data. Special techniques are needed to interpret and extract the video data correctly.
- SCD format example: The article describes an SCD file, where metadata and video frames are indexed by date and time. The proprietary format introduces challenges like stream identification, affecting the correct display of video streams in non-proprietary players.
- Amped Engine: Amped Software’s tools help address these challenges by identifying, extracting, formatting, and decoding proprietary video formats, enabling forensic investigators to work with otherwise difficult-to-process video files.
To bring some context to this article, it is important to go back in time to understand how proprietary data became a little problematic. However, it is a shame that we have not got a DeLorean, fitted with a flux capacitor, to actually go back. Imagine the time, money, and effort saved for every judicial system around the world if we never had to deal with the headache of thousands of different types of digital video. Even now, when analyzing a newly discovered format here at Amped Software, it is sometimes difficult to comprehend how the format could have been conceived in the first place.
What happened then?
History
Just as everyone was breathing a sigh of relief that the Y2K bug didn’t take down the world’s computer systems, came the rise of digital video. It didn’t move as fast as Artificial Intelligence has over recent years but it was still pretty impressive considering what had to be done.
Video surveillance has always followed prosumer video and broadcast. That is where the initial development usually is, followed by the products for the home consumer. However, certain issues played out that meant that some early developers decided to go their own way and this has just continued.
The first issue was standards. Getting these ratified takes time, and well, why wait when you don’t have to? As the manufacturer created their ecosystem, they did not care that no one else could decode the video correctly.
The next was perhaps licensing. Codec developers were licensing their technology. So, rather than use that, surely it’s easier to do something similar and then call it your own?
Finally, we have marketing. Everyone wants to say they have the best encoding technology. In reality, in the early years, it was terrible. How anyone thought that a 352 x 288 highly compressed jpeg was better than an analog PAL video field or frame is still a head-scratcher!
The funny thing (although it wasn’t at the time), is that much of this very poor digital footage was then placed back onto tape as there was no method to export the digital data!
Early Generation DVR
In this image, we have an early generation Digital Video Recorder. Around the back, there are 4 analog camera inputs and an analog video output. This output was then bizarrely, usually connected to a video tape recorder. There was no native digital export option. As you can see, this system did have a removable Hard Disk Drive (HDD) caddy. However, that data could not be read by anything other than the device it was connected to!
The purpose of this DVR had nothing to do with video quality. It was simply designed so users did not have to change the tape every day.
Luckily, as we mentioned in the previous introduction to this series, things did slowly change with regard to the encoding type and ease of exporting. However, the free reign given to manufacturers in the early years still causes problems today regarding the closed formatting of the exports.
Even when a codec is deemed proprietary, it is often simply a modification to a standard. An example of this is the JPEG standard. This can be modified in several ways and the changes are often unpublished. Consequently, the identification of the changes and the methods to reconstruct the data can take some time.
Export of Native Proprietary Data
In our CCTV Acquisition series, we detailed all of the common export methods from modern video surveillance systems.
We learned that acquiring native data is not as simple as it is often imagined. With multiple options for data type, and then several transfer options for that data, it often comes down to testing. Establishing which route allows the proprietary formatting to be retained helps to maintain integrity and authenticity. It also makes the data much easier to forensically manage.
Let us look further at the SCD format we discussed in the previous post.
SCD Format
The HDD from within the DVR has no partition format. This is what often stumps computer examiners, as they are looking for known partition types and a file-based system. There are no files on this HDD example. According to Windows Disk Management, and many forensic HDD tools, it is one huge pot of “unallocated data”.
That data, however, does form something when interpreted correctly. To start with there is an index. Analysis reveals that the index relates to real dates and times. These are associated with a data offset. When we navigate to that position on the drive, we can identify video data.
This is how the search function works on the CCTV device. There is a Graphical User Interface (GUI), that communicates with the index to playback or export the required video.
Although this is a common HDD structure, the indexing methods differ widely. There are hundreds of different methods across the thousands of video surveillance systems. With sub-stream recordings and multi-codec capability, you can imagine how complicated the modern NVR HDD format is. And let us not even get into multi-drive systems and RAIDs.
With our single drive example, if an acquisition using the HDD was the only method selected, there are two common techniques.
Bypassing the Index
The first is bypassing the index. It may be possible to bypass the index and navigate directly to known video data, such as the positions marked as Cam1 in our example. However, the consequence of missing the index is that the video may have no reference to a date and time. You could have a mix of video streams for many days or weeks but have no way of knowing when they refer to. You may be lucky and have a pixel-embedded timestamp in the video but there is an added challenge we will come to shortly.
Single Date and Time Index Point
The second technique is using only a single date and time index point.
It may be possible to interpret the index and select a start and end point to extract the data. An effect often seen from taking this approach is that you then have a starting time and a duration, but no time and date information within the video stream itself. This can cause timing and analysis challenges further in the investigation.
Separation of Streams
With both techniques, the added challenge is that the video streams themselves have to be separated. The time references where the video is, but there may be lots of different videos linked to that same time.
Finally, a decision must be made on how to format the extracted data. If an application forces a format on you, is it suitable for that data?
Regardless of the challenges HDD acquisition brings, there are occasions when this is the only option. Corrupted indexes, damaged devices, and password protection will mean HDD recovery may have to be used, regardless of the loss of native formatting.
As is shown in the diagram, the native export from the device has the same data, but it is formatted differently. The format is designed for the developer’s playback software.
SCD Format
In this example, the file starts with a header. Within the header is a start and end date and time. There is usually also other metadata in file headers.
Following on, there is the video data but, importantly, each data segment has a date and time reference. The purpose of this is to aid the proprietary players searching and scrubbing capability. Without that data, searching and scrubbing may not be accurate.
You may find that, when searching and scrubbing within proprietary player applications, you always go to an I-frame. This may be a sign that the index within the loaded file only relates to the I-frames.
You may also come across scrubbing errors. This is commonly caused by the export only having a single date and time reference at the start. The player must then interpolate the timing data during playback, using the duration of the video.
Let us get back to our SCD file and look at how it is interpreted.
The Date and Time
There are several standards for date and time within digital data but, guess what? CCTV manufacturers do not have to use them and often create their own. Luckily, in our case, they have used the Unix standard.
So, in our nice example, we have a Unix timestamp at the start of every set of video frames.
But what if it wasn’t that easy?
What if there was a single date and time at the start and then a binary counter to add on the MS to the initial start point?
What if the individual components of the date and time were stored as continuous binary data?
We could carry on, but if you can think of a what-if, it likely exists as a method of linking video to real dates and times.
We now know the header, the date and time format, and the locations of the date and time information. Next comes the video.
The Video
There is a standard for controlling and managing several streams of a video within a single container format. However, this is another area where CCTV manufacturers have decided to make some changes.
In our example, the video frames are not stored “stream-based”. They are “time-based”.
If you were to carve the video data from the first frame, without identifying the stream ID, you may find something like this.
Here we have the first 18 frames from the video data. The first 4 frames are I-frames relating to the 4 individual cameras. We will learn more about frame types in the next article in this series.
As the video was encoded using a standard, (H264), it would be possible to decode it. However, the stream identifiers, in hex, are proprietary and not understood by standard decoders. They are individual to this specific format.
The consequence, as shown by the red arrows at the top, is that after the full (I) frames are decoded, the first predictive (P) frame references the wrong I-frame, resulting in a corrupted image. This error propagates with every consecutive P-frame until the next I-frame. It then starts all over again!
Within the proprietary player, that application is aware of the stream identifier. It can link the frames correctly, as is shown by the green arrows at the bottom.
Let us look at proprietary players for a moment.
Proprietary Video Players
There are hundreds of players, designed, in their most basic form, to decode the proprietary video data away from the device that recorded it. They are part of an ecosystem that will either be a single product line, a manufacturer/brand, or a group. They range from very basic to advanced.
The player is developed to decode the proprietary data. It can read the header, display important information, and allow the date and time to be searched. It can present the video streams individually as it knows the information to look for, to separate the camera views. However, these applications are not, in the main, designed for any sort of forensic work. Some have come close over the years, but they always seem to miss important functions.
Some of the most common challenges when having to deal with proprietary players:
- Installation requirements
- Different versions for different codecs
- The sheer number of different players
- Requirements for old operating systems
- Limited playback control
- Uncontrolled scaling of the viewer
- Hidden in-player processing
- Skipping/dropping frames on playback
- Cropping of video
- Poor color/luminance presentation
- Only single-image export
- Video export adding further compression
- No native export
- No data information
- Difficult to use
The most problematic to the analyst is when the player does things to the video without telling you. The hidden in-player processing is unfortunately very common and it hides the “real” image from the viewer. It is for this reason that screen-capturing quickly dropped down the list of conversion methods many years ago. Yes, it is required in extreme circumstances, but it’s a last resort.
So, what is at the top of the list for dealing with proprietary formats?
Lossless Extraction of Proprietary Data
We have learned so far that the video surveillance industry has a habit of making things interesting when it comes to the data they create and export:
- Undocumented closed formatting
- Proprietary timing data and control methods
- Proprietary stream identification
- Coding manipulation
Dealing with these challenges takes work. Thankfully, this is where the Amped Engine comes in. This powerful application is integrated into all Amped Software video products that require conversion capability. Now available to also be used within Digital Evidence Management Systems (DEMS) worldwide, the Amped Engine has unique capabilities when compared to standard-based analysis or conversion tools.
The first stage is the identification of the format.
Here we have our SCD file, along with a proprietary player and a few other files.
It is worth spending a moment to understand what these are.
Dynamic Link Library (DLL) files contain sets of instructions. The ones included here originate from the FFmpeg Multimedia Framework. If you see files named like this with players, or other video tools, they will probably be using parts of that framework for some processes
We know that our video is in the SCD file, and can see that it has a naming convention of a date and time. When dragged into Amped FIVE, the application automatically runs a few tests, using the file extension and also the data within the file header. If it cannot be automatically decoded, or it matches a known proprietary format, then the following options will appear:
We will look at all of these options in the next posts. But, before we finish, let us go through and list what we want Amped Engine to do with our proprietary SCD file.
Identify
Identification of the format is not the same as simply reading the file extension. File extensions are not controlled. Any developer can name their file type with any extension. Consequently, there could be two very different SCD file formats, requiring very different data extraction techniques.
Extract
The first thing is the unformatted video. Rather than one big chunk, where it is all mixed up, we need to separate the camera streams. This ensures that other videos, the date and time information, or other metadata do not corrupt and interfere with the correct decoding.
Next, we have the date and time information. This must not only be extracted but referenced to the frame numbers in the video.
With other formats, there could be audio data too.
Format
After the data is extracted, it must be formatted correctly, using an open standard. The video therefore may be placed into a compatible container, such as Matroska (MKV). For the date and time, our .time file format is simply open text with Frame Number > Frame Type > Date Time. We will take a closer look at this in later posts.
Decode
The final stage is the presentation of that data, such as loading it into the viewer within Amped FIVE.
Finally
We have learned that CCTV investigators and video analysts live in a world full of unknown data. That data requires careful handling and interpretation for it to be decoded correctly.
Our SCD example was a fairly basic format, but even then, if the data had not been extracted correctly, the video would have been corrupted. There would also be no date and time to control, compare, or verify the time or speed of playback. To take a closer look at what can happen, have a re-read of this post.
With modern exports, there are several other issues often encountered. All solutions come fully embedded into Amped Engines’ capabilities.
- Sub-stream extraction – the correct identification of main video streams and also the lower quality network sub-streams.
- Multi-section streams – multiple streams of video combined into a single 2×2 or 3×3 video.
- Multi-codec formats – single proprietary containers containing streams encoded with different codecs, along with different timing controls.
- Multi-file formats – multiple files that must be integrated to form a single output of video and reference time.
- Codec modifications – proprietary codecs based on a standard, where it is necessary to reconstruct the data to conform to the originating encoding type.
We have also learned that after Amped Engine completes stream extraction, the data must be correctly formatted. Therefore, in the next post, we will continue with formats and codecs. Once we have a grasp on them we can come back to the conversion of both proprietary and standard formats.
Until then, stay safe.
