More about proprietary video formats

In a recent post, Larry Compton of DME Resources makes some excellent points about video format conversion.

Jim Hoerricks adds some further insights that are also related to Amped FIVE workflow, especially concerning our proprietary video conversion feature we are constantly improving.

The bottom line of the two articles is that, even if you have third party tools to decode the proprietary video formats, they are called “proprietary” for a reason. Thus you cannot be 100% sure of the results unless you test, verify, and analyze in depth.

I wanted to add something to the discussion. The following extracts are from Larry’s post, following some comments from my experience:

“PROPRIETARY FORMATS ARE…PROPRIETARY

Seems like a no-brainer, right? But to truly understand what this means one must first understand what a file format is; it is, in gist, the structure and format of the data within the file. Standardized file formats are usually maintained by a standards body, are defined in intricate detail, and they are published so that any geek who may want to learn more about how, where, and what types of data can be stored inside that file type, may do so relatively easily. Proprietary formats are the exact opposite.”

It should be like this, but actual implementations of image and video files in different devices and software is often not 100% compliant to the standards. The inverse of this may also be correct where the standards are so broad that many of the products support only a small subset. Even for seemingly standard formats, usually if you open the same video or image file with a different software or library and then save it to a bitmap the result will be slightly different. In some cases the difference can be pretty significant. Even on standard files decoded using different versions of FFmpeg, or by installing different third party codecs on the system, you may have some very different results. Sometimes this has consequences in the forensics settings and may require some explanation.

A practical illustration of this comes from a video file I had generated with my digital camera: the video was captured with extreme low light conditions; after opening the video with DirectShow and applying exposure compensation there was a lot of detail (even if with very strong noise). When I opened it with an old version of FFmpeg and applied the same exact settings, there was very little information left, since the decoding library suppressed everything. Also for this reason we included in Amped FIVE the possibility to select your video decoding framework for every file, since you may want to check how the file is rendered by different libraries and codecs as a means to validate results

“A proprietary format is not published, and may only be truly known in intricate detail by the person or company who created it. In addition to this, many use various means of obfuscating the data. Even the geekiest of geeks may never be able to thoroughly reverse engineer the format to a point where they can say, for absolutely certainty, that they are able to extract all of the data AND metadata from that proprietary format.”

I agree, in Amped FIVE we are trying to understand the codecs used and some metadata like different camera streams and timestamps. But we cannot decode everything and we cannot be sure about the correct decoding of the data. Nevertheless we are continuously improving and making the conversion process more reliable. A year ago, we were only decoding a few proprietary formats, and today many.

“Often the posts I’ve read and the discussions I’ve heard regarding proprietary formats and FFmpeg are preceded by a brief comment to the effect of “Most proprietary video file formats contain standard multimedia streams.” In my experience “most” do not use standard streams, but certainly some do. Experienced DME technicians and analysts can tell you though, that trusting a DCCTV manufacturer to adhere to standards even when they claim to, is risky business.”

This depends on the sample you use for your statistics. In the database of cases we have, more than half use a standard codec in a non-standard stream or something similar, which can be decoded with minimal intervention on the bytestream. If you add players which provide some DirectShow / VideoForWindows DLL or full codecs, which allows decoding and conversion in standard players, the percentage is again increasing. Of course, it depends on the sample size and the popularity of DVRs in your area. However, my experience has taught me that I can’t trust many DVR producers to present the data much better than our independent conversion. It should be better and it should be easy, but it’s not always the case. Situations like these are very common:

  • The player lets you choose the video resolution without specifying what is the real one;
  • The player applies some post processing unknown to the user;
  • When saving a frame the player actually saves a snapshot on the screen;
  • When exporting a video there’s no codec option and the result is a full recoding with a relevant loss of quality;
  • The wrong timestamp is shown on the video even on its proprietary video player because of different time settings on the PC where the player is installed.

“RE-WRAPPING AND TRANSCODING

When we use FFmpeg, Libav or any other tool to re-wrap or transcode a proprietary multimedia file, these tools are completely ignoring the container itself. So what are we missing? Metadata for sure, but could we be overlooking other data or even entire streams of data that may be important to the proper playback and interpretation of the data? Yes, absolutely.”

I fully agree with this. We cannot limit ourselves to a dumb conversion. In fact, we are already trying to understand more in depth how the video data and the metadata are structured into the file.

“What about when we use one of these tools to transcode streams pulled from proprietary containers? Could we miss entire frames or new slice data within a frame? If you’ve ever used FFmpeg or Libav and you’ve seen yellow warning text or red error text in the command line window as it cooks through your file, you know the answer to that question is yes.”

This may happen and actually often happens. But, it also very frequently happens with what should be standard video files. Even those are not always compliant. Also, if you analyze the structure of simple JPEG files there are huge differences between implementation in different cameras. For reading vendor specific metadata, you frequently need the camera software for images too, but how often are we using that? And honestly, let me repeat that I don’t even trust very much the DVRs player software. Often if we use the proprietary player, we are playing on the safe side only from the formal point of view, but not on the technical one. We can just check the box and say: “I trust the player because it’s what the producer gives to me”. But, if you actually look at it from a critical technical standpoint I wouldn’t bet anything on its reliability (in general, but there are exceptions for some brands with very good players). The higher end VMS/DVR generally is better, but with the low end DVRs, you’re lucky if the player works at all sometimes.

“I can’t tell you how many times I’ve worked with proprietary viewers that display their own data incorrectly or incompletely. My recommendation is always validate your tools for the task at hand. If you’re considering using FFmpeg, Libav or another 3rd party tool to process proprietary file formats, where feasible, your results should be compared to the proprietary application that was intended to present that data.”

Again, I fully agree with Larry. We always need to validate the results from the player and compare the alternatives. One common issue there is with FFmpeg is with the often variable frame rate of DVR videos. It doesn’t work well with it and it may be set to a standard default value. Thus, you need to rely on timestamp for a proper timing.

Jim Hoerrickscommented about Larry’s post on his blog, adding some notes about Amped FIVE, and he did this better that I could have done:

“Because FIVE uses FFmpeg, I’m always worried that the basic user will accept at face value what is created through the Load filter when using the Change/Convert tools. DON’T!!! You’re an analyst. Test. Re-test. Verify. Sure, FIVE is easy and fast. But, nothing is perfect … especially when dealing with some of the most dodgy evidence known to the justice system – proprietary digital CCTV files. If the converted file has a different frame count than the original, the conversion didn’t work correctly. If your converted file’s I Frames aren’t in the same place as they were in the original file … etc. With FIVE’s support, you can bring these issues up to the developers. I may send them a few clips per week for further analysis of the conversion issues that come up.”

Nevertheless, you do not always need an in depth analysis of a video file, but we need to take a quick look for triage or investigative reasons. In those cases, time is more important than quality. In an urgent case, losing a couple of days to search for the proper player can easily damage the investigation or allow a suspect time to get away.

In most cases, an independent conversion is still much better than alternative solutions like screen capture, which is often more prone to errors and quality loss and much more time consuming. Screen capture is often a life saver (and it’s included and fully integrated in Amped FIVE), but it should be used only as a last resort, not as a first measure.

In some cases, Amped Software is in direct contact with the system producer: for example, we are a Milestone Solution Partner and thus we have access to the vendor SDK. In Amped FIVE you can natively and “officially” decode Milestone video and metadata, and we are currently in contact with other major vendors. This is great on the high end where you have that cooperation, but on the low end it isn’t possible.

After this long discussion, I think we can summarize a few very important points:

  1. Proprietary formats are proprietary. We can often convert them, but it’s an action independent from the producer, thus we cannot guarantee the results are 100% compliant.
  2. The quality and reliability of proprietary DVR players is usually very low, in general, you can’t trust them more than an independent conversion.
  3. Often there are proprietary metadata in the video stream: we are working on adding the ability to include those too.
  4. Most of the time you don’t need to go crazy about details. It’s very disappointing to lose hours or days looking for a player to decode a specific file and then discover you don’t care about its content. When it’s possible to convert it with FIVE (or other independent tools) with a single click, it will save you a lot of time and that may be important.
  5. If the video becomes a critical piece of evidence, in that case, you MUST cross-verify it with other tools and the official player, especially if there is some doubt regarding the reliability of the conversion.

Did anybody ever tell you that being a forensic video analyst was an easy job? clip_image001