Cowboys versus Bureaucrats: Attitude and Tools

There were a couple of interesting discussions this week which prompted me to write this blog post. One is related to the scientific methods used during the analysis of images and videos, the other relates to the tools used.

There was a pretty interesting and detailed conversation that happened on an industry specific mailing list where a few experts debated about the scientific and forensic acceptability of different methodologies. This discussion began with the reliability of speed determination from CCTV video but then evolved into a more general discussion.

There are two extreme approaches to how forensic video analysts work: let’s call one group the cowboys and the other the bureaucrats. I’ve seen both kinds of “experts” in my career, and – luckily – many different variations across this broad spectrum.

What is a cowboy? A cowboy is an analyst driven only by the immediate result, with no concern at all for the proper forensic procedure, the reliability of his methods and proper error estimation. Typical things the cowboy does:

  • To convert a proprietary video, he just does a screen capture maximizing the player on the screen, without being concerned about missing or duplicated frames.
  • Instead of analyzing the video and identify the issues to correct, he just adds filters randomly and tweaks the parameters by eye without any scientific methodology behind it.
  • He uses whatever tool may be needed for the job, recompressing images and videos multiple times, using a mix of open source, free tools, commercial tools, plugins, more or less legitimate stuff, maybe some Matlab or Python script if he has the technical knowledge.
  • He will use whatever result “looks good” without questioning its validity or reliability.
  • If asked to document and repeat his work in detail he’ll be in deep trouble.
  • If asked the reason and validity of choosing a specific algorithm or procedure, he will say “I’ve always done it like this, and nobody ever complained”.
  • When asked to improve a license plate he will spell out the digits even if they are barely recognizable on a single P frame and probably are just the result of compression artifacts amplified by postprocessing.
  • When asked to identify a person, he will be able to do so with absolute certainty even when comparing a low-quality CCTV snapshot with a mugshot sent by fax.
  • When sending around results to colleagues he just pastes processed snapshots into Word documents.
  • When asked to authenticate an image, he just checks if the Camera Make and Model is present in the metadata.

A bureaucrat, on the other hand, is an analyst who is driven to the extreme, by proper procedural constraints, even at the risk of not getting any result at all. What the bureaucrat does:

  • He is limited at looking at the CCTV footage, applying very little or no processing at all. It must be said that sometimes the analyst is not a bureaucrat by choice, but because of specific policies set by his superiors. They are afraid of doing mistakes so they do nothing.
  • Sometimes he is allowed to do very limited and well-documented processing. Maybe, if he’s very lucky, he will get permission to apply interpolation. His chiefs are so worried about possible questioning in the courtroom, so to avoid any risk, they do nothing. Maybe even throw away evidence.

There is a slight variation of the bureaucrat. An “analyst” that does nothing, not because he’s not allowed to, but simply because he doesn’t know any better, doesn’t have the tools, the skills or the will to work. What does he do? Simply looks at the video, if he’s able to get it into a viewable format. If there is something easy to see, will get it, otherwise, he will throw it away. Maybe we should call him the lazy cowboy rather than a bureaucrat.

It must also be added that the cowboy is not inherently an uncaring person. Most of the time he’s just the person chosen for the role and has no technical preparation or training. After all, do we really need an expert for analyzing videos and photos? Everybody can do it! Well, not really. The problem is that you don’t know what you don’t know.

Luckily the vast majority of our users fit in the middle of this broad spectrum. Let’s call him the pragmatic analyst. How does the pragmatic analyst work?

  • He does a lot of research and testing on methodologies before using them.
  • He knows what tool is the right one for the job, the risks, and implications of using it versus another.
  • He uses methodologies that are forensically and scientifically sound.
  • He’s able to reproduce his work.
  • He’s able to document and let someone else repeat his work.
  • He’s able to estimate the reliability of his conclusion.
  • He does not usually give an opinion in absolute terms, but exposes the facts and the numbers (when possible).

In conclusion, a pragmatic analyst’s philosophy is in line with our vision of “justice through science”. He shows facts, not opinions.

There are many nuances in the broad pragmatic range. Our users who work in scientific and forensic departments must be very careful about everything and they keep Amped FIVE’s automatically generated report at heart.

On the other hand, people working in the field of operations would be more concerned about quick results than courtroom needs. They don’t usually need the report, but if they do, it’s there. But at Amped, we cover both needs at the same time by providing our users with a scientific workflow that is easy and quick and with the automatic report generation. So, if your operation activities need to become evidence, you are already covered. No need to worry.

Also for the same kind of job, different people will have a different sensibility towards the reliability of some methods versus some others. If the science is published, that’s easy, but sometimes there are requests so specific and difficult to find something relevant in literature. At that point, you can count only on yourself and on the scientific method:

  1. Define a question
  2. Gather information and resources (observe)
  3. Form an explanatory hypothesis
  4. Test the hypothesis by performing an experiment and collecting data in a reproducible manner
  5. Analyze the data
  6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis
  7. Publish results
  8. Retest (frequently done by other scientists)

At some point, you need to introduce some statistical elements. A sample size of one may often not be enough, but sometimes is what you have, so you need to recreate more data to test in known conditions. After all, if you just check the numbers 3, 5 and 7 you could infer that all the odd numbers are prime…

Let’s talk now about the tools.

When I started developing Amped FIVE, 10 years ago (!!!), there were already forensic video tools on the market, but the vast majority of analysts actually worked in this way:

  • Used dozens of different tools, some for images and some for videos, with specific plugins or scripts for performing various operations.
  • Spent hours testing different combinations of tools and processing, importing and exporting from one tool to another. For example, grabbing a frame to a video, and then deblur it into a specific deblurring plugin.
  • What happened is that they spent too much time, often for little results.
  • If some result was obtained, documenting the processing and repeating it was a nightmare.
  • Normally they didn’t know how the tools were supposed to work internally and the methodologies used.

You can see that there is a lot in common with the cowboy methodology. We must say that a pragmatic analyst can also work properly in this situation, but it will be much harder.

I started developing Amped FIVE with the aim to have a self-contained solution which could let you work quickly, effectively and scientifically from the beginning to the end of a case. Is there ANY other tool (open source, free or paid) which allows you to work with an image from start to finish, doing all these things?

  • Work with images, videos, sequence of images, live video, VMS systems and DVR formats.
  • Select the frames to work on (motion detection, various frames selectors, demultiplexing…).
  • Enhancing and restoring (adjustments, deblurring, superresolution, integration, interpolation, denoising, sharpening, component separation, frequency removal…).
  • Analyzing the image and take measurements.
  • Redaction (pixellating faces, applying spotlight effects, putting annotations…).
  • Automatic report generation.

Of course, there are many other tools, even free, that can do a little of what we do. We do not claim to be the only ones to be able to deblur an image, but how many allow you to do all these things together, in a single package with more than 100 filters, takes one minute to install, runs on every Windows PC from Windows XP to Windows 10, has been extensively used and scrutinized in courts all over the world, and saves you hours of work every day?

The same goes for Amped Authenticate. Also, in this case, there are dozens of little small tools which allow you to check metadata, JPEG quantization tables and even basic tampering detection functions like ELA. Some are even free to use on the Internet. But do you really want to upload your evidence on a random website? Is there any other tool which integrates all kind of metadata and format analysis, performs camera identification and tampering detection, and optionally integrates with web services to search for similar images or photos coming from the same kind of device?

So, the question is, can you do the job with free tools? Sure, but only part of it. It will take much more time, it requires more skills on your side and the result will be more difficult to reproduce, document and justify. You, yourself, will need to prove what Amped and your fellow analysts have already proven for you in the last 10 years in courts worldwide. And probably you will not get results as good, even if you tried hard.

Granted, our products, are not free – nor your valuable time as an analyst is free to the government. Our software cost can be easily recovered in a matter of months, if not weeks, and increases security for your community. When the safety, the security and the freedom of your people is at stake, are you the one willing to take the risk of not choosing the optimal tool for the job?

We are here to help those scared of doing anything and give them the tools they need within a protected environment. We also will not support analysis techniques that could be called in question or jeopardize their case. When you can trust the software (and the user), you can trust the result.

I want to add a final personal note. For the many of you who know me, either in person or through contact via email or other channels in the last years, you may reckon that I am normally quite calm. This post may feel a bit heated and over my usual style, but it really touches some points which are very important to me. This is the stuff I have been working on every single day for the last 10 years, and if I write something, it is not just related to our sales and marketing strategy. I write it because I honestly feel it is a very important argument. Unfortunately, there is a lot of misinformation in our field, self-proclaimed experts and sub-optimal practices and tools (free and commercial) which promise much more than they are actually capable of.

It is in the best interest for us all to have an honest conversation about what is good and what is not good, people who do not work properly can create a lot of troubles for our whole community and practices, creating false myths and expectations. And, even worse, to let the guilty free or to accuse the innocent.