We work in the field of forensic video analysis, which is generally intended as the analysis of the images themselves and their context in a legal setting. For this reason, our customers often ask us if our products are valid for court use and if they have been validated and certified. We have written this post as an answer to the most common questions related to this topic.
You can also download this as a PDF document here.
What are the scientific foundations of Amped Software products?
All the processes implemented in our software follow the principles of scientific methodology. Any process follows these basic principles:
- Accuracy (Reliability): our tools and training program help users avoid processing errors caused by the implementation of an inappropriate tool or workflow and help mitigate the impact of human factors / bias.
- Repeatability: the same process, executed by the same user at a different time, must lead to the same result. The project format in Amped FIVE, for example, does not save any image data. Every time a project is reopened, all the processing happens again starting from the original data. In the event that a project file is lost or as a part of a validation or other test scenario, the same user can repeat the steps and settings, guided by the tool’s report, and achieve the same results.
- Reproducibility: another user with the proper competency, should be able to reproduce the same results. Amped FIVE generates a complete report detailing all the steps of the processing, the settings / parameters applied, a description of the algorithms employed in the processing and the scientific references for those algorithms (when applicable). In this way, another user, with a different tool set or by implementing the same algorithms, should be able to reproduce the same results. Given the huge number of implementation details and possible differences, it is not expected to produce a bit by bit copy of the results, but only to produce an image of similar informative content.
Additionally, we apply strict due diligence on the applicability of the algorithms for the forensic environment. Not every algorithm is, in fact, properly applicable in a forensic science setting. We cannot use algorithms which have a random component because they would not be reproducible and repeatable (when we do, we set a fixed seed for the random number generation) and we cannot use algorithms which “add” external data to the original, for example improving the quality of a face with information added from an average face. All information is derived from the actual evidence file.
We employ algorithms which have been validated by the scientific community through peer review, such as university textbooks, scientific publications, or conference papers. If for some specific task, there are not good enough algorithms available or we need to adapt existing algorithms, we describe the algorithm and attempt to publish them in scientific journals.
Are Amped Software products validated or certified officially for forensic use?
As far as we are aware, there are no international organizations which formally validate products for image and video analysis. Additionally, every case and the evidence files associated with that case is unique. Thus it is not possible to objectively specify a global validation scheme for a given tool. Many of our customers have performed internal, formalized validation for lab accreditation purposes. Some of our customers have also completed validation of our products through external vendors.
Are Amped Software products admissible by courts?
We are aware that our software has been used on countless cases worldwide. We have not been made aware of any issue with admissibility or a situation where our tools were specifically discredited.
However, it must be taken into account that, while we have created a workflow which tries to guide the user in a logical and scientific way, our software is, at the end, just a tool. As with any tool, it can be used incorrectly. For this reason, we strongly encourage our customers / users to attend our training sessions. Our curriculum is designed to help the user learn how to operate our tools properly within the workflow in order to avoid mistakes.
Who is using Amped Software products?
Our products are used in government and private labs. Our tools assist local / state / federal law enforcement labs, national intelligence agencies, military units, and private forensic technicians. Our tools are used in more than 80 countries around the world. We also have an active user community. Additionally, all of our software products are in use at the Law Enforcement / Emergency Services Video Association’s (LEVA) teaching laboratory at the University of Indianapolis.
What about the ISO/IEC 17025 accreditation?
ISO/IEC 17025 is the primary ISO standard used by testing and calibration laboratories. As such, it is being widely adopted by government and private forensic labs to guarantee a proper standard of operational competence and quality. Forensic Video Analysis is covered under the Forensic Analysis of Digital Data category. Amped Software products are designed specifically for the Forensic Science community. Additionally, they support all requirements set out by the UK Forensic Science Regulator. We actively monitor customer feedback received as part of the accreditation process with the aim of improving the usability of our products in ISO accredited labs. As such, we regularly update our software documentation (format/content) to ensure compatibility with the “Controlled Document” environment set out by the ISO standard.
What about NIST?
The National Institute of Standards and Technology (NIST) is a non-regulatory federal agency within the U.S. Department of Commerce. NIST’s mission is to promote innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life.
Amped Software products follow the US NIST SP-800 86 guidelines for examination, analysis and reporting. All the algorithms implemented are scientifically valid and the entire workflow is automatically documented.
NIST is working with the forensic science community and has established the new Organization of Scientific Area Committees (OSAC). OSAC coordinates the development of standards and guidelines for the forensic science community to improve quality and consistency of work in the forensic science community. It should be noted here, in terms of transparency, that an Amped Software employee is an active member of the OSAC Video/Imaging Technology and Analysis (VITAL) Subcommittee.
Given our strictly scientific and transparent approach it is likely that our tools will already be compliant with the new standards generated through the OSAC process and published by the ASTM (E30 Committee of Forensic Science).
How do Amped Software products fit into the US legal framework?
The following is a general guide to the legal framework concerning the handling of multimedia evidence in the United States, or under United States jurisdiction. It is not to be considered legal advice. Given case law and precedent, the framework varies from state to state.
Frye States:
- Many jurisdictions continue to adhere to Frye, including California, Colorado, the District of Columbia, Illinois, Kansas, Maryland, Michigan, Minnesota, Mississippi, Nebraska, New Jersey, New York, Pennsylvania, and Washington. These jurisdictions include almost all of the most populous states (save Florida, Texas, and Ohio) in the United States and together contain almost half of the American population.
- To meet the Frye standard, scientific evidence presented to the court must be interpreted by the court as “generally accepted” by a meaningful segment of the associated scientific community. This applies to procedures, principles or techniques that may be presented in the proceedings of a court case.
- The root science or associated scientific community of “forensic video / image analysis” is video / image analysis. Adding the term “forensic” is only used to indicate that the video / image analysis is being performed for court / legal purposes to answer questions put before the Trier of Fact.
- The algorithms that make up Amped Software’s products come from peer-reviewed and published academic / scientific sources.
- It is in this way that Amped Software’s products are not “new or novel” and thus pass the “general acceptance test” under Frye.
- For instances in which Amped Software products are called into question by the court, Amped Software is able to supply a time limited but fully functioning copy of the software product to the court for purposes of a “Frye Hearing.”
Daubert States:
- The Judge is the “gatekeeper.” The task of “gatekeeping”, or assuring that scientific expert testimony truly proceeds from “scientific knowledge”, rests on the trial judge.
- Relevance and reliability. This requires the trial judge to ensure that the expert’s testimony is “relevant to the task at hand” and that it rests “on a reliable foundation”.
- Scientific knowledge = scientific method/methodology. A conclusion will qualify as scientific knowledge if the proponent can demonstrate that it is the product of sound “scientific methodology” derived from the scientific method.
- Relevant factors. The Daubert Court defined “scientific methodology” as the process of formulating hypotheses and then conducting experiments to prove or falsify the hypothesis, and provided a nondispositive, nonexclusive, “flexible” set of “general observations” that it considered relevant for establishing the “validity” of scientific testimony.
- Empirical testing: whether the theory or technique is falsifiable, refutable, and/or testable.
- Whether it has been subjected to peer review and publication.
- The known or potential error rate.
- The existence and maintenance of standards and controls concerning its operation.
- The degree to which the theory and technique is generally accepted by a relevant scientific community.
- As noted above, Amped Software’s products employ peer reviewed and published algorithms.
- The workflows that are taught to customers incorporate a question/answer methodology, i.e. the scientific method. The tools support the methodology.
- When calculations are performed in Amped Software products, as in the case of three dimensional photogrammetry (single view metrology), the error rate is calculated and presented as part of the report. The report cites the source of the algorithm, and thus the source of the calculation of the error rate.
- General acceptance of Amped Software’s products and filters is the same for Daubert states as it is for Frye states.
- Given the thoroughness of the report that is generated by Amped Software products, reliability, repeatability, and reproducibility questions are completely satisfied.
- For instances in which Amped Software products are called into question by the court, Amped Software is able to supply a time limited but fully functioning copy of the software product to the court for purposes of a “Daubert Hearing.”
Federal Courts:
- Federal courts generally follow the Daubert standard.
- Federal Rules of Evidence – 702 – Testimony by Experts
- A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if:
a. The expert’s scientific, technical, or other specialized knowledge will help the Trier of Fact to understand the evidence or to determine a fact in issue;
b. The testimony is based on sufficient facts or data;
c. The testimony is the product of reliable principles and methods; and
d. The expert has reliably applied the principles and methods to the facts of the case.
- A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if:
- Amped Software’s training curriculum helps to satisfy the issues raised in Rule 702 by providing the user with a reliable, repeatable, reproducible workflow for a tool set based in peer-reviewed and published science.
- In general, more than 90% of an analyst’s work will be at the “technician” level and not the “expert” level. It is important to note that it is up to the Judge to declare an analyst an “expert.” This qualification lasts only for the duration of a specific case. The difference between the two is that the technician merely reports about the process undertaken and the results received, but only the expert is allowed to offer an opinion as to the meaning of the results. As an example, a technician may make an image of video brighter, allowing the Trier of Fact to see what is going on within the video. In this case, a report of the process is necessary to assure that the process is valid. An expert would offer an opinion as to what’s going on within that scene; i.e. how many, how fast, how tall, and etc.
- The reports produced by Amped Software’s products are suitable for both technician and expert level testimony. The reports formatting is available in Microsoft Word Document, Adobe PDF (PDF/A), or HTML formats (user choice).
Military Courts
- Military courts generally follow the Federal / Daubert standard.
- Military courts are governed by the Uniform Code of Military Justice and the Manual for Courts-Martial United States.
- For admissibility issues in military court, see the Daubert standard.
How do Amped Software products fit into the UK legal framework?
Amped Software products and related training follow the UK Forensic Regulators guidance on Video Analysis.
Of particular interest:
8.1.1 The method shall be validated, or any existing validation to be verified, as laid out in the Codes. The functions used in hardware and software tools where operation has an impact in obtaining results are to be validated as part of that validation of the method.
The Codes require software to be assessed for the impact on results and is documented in sufficient detail based on that assessment. In general, the validation requirement is for the overall method, rather than individual software packages and all the functions they contain.
In a courtroom environment, it is usual for Counsel to lead the introduction into a specific process and it is during this stage that any software and its use is detailed.
The first stage though is proving competency in the subject matter. This is specifically important if you are to be granted ‘Expert Witness’ status by the court.
In R. v. Cooper [1998] EWCA Crim. 2258: “An expert’s opinion is admissible to furnish the court with scientific information which is likely to be outside the experience and the knowledge of a judge or jury. If, on the other hand, on the proven facts or on the nature of the evidence, a judge or jury can form their own conclusions without help, then the opinion of an expert is unnecessary.”
In R. v. Atkins & Atkins [2009] EWCA Crim. 1876: “… leaving the jury to make up its own mind about the similarities and dissimilarities, with no assistance at all about their significance, would be to give the jury raw material with no means of evaluating it.”
The second case could also be used in determining reliability of visual data such as compression type and artefacts. For the untrained eye to base a judgement on the presence of an object, when that object is either the result of compression or distorted by compression, would be to give the jury raw material with no means of evaluating it.
The second stage is to prove competency in the software used. The easiest way to do this is training in that software package and identifying the limitations of that software package related to the task being performed.
The third stage details the process performed. This is where the software would be used. The key here is to prove scientific method, its reliability and its acceptance within the community.
The next stage we have is the validation of results. Even in cases surrounding the basic functions of cropping and resizing, the process should be validated to ensure the evidential worth of a result has not reduced or changed as a result of the process used. The ‘meaning’ of the evidence must also not be changed as a result of any process.
Finally, we have the report and presentation. The reports produced by Amped Software products support the scientific method stage of repeatability to ensure the evidence being presented is based on fact rather than chance.
In a courtroom environment it would then be up to the opposing council to discredit any of these stages.
So what is the bottom line?
At Amped Software, we strive for the highest level of scientific integrity and transparency. Rather than design our applications to a specific international standard or guideline, by grounding our tools based on peer-reviewed science, our tools are able to be employed across a wide variety of disciplines where video / images will be used as evidence. In this way, we are able to assist not only forensic video / image analysts, but also latent print examiners, questioned document examiners, shooting incident analysts, firearms and tool mark examiners, and many more.
If you have questions about our products or how they will fit within your laboratory or workplace, please feel free to contact us.
