Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods
What it does
Assesses the state of scientific validity and reliability of genetically-based forensic science methods used in the legal system, among other aims.
The President’s Council of Advisors on Science and Technology (PCAST) issued a report, Forensic Science in Criminal Courts, which examines the current state of forensic science used throughout the legal system. For the various forensic science methods listed below, the report synthesizes the current literature in order to determine the scientific validity of those methods. It also compares the validity of certain forensic tests to claims made by expert witnesses introducing them in a courtroom. The report concludes by offering recommendations to the Federal government for strengthening and promoting forensic science.
The report distinguishes scientific validity between:
- Foundational validity – reliability of a method to be repeatable, reproducible and accurate in a scientific setting; and
- Validity as applied – reliability of a method in practice, which includes the risk of human error.
The report examines forensic science methods including:
- DNA analysis of single-source and simple-mixture samples;
- DNA analysis of complex-mixture samples;
- Bitemark analysis;
- Latent fingerprint analysis;
- Firearms analysis;
- Footwear analysis; and
- Hair analysis.
Those forensic methods grounded in genetic analysis are the primary focus of this brief.
Regarding genetic analysis tests, the report concludes that single source and simple-mixture samples are relatively reliable and accurate, but that complex-mixture analysis is highly variable and requires further development and standardization.
Recommendations of the PCAST report pertaining to genetic forensic analysis include:
- Ongoing evaluations on the scientific validity of current and newly developed forensic technologies;
- The conversion of identification methods (particularly DNA analysis of complex-mixtures, latent-fingerprint analysis, and firearms analysis) into standardized, objective processes requiring minimal human judgment; and
- A vigorous Federal Bureau of Investigation (FBI) laboratory research program to improve forensic science.
DNA evidence is one of the primary tools used during forensic analysis. DNA is the set of molecules containing our genetic code and is 99% identical among all humans. The remaining 1% can be different enough to identify individuals and relationships. Identifying an individual can be done through a process called DNA profiling (also known as genetic fingerprinting).
To create a DNA profile, DNA is isolated from a biological sample (e.g., blood, saliva, hair, or skin) and then analyzed for distinct genetic patterns that differ among humans. These patterns are then compared between samples in order to determine the likelihood that they could have come from the same person. When comparing samples, statistical analysis is used to assess if enough similarities occur to indicate the samples likely originated from the same person, or if a familial relationship exists between the two samples.
As differentiated in the PCAST report, forensic DNA evidence falls into two categories:
1. DNA analysis of single-source and simple-mixture samples: Single-source samples are biological samples that contain material exclusively from one individual. Simple-mixture samples, on the other hand, contain biological material from two individuals, one known and one unknown. In the case of simple-mixture samples, the alleles of the known individual can be “subtracted” from the total sample in order to create a DNA profile of the unknown individual. This is a largely objective and routine process with a straightforward statistical analysis. Errors do occur, but the chances of an incorrect identification, according to the report, are close to 1 in 1 billion if the technician is well trained.
2. DNA analysis of complex-mixture samples: Complex-mixture samples are biological samples that contain material from multiple individuals in unknown proportions. A fundamental difference between analysis of complex- and simple-mixtures is that the procedure for complex-mixtures involves a higher degree of interpretation by the technician. According to the report, due to the complexity of DNA and the variations that could be present at any given genetic locus, it can be extremely challenging to separate out distinct individual DNA profiles when considering multiple unknown sources of DNA. The statistical analysis to determine the probability of an individual’s DNA being present in the sample currently involves subjective calls on which alleles to assign to which DNA sample source; changes in statistical analysis tools and subjective calls can lead to vastly different results. The accuracy of identifications and exclusions made by DNA analysis of complex-mixtures is sometimes contested in court and widely variable.
One of the major issues brought up in the report is that prosecutors are presenting DNA results claiming high validity based on the statistical analysis that is applied in any given case. The general consensus of the report is that, in most cases, complex-mixture analysis has minimal foundational validity and statistical analysis methods are highly subjective.
The report concludes that the DNA analysis of complex-mixture samples relies upon subjective review of data and lacks scientific foundation. Although promising, the use of software to resolve mixtures, such as for the 2016 trial of Oral Hillary, has suggested that automated methods are inconsistent and not yet ready for widespread use. The report also references the potential use of probabilistic genotyping algorithms to address mixture resolution.
Sara Katsanis, MS is a Duke University instructor in the Duke Initiative for Science & Society. Her research focuses on policies for DNA testing in law enforcement and human rights contexts.
Gregory A. Wray, PhD is a faculty member at Duke University in the Department of Biology. His research lies at the intersection of evolutionary genomics, molecular biology, and systems biology.
Endorsements & Opposition
- The American Academy of Forensic Sciences (AAFS) released a public statement on October 5, 2016: “The organization recognizes the need for improvement, where needed, and view the findings in the President’s Council of Advisors on Science and Technology (PCAST) report as notice of needed validation and improvement. While the Academy does not endorse every statement within the PCAST report, we appreciate the efforts of PCAST to clarify the scientific meaning of validity with respect to feature comparison analysis.” AAFS is a multi-disciplinary professional organization focusing on forensic science.
- Jim Pasco, executive director of the Fraternal Order of Police, in a quote reported by the Wall Street Journal on September 1, 2016, prior to the formal publication of the report: “What they’ve done is turn the accepted reliability of expert witnesses and their evidence on their heads.… As a result there will be people who are not going to go to jail who should be incarcerated and some who are currently incarcerated will be released. The effect will be a threat to the public safety of American citizens.”
- The FBI released a public document outlining comments and critique of the PCAST report on September 10, 2016. While the comments acknowledged the value of highly accurate forensic information, “the FBI disagrees with many of the scientific assertions and conclusions of the report.”