Repeatability and reproducibility are key indicators of the quality of experiments and analyses. Protocols, data, and research findings are shared worldwide within the scientific community, and if studies cannot be repeated or reproduced, concerns about their trustworthiness start to arise.
Image Credit: Gorodenkoff/Shutterstock.com
The term repeatability refers to the capacity to obtain the same result when multiple analyses are performed by the same operators, using the same measurement procedures and systems, under the same conditions, at different times.
The same team with the same experimental setup should be able to get the same result (within the margins of experimental error). It is a way for scientists to verify the genuineness and reliability of their own results.
Reproducibility concerns the possibility that a measurement can be obtained with stated precision by a different team in a different location, using the same measuring system.
If experiments are reproducible, a different team using the same methods can obtain the same results. This proves that the results obtained in one research lab are not artifacts of their unique setup.
Since the field of analytical chemistry involves measuring things, these measurements must be repeatable and reproducible. An experiment or analysis should be described with enough level of detail that users with sufficient skills and means can follow the procedure reported and obtain the same results.
The Importance of Being Reproducible in Analytical Chemistry
There are major implications associated with repeatability and reproducibility. From an analytical viewpoint, the food and pharma industries rely on the quality of a particular analysis to regulate a new drug candidate or to issue on the market a certain food product that has gone through quality checks where specific standards must be met. Similarly, having access to reliable data and experiments is crucial in environmental studies.
Two extensive studies on air pollution and mortality conducted in the United States in the 1990s – the six cities study and the American Cancer Society study – presented evidence that differences in air pollution concentrations between cities were strongly associated with mortality rates in those cities.
The results were cited by the US Environmental Protection Agency (EPA) in its revision of the National Ambient Air Quality Standards for fine particles. However, the studies were initially seen with skepticism, and many did not trust the methods used.
A third party, the Health Effects Institute (HEI), conducted a reanalysis and successfully reproduced (almost perfectly) the results of both studies. Even now, after almost three decades, follow-up studies lead to very similar conclusions, confirming the trustworthiness of the initial studies.
Image Credit: Gorodenkoff/Shutterstock.com
Lack of Reproducibility in Analytical Chemistry can Lead to a “Crisis”
Failure to reproduce experiments is somehow inevitable, with many different variables involved. However, a survey conducted a few years ago on 1,576 researchers revealed that more than 70% have failed to reproduce the results from other scientists’ experiments. More than 50% have even failed to repeat their own results.
The survey highlights potential issues with the reproducibility of results, specifically concerning scientific research, with consequent lack of trust. Several disciplines were part of the study, including the physical and chemical sciences.
Although the percentage of failure was noticeably high, most researchers did not consider the lack of reproducible results to be linked to fraud. Still, it appeared to arise from other reasons, such as investigator bias and issues with statistical analyses. Other factors are the lack of clear reporting of the experimental details and the lack of data repositories to examine and validate a study’s results.
What can be done to Ensure Repeatability and Reproducibility in Analytical Chemistry?
A certain degree of variation is inevitable. Nevertheless, confidence in the results is critical, and a high rigor in the methodology must be ensured. Different stakeholders, including researchers, policymakers, and funding agencies, are addressing the lack of reproducibility in many areas.
In this regard, some funding agencies and scientific publishers have instituted a series of guidelines and checklists to set out a systematic approach to identify and quantify sources of uncertainties – i.e., the error associated with replicate weighing and pipetting.
Several factors can contribute to experiments being not repeatable or reproducible. First is the inaccurate definition of what is actually being measured. In some cases, the measured sample may not represent the whole sample.
Environmental conditions or instrument resolution can affect the outcome of an analysis, and so do the standards and reference materials used. Some experiments may have approximations and assumptions incorporated into the measurement method. Last but not least, personal bias can also be a factor that affects the result of an analysis.
Visual tools can assist in the systematic identification of uncertainty. For instance, experimental protocols can be graphed as a series of steps to examine each step for sources of uncertainty.
Scientists should carefully consider their experimental design and the statistical approaches they plan to use. More importantly, they should always provide detailed descriptions of all procedures.
Sources:
- Peng, R. D. & Hicks, S. C. (2021). Reproducible Research: A Retrospective. Annu Rev Public Health, 42, 79-93.10.1146/annurev-publhealth-012420-105110
- Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533, 452-454.10.1038/533452a
- Plant, A. L. & Hanisch, R. J. (2020). Reproducibility in Science: A Metrology Perspective. Harvard Data Science Review, 2.10.1162/99608f92.eb6ddee4
- Plant, A. L., Becker, C. A., Hanisch, R. J., Boisvert, R. F., Possolo, A. M. & Elliott, J. T. (2018). How measurement science can improve confidence in research results. PLoS Biol, 16, e2004299.10.1371/journal.pbio.2004299
Further Reading