Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Matters Arising
  • Published:

Transparency and reproducibility in artificial intelligence

Matters Arising to this article was published on 14 October 2020

The Original Article was published on 01 January 2020

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Data availability

No data have been generated as part of this manuscript.

References

  1. McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020).

    Article  ADS  CAS  PubMed  Google Scholar 

  2. Bluemke, D. A. et al. Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers—from the Radiology editorial board. Radiology 293, 315–316 (2019).

    Article  Google Scholar 

  3. Gundersen, O. E., Gil, Y. & Aha, D. W. On reproducible AI: towards reproducible research, open science, and digital scholarship in AI publications. AI Mag. 39, 56–68 (2018).

    Article  Google Scholar 

  4. Crane, M. Questionable answers in question answering research: reproducibility and variability of published results. Trans. Assoc. Comput. Linguist. 6, 241–252 (2018).

    Article  Google Scholar 

  5. Sculley, D. et al. in Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2503–2511 (Curran Associates, Inc., 2015).

  6. Stodden, V. et al. Enhancing reproducibility for computational methods. Science 354, 1240–1241 (2016).

    Article  ADS  CAS  PubMed  Google Scholar 

  7. Hutson, M. Artificial intelligence faces reproducibility crisis. Science 359, 725–726 (2018).

    Article  ADS  PubMed  Google Scholar 

  8. Bzdok, D. & Ioannidis, J. P. A. Exploration, inference, and prediction in neuroscience and biomedicine. Trends Neurosci. 42, 251–262 (2019).

    Article  CAS  PubMed  Google Scholar 

  9. Gundersen, O. E. & Kjensmo, S. State of the art: Reproducibility in artificial intelligence. In Thirty-second AAAI Conference on Artificial Intelligence (AAAI-18) 1644–1651 (2018).

  10. Shorten, C. & Khoshgoftaar, T. M. A survey on Image Data Augmentation for Deep Learning. J. Big Data 6, 60 (2019).

    Article  Google Scholar 

  11. Lee, R. S. et al. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4, 170177 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  12. Wallach, J. D., Boyack, K. W. & Ioannidis, J. P. A. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol. 16, e2006930 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  13. Amann, R. I. et al. Toward unrestricted use of public genomic data. Science 363, 350–352 (2019).

    Article  ADS  CAS  PubMed  Google Scholar 

  14. Carlson, B. Putting oncology patients at risk. Biotechnol. Healthc. 9, 17–21 (2012).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank S. McKinney and colleagues for their prompt and open communication regarding the materials and methods of their study. This work was supported in part by the National Cancer Institute (R01 CA237170).

Author information

Authors and Affiliations

Authors

Consortia

Contributions

B.H.-K. and G.A.A. wrote the first draft of the manuscript. B.H.-K. and H.J.W.L.A. designed and supervised the study. A.H., F.K., T.S., R.K., S.-A.S., W.T., R.D.W., C.E.M., W.J., J.D., C.F., L.W., B.W., C. McIntosh, A.G., A.K., C.S.G., T.B., M.M.H., J.T.L., K.K., W.H., A.B., J.P., R.T., T.H., J.P.A.I. and J.Q. contributed to the writing of the manuscript.

Corresponding author

Correspondence to Benjamin Haibe-Kains.

Ethics declarations

Competing interests

A.H. is a shareholder of and receives consulting fees from Altis Labs. M.M.H. received a GPU Grant from Nvidia. H.J.W.L.A. is a shareholder of and receives consulting fees from Onc.AI. B.H.K. is a scientific advisor for Altis Labs. C.M. holds an equity position in Bridge7Oncology and receives royalties from RaySearch Laboratories. A.K. is on the SAB of ImmuneAI Inc, a consultant for Biogen Inc., a scientific co-founder of RavelBio Inc. and a shareholder of Freenome Inc. G.A.A., F.K., L.W., B.W., C.S.G., J.T.L., W.H., A.B., J.P., R.T., T.H., J.P.A.I. and J.Q. declare no other competing interests related to the manuscript.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haibe-Kains, B., Adam, G.A., Hosny, A. et al. Transparency and reproducibility in artificial intelligence. Nature 586, E14–E16 (2020). https://doi.org/10.1038/s41586-020-2766-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-020-2766-y

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing: Translational Research

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Get what matters in translational research, free to your inbox weekly. Sign up for Nature Briefing: Translational Research