Display options
Cite
Lee K, Dobbins NJ, McInnes B, et al. Transferability of neural network clinical deidentification systems. J Am Med Inform Assoc. 2021;28(12):2661-2669doi: 10.1093/jamia/ocab207.
Lee, K., Dobbins, N. J., McInnes, B., Yetisgen, M., & Uzuner, �. �. (2021). Transferability of neural network clinical deidentification systems. Journal of the American Medical Informatics Association : JAMIA, 28(12), 2661-2669. https://doi.org/10.1093/jamia/ocab207
Lee, Kahyun, et al. "Transferability of neural network clinical deidentification systems." Journal of the American Medical Informatics Association : JAMIA vol. 28,12 (2021): 2661-2669. doi: https://doi.org/10.1093/jamia/ocab207
Lee K, Dobbins NJ, McInnes B, Yetisgen M, Uzuner Ö. Transferability of neural network clinical deidentification systems. J Am Med Inform Assoc. 2021 Nov 25;28(12):2661-2669. doi: 10.1093/jamia/ocab207. PMID: 34586386; PMCID: PMC8633667.
Copy Download .nbib Format: NLM
Share it on

J Am Med Inform Assoc. 2021 Nov 25;28(12):2661-2669. doi: 10.1093/jamia/ocab207.

Transferability of neural network clinical deidentification systems.

Journal of the American Medical Informatics Association : JAMIA

Kahyun Lee, Nicholas J Dobbins, Bridget McInnes, Meliha Yetisgen, Özlem Uzuner

Affiliations

  1. Department of Information Science and Technology, George Mason University, Fairfax, Virginia, USA.
  2. Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, Washington, USA.
  3. Department of Computer Science, Virginia Commonwealth University, Richmond, Virginia, USA.

PMID: 34586386 PMCID: PMC8633667 DOI: 10.1093/jamia/ocab207

Abstract

OBJECTIVE: Neural network deidentification studies have focused on individual datasets. These studies assume the availability of a sufficient amount of human-annotated data to train models that can generalize to corresponding test data. In real-world situations, however, researchers often have limited or no in-house training data. Existing systems and external data can help jump-start deidentification on in-house data; however, the most efficient way of utilizing existing systems and external data is unclear. This article investigates the transferability of a state-of-the-art neural clinical deidentification system, NeuroNER, across a variety of datasets, when it is modified architecturally for domain generalization and when it is trained strategically for domain transfer.

MATERIALS AND METHODS: We conducted a comparative study of the transferability of NeuroNER using 4 clinical note corpora with multiple note types from 2 institutions. We modified NeuroNER architecturally to integrate 2 types of domain generalization approaches. We evaluated each architecture using 3 training strategies. We measured transferability from external sources; transferability across note types; the contribution of external source data when in-domain training data are available; and transferability across institutions.

RESULTS AND CONCLUSIONS: Transferability from a single external source gave inconsistent results. Using additional external sources consistently yielded an F1-score of approximately 80%. Fine-tuning emerged as a dominant transfer strategy, with or without domain generalization. We also found that external sources were useful even in cases where in-domain training data were available. Transferability across institutions differed by note type and annotation label but resulted in improved performance.

© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: [email protected].

Keywords: deidentification; domain generalization; generalizability; transferability

References

  1. J Biomed Inform. 2015 Dec;58 Suppl:S11-S19 - PubMed
  2. J Biomed Inform. 2017 Nov;75S:S34-S42 - PubMed
  3. Appl Clin Inform. 2017 May 31;8(2):560-580 - PubMed
  4. J Am Med Inform Assoc. 2007 Sep-Oct;14(5):550-63 - PubMed
  5. J Biomed Inform. 2015 Dec;58 Suppl:S60-S66 - PubMed
  6. J Am Med Inform Assoc. 2007 Sep-Oct;14(5):564-73 - PubMed
  7. J Biomed Inform. 2017 Nov;75S:S4-S18 - PubMed
  8. Methods Inf Med. 2006;45(3):246-52 - PubMed
  9. J Am Med Inform Assoc. 2017 May 1;24(3):596-606 - PubMed
  10. Proc AMIA Annu Fall Symp. 1996;:333-7 - PubMed
  11. AMIA Annu Symp Proc. 2018 Apr 16;2017:1070-1079 - PubMed
  12. J Biomed Inform. 2017 Nov;75S:S19-S27 - PubMed
  13. J Biomed Inform. 2015 Dec;58 Suppl:S6-S10 - PubMed
  14. BMC Med Res Methodol. 2010 Aug 02;10:70 - PubMed
  15. BMC Med Inform Decis Mak. 2017 Dec 01;17(1):155 - PubMed
  16. Artif Intell Med. 2008 Jan;42(1):13-35 - PubMed

Publication Types

Grant support