Display options
Cite
Scaiano M, Middleton G, Arbuckle L, et al. A unified framework for evaluating the risk of re-identification of text de-identification tools. J Biomed Inform. 2016;63:174-183doi: 10.1016/j.jbi.2016.07.015.
Scaiano, M., Middleton, G., Arbuckle, L., Kolhatkar, V., Peyton, L., Dowling, M., Gipson, D. S., & El Emam, K. (2016). A unified framework for evaluating the risk of re-identification of text de-identification tools. Journal of biomedical informatics, 63174-183. https://doi.org/10.1016/j.jbi.2016.07.015
Scaiano, Martin, et al. "A unified framework for evaluating the risk of re-identification of text de-identification tools." Journal of biomedical informatics vol. 63 (2016): 174-183. doi: https://doi.org/10.1016/j.jbi.2016.07.015
Scaiano M, Middleton G, Arbuckle L, Kolhatkar V, Peyton L, Dowling M, Gipson DS, El Emam K. A unified framework for evaluating the risk of re-identification of text de-identification tools. J Biomed Inform. 2016 Oct;63:174-183. doi: 10.1016/j.jbi.2016.07.015. Epub 2016 Jul 15. PMID: 27426236.
Copy Download .nbib Format: NLM
Share it on

J Biomed Inform. 2016 Oct;63:174-183. doi: 10.1016/j.jbi.2016.07.015. Epub 2016 Jul 15.

A unified framework for evaluating the risk of re-identification of text de-identification tools.

Journal of biomedical informatics

Martin Scaiano, Grant Middleton, Luk Arbuckle, Varada Kolhatkar, Liam Peyton, Moira Dowling, Debbie S Gipson, Khaled El Emam

Affiliations

  1. School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada; Privacy Analytics Inc., Ottawa, Canada.
  2. Privacy Analytics Inc., Ottawa, Canada.
  3. Children's Hospital of Eastern Ontario Research Institute, Ottawa, Canada.
  4. School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada.
  5. Michigan Institute for Data Science (MIDAS), University of Michigan Medical School, Office of Research, Ann Arbor, United States.
  6. Department of Pediatrics, University of Michigan, Ann Arbor, United States.
  7. School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada; Privacy Analytics Inc., Ottawa, Canada; Department of Pediatrics, University of Ottawa, Ottawa, Canada; Children's Hospital of Eastern Ontario Research Institute, Ottawa, Canada. Electronic address: [email protected].

PMID: 27426236 DOI: 10.1016/j.jbi.2016.07.015

Abstract

OBJECTIVES: It has become regular practice to de-identify unstructured medical text for use in research using automatic methods, the goal of which is to remove patient identifying information to minimize re-identification risk. The metrics commonly used to determine if these systems are performing well do not accurately reflect the risk of a patient being re-identified. We therefore developed a framework for measuring the risk of re-identification associated with textual data releases.

METHODS: We apply the proposed evaluation framework to a data set from the University of Michigan Medical School. Our risk assessment results are then compared with those that would be obtained using a typical contemporary micro-average evaluation of recall in order to illustrate the difference between the proposed evaluation framework and the current baseline method.

RESULTS: We demonstrate how this framework compares against common measures of the re-identification risk associated with an automated text de-identification process. For the probability of re-identification using our evaluation framework we obtained a mean value for direct identifiers of 0.0074 and a mean value for quasi-identifiers of 0.0022. The 95% confidence interval for these estimates were below the relevant thresholds. The threshold for direct identifier risk was based on previously used approaches in the literature. The threshold for quasi-identifiers was determined based on the context of the data release following commonly used de-identification criteria for structured data.

DISCUSSION: Our framework attempts to correct for poorly distributed evaluation corpora, accounts for the data release context, and avoids the often optimistic assumptions that are made using the more traditional evaluation approach. It therefore provides a more realistic estimate of the true probability of re-identification.

CONCLUSIONS: This framework should be used as a basis for computing re-identification risk in order to more realistically evaluate future text de-identification tools.

Copyright © 2016 The Authors. Published by Elsevier Inc. All rights reserved.

Keywords: Data sharing; De-identification; Evaluation framework; Medical text; Natural language processing; Re-identification risk

MeSH terms

Publication Types