Skip to content

Commit 53fa513

Browse files
chufangaoclaude
chufangao
and
claude
authored
Add synthetic-EHR generative evaluation metrics (#1148)
* Add synthetic-EHR generative evaluation metrics Adds pyhealth/metrics/generative/, a subpackage for evaluating synthetic EHR data along privacy, utility, and statistical-fidelity axes: - privacy.py: NNAAR, membership inference attack, discriminator privacy - utility.py: machine learning efficacy (TRTR vs TSTR), code-prevalence similarity (R2, Pearson, RMSE) - utils.py: shared data prep, an LSTM classifier, and a random-forest baseline - evaluate_synthetic_ehr(): convenience orchestrator for the full suite These functions are ported from a standalone evaluation script. The MIMIC-specific data-loading/CLI glue is dropped; the metrics work on any flat EHR dataframe. Public functions are re-exported from pyhealth.metrics. Adds unit tests in tests/core/test_generative_metrics.py and Sphinx docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add synthetic-EHR generative evaluation metrics Adds pyhealth/metrics/generative/, a subpackage for evaluating synthetic EHR data along privacy, utility, and statistical-fidelity axes: - privacy.py: NNAAR, membership inference attack, discriminator privacy - utility.py: machine learning efficacy (TRTR vs TSTR), code-prevalence similarity (R2, Pearson, RMSE) - utils.py: shared data prep, an LSTM classifier, and a random-forest baseline - evaluate_synthetic_ehr(): convenience orchestrator for the full suite These functions are ported from a standalone evaluation script. The MIMIC-specific data-loading/CLI glue is dropped; the metrics work on any flat EHR dataframe. Public functions are re-exported from pyhealth.metrics. Adds unit tests in tests/core/test_generative_metrics.py and Sphinx docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * add baselines * removed halo save file and updated promptehr to be more paper accurate * update docs * update docs * Update pyhealth.models.HALO.rst --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent acaf7b7 commit 53fa513

31 files changed

Lines changed: 6310 additions & 1 deletion

‎docs/api/metrics.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ For applicable tasks, we provide the relevant metrics for model calibration, as
77
Among these we also provide metrics related to uncertainty quantification, for model calibration, as well as metrics that measure the quality of prediction sets
88
We also provide other metrics specically for healthcare
99
tasks, such as drug drug interaction (DDI) rate.
10+
For synthetic (generative) EHR data, we provide privacy, utility, and statistical
11+
fidelity metrics.
1012

1113

1214
.. toctree::
@@ -19,3 +21,4 @@ tasks, such as drug drug interaction (DDI) rate.
1921
metrics/pyhealth.metrics.prediction_set
2022
metrics/pyhealth.metrics.fairness
2123
metrics/pyhealth.metrics.interpretability
24+
metrics/pyhealth.metrics.generative

‎docs/api/metrics/pyhealth.metrics.generative.rst‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
pyhealth.metrics.generative
2+
===================================
3+
4+
Evaluation metrics for synthetic (generative) EHR data, covering privacy,
5+
utility, and statistical fidelity.
6+
7+
.. currentmodule:: pyhealth.metrics.generative
8+
9+
.. autofunction:: evaluate_synthetic_ehr
10+
11+
Privacy metrics
12+
-------------------------------------
13+
14+
.. autofunction:: calc_nnaar
15+
16+
.. autofunction:: calc_membership_inference
17+
18+
.. autofunction:: compute_discriminator_privacy
19+
20+
Utility and fidelity metrics
21+
-------------------------------------
22+
23+
.. autofunction:: compute_mle
24+
25+
.. autofunction:: compute_prevalence_metrics

‎docs/api/models.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,11 @@ API Reference
200200
models/pyhealth.models.TFMTokenizer
201201
models/pyhealth.models.GAN
202202
models/pyhealth.models.VAE
203+
models/pyhealth.models.HALO
204+
models/pyhealth.models.GPT2
205+
models/pyhealth.models.PromptEHR
206+
models/pyhealth.models.MedGAN
207+
models/pyhealth.models.CorGAN
203208
models/pyhealth.models.SDOH
204209
models/pyhealth.models.VisionEmbeddingModel
205210
models/pyhealth.models.TextEmbedding

‎docs/api/models/pyhealth.models.CorGAN.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
pyhealth.models.CorGAN
2+
===================================
3+
4+
CorGAN: a Correlation-capturing Convolutional GAN for synthetic EHR generation.
5+
A 1D-CNN (or linear) autoencoder captures local code correlations, and a WGAN
6+
generator/critic are trained in the autoencoder's latent space. Ported from the
7+
reference implementation
8+
(`cor-gan <https://github.com/astorfi/cor-gan>`_) and wrapped as a PyHealth
9+
:class:`~pyhealth.models.BaseModel`.
10+
11+
Reference:
12+
Torfi, A., & Fox, E. A. (2020).
13+
*CorGAN: Correlation-Capturing Convolutional Generative Adversarial
14+
Networks for Generating Synthetic Healthcare Records.*
15+
In Proceedings of the 33rd International FLAIRS Conference.
16+
https://arxiv.org/abs/2001.09346
17+
18+
.. autoclass:: pyhealth.models.CorGAN
19+
:members:
20+
:undoc-members:
21+
:show-inheritance:

‎docs/api/models/pyhealth.models.GPT2.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
pyhealth.models.GPT2
2+
===================================
3+
4+
A decoder-only GPT-2 baseline for unconditional synthetic EHR generation,
5+
wrapped as a PyHealth :class:`~pyhealth.models.BaseModel`. Patient visit-code
6+
sequences are serialized into causal-LM token streams and modeled
7+
autoregressively.
8+
9+
Reference:
10+
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019).
11+
*Language Models are Unsupervised Multitask Learners.* OpenAI.
12+
https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
13+
14+
.. autoclass:: pyhealth.models.GPT2
15+
:members:
16+
:undoc-members:
17+
:show-inheritance:

‎docs/api/models/pyhealth.models.HALO.rst‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
pyhealth.models.HALO
2+
===================================
3+
4+
HALO (Hierarchical Autoregressive Language model) for synthetic EHR generation.
5+
A faithful port of the reference implementation
6+
(`HALO_Inpatient <https://github.com/btheodorou99/HALO_Inpatient>`_),
7+
wrapped as a PyHealth :class:`~pyhealth.models.BaseModel`.
8+
9+
Reference:
10+
Theodorou, B., Xiao, C., & Sun, J. (2023).
11+
*Synthesize high-dimensional longitudinal electronic health records via
12+
hierarchical autoregressive language model.*
13+
Nature Communications, 14, 5305.
14+
https://www.nature.com/articles/s41467-023-41093-0
15+
16+
.. autoclass:: pyhealth.models.HALO
17+
:members:
18+
:undoc-members:
19+
:show-inheritance:

‎docs/api/models/pyhealth.models.MedGAN.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
pyhealth.models.MedGAN
2+
===================================
3+
4+
MedGAN: a bag-of-codes Generative Adversarial Network for synthetic EHR
5+
generation. An autoencoder is pre-trained on multi-hot patient records, then a
6+
GAN with residual generator and minibatch-averaging discriminator is trained in
7+
the autoencoder's latent space. Ported from the reference implementations
8+
(`medgan <https://github.com/mp2893/medgan>`_ and its PyTorch reimplementation)
9+
and wrapped as a PyHealth :class:`~pyhealth.models.BaseModel`.
10+
11+
Reference:
12+
Choi, E., Biswal, S., Malin, B., Duke, J., Stewart, W. F., & Sun, J. (2017).
13+
*Generating Multi-label Discrete Patient Records using Generative
14+
Adversarial Networks.*
15+
In Proceedings of Machine Learning for Healthcare (MLHC) 2017.
16+
https://arxiv.org/abs/1703.06490
17+
18+
.. autoclass:: pyhealth.models.MedGAN
19+
:members:
20+
:undoc-members:
21+
:show-inheritance:

‎docs/api/models/pyhealth.models.PromptEHR.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
pyhealth.models.PromptEHR
2+
===================================
3+
4+
PromptEHR: prompt-learning BART for synthetic EHR generation. A port of the
5+
reference implementation
6+
(`PromptEHR <https://github.com/RyanWangZf/PromptEHR>`_) that consumes the
7+
standard PyHealth interface and learns via a span-infilling objective with a
8+
reparameterized soft prompt.
9+
10+
Reference:
11+
Wang, Z., & Sun, J. (2022).
12+
*PromptEHR: Conditional Electronic Healthcare Records Generation with
13+
Prompt Learning.*
14+
In Proceedings of EMNLP 2022.
15+
https://aclanthology.org/2022.emnlp-main.185/
16+
17+
.. autoclass:: pyhealth.models.PromptEHR
18+
:members:
19+
:undoc-members:
20+
:show-inheritance:

‎docs/api/tasks.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,7 @@ Available Tasks
212212
COVID-19 CXR Classification <tasks/pyhealth.tasks.COVID19CXRClassification>
213213
DKA Prediction (MIMIC-IV) <tasks/pyhealth.tasks.dka>
214214
Drug Recommendation <tasks/pyhealth.tasks.drug_recommendation>
215+
EHR Generation <tasks/pyhealth.tasks.generate_ehr>
215216
Length of Stay Prediction <tasks/pyhealth.tasks.length_of_stay_prediction>
216217
Medical Transcriptions Classification <tasks/pyhealth.tasks.MedicalTranscriptionsClassification>
217218
Mortality Prediction (Next Visit) <tasks/pyhealth.tasks.mortality_prediction>

‎docs/api/tasks/pyhealth.tasks.generate_ehr.rst‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
pyhealth.tasks.generate_ehr
2+
===========================================
3+
4+
Task that turns a longitudinal EHR dataset into per-patient, per-visit code
5+
sequences for training unconditional synthetic-EHR generators (HALO, GPT2,
6+
PromptEHR, MedGAN, CorGAN), plus helpers to flatten generated output into the
7+
long-form dataframe consumed by :mod:`pyhealth.metrics.generative`.
8+
9+
Task Classes
10+
------------
11+
12+
.. autoclass:: pyhealth.tasks.generate_ehr.EHRGeneration
13+
:members:
14+
:undoc-members:
15+
:show-inheritance:
16+
17+
.. autoclass:: pyhealth.tasks.generate_ehr.EHRGenerationMIMIC3
18+
:members:
19+
:undoc-members:
20+
:show-inheritance:
21+
22+
.. autoclass:: pyhealth.tasks.generate_ehr.EHRGenerationMIMIC4
23+
:members:
24+
:undoc-members:
25+
:show-inheritance:
26+
27+
Helper Functions
28+
----------------
29+
30+
.. autofunction:: pyhealth.tasks.generate_ehr.decode_dataset
31+
32+
.. autofunction:: pyhealth.tasks.generate_ehr.to_evaluation_dataframe

0 commit comments

Comments
 (0)