GBM Baseline results higher than those reported in the paper?

[Georg-va]

First of all, thank you very much for this great work and dataset.

We are currently trying to reproduce your experiments using the provided code and data. However, we are observing the following discrepancies:

1. GBM results are consistently higher than those reported in your paper (especially for mortality prediction).
2. MOTOR model performance is lower than the reported results.
3. For mortality prediction, GBM outperforms MOTOR for all three prediction windows using only EHR data.
4. Readmission results are worse than in the paper.
5. PH metrics are significantly off (see attached images).
   We would appreciate it if you could help us identify whether our data processing or execution steps could be causing these differences.

Fig 1: Our auroc results using provided data and code

Fig 2: Results shown in the INSPECT paper

Data Preprocessing Steps:
Here are the exact steps we performed to get the scripts running:

Data source: Used the 100 parquet files provided on the Redivis website (meds_omop_inspect.tar.gz).
Input csv:

• Replaced MEDS_BIRTH and MEDS_DEATH with codes: SNOMED/184099003 and Condition Type/OMOP4822053, respectively
• Combined all parquet files into a single large CSV for use in the script.
• Script 1 (1_csv_to_database):Used to create the patient database.
• Converted numeric_value and textual_value columns into a unified value column (to get the script running).
• Included lab_units and visit_ids.

Masteranon file: Since no masteranon file was found, we created it using the following:

• labels_20250611.tsv
• study_mapping_20250611.tsv (for patient_id, procedure_datetime, etc.)
• splits_20250611.tsv (for predefined splits)
• All were mapped on image_id (datasets from Aimi).
• Labels: Label prediction (e.g., for readmission) did not work → We overwrote with provided labels (labels_20250611.tsv).
• For mortality prediction, our generated labels had 99.68% overlap with the provided labels.

Current Issues / Questions

1. Do you see any issues with the above steps that might explain the higher GBM and lower MOTOR results?
2. Have you observed variability in GBM vs. MOTOR performance across runs?
3. Could you provide guidance on how to correctly generate labels for readmission? (Our attempts failed, and we had to fall back to provided labels.)

[zzs95]

Hi @Georg-va, thanks for sharing the preprocessing details! I’m trying to reproduce the pipeline with the Redivis meds_omop_inspect.tar.gz data but ran into some issues.

Could you clarify (1) which version of femr / femr-cuda you used, and (2) whether the replacement of MEDS_BIRTH and MEDS_DEATH with SNOMED/184099003 and Condition Type/OMOP4822053 was done directly in the combined CSV before running etl_simple_femr? (3) When I run etl_simple_femr in 1_csv_to_database.py, I get the error prefix_index = concept_id.index("/") ValueError: substring not found Did you perform any additional preprocessing or mapping steps to fix this?

Thanks!

UPDATE
(2) The replacement of MEDS_BIRTH and MEDS_DEATH with SNOMED/184099003 and Condition Type/OMOP4822053 was done directly in the combined CSV before running etl_simple_femr.