Skip to content

Nov 2023 with fixed NA #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Nov 2023 with fixed NA #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9d618b3
Update main.yml
brieanne-forbes May 16, 2024
f96c41e
Change all N/A values to NA.
brieanne-forbes May 16, 2024
6aa569b
00151 on Fri May 17 04:02:33 UTC 2024
May 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ jobs:
pw-user: 'sfgary'
resource-pool-names: 'cloud'
workflow-name: 'sl_core'
workflow-parameters: '{"startCmd": "workflow.sh", "repos": { "ml_arch_repo": "[email protected]:parallelworks/dynamic-learning-rivers", "ml_arch_branch": "ICON-ModEx-v2", "push_to_gh": "true", "ml_code_repo": "https://github.com/parallelworks/sl_core", "ml_data_repo": "https://github.com/parallelworks/global-river-databases", "whost": { "type": "computeResource", "id": "636dec4b565b200008495822" }, "miniconda_loc": "/home/__USER__/.miniconda3", "my_env": "superlearner" }, "superlearner": { "train_test_data": "scripts/prep_06_output_final_train.csv", "predict_data": "scripts/prep_06_output_final_predict", "num_inputs": "25", "superlearner_conf": "sample_inputs/superlearner_conf_TTR_mms_log10_neg.py", "work_dir_base": "ml_models/sl_", "hpo": "true", "cross_val_score": "false", "smogn": "false", "onnx": "false", "backend": "loky", "n_jobs": "8", "num_inst": "10", "fpi_corr_cutoff": "50" } }'
workflow-parameters: '{"startCmd": "workflow.sh", "repos": { "ml_arch_repo": "[email protected]:parallelworks/dynamic-learning-rivers", "ml_arch_branch": "Nov-2023-fixedNA", "push_to_gh": "true", "ml_code_repo": "https://github.com/parallelworks/sl_core", "ml_data_repo": "https://github.com/parallelworks/global-river-databases", "whost": { "type": "computeResource", "id": "636dec4b565b200008495822" }, "miniconda_loc": "/home/__USER__/.miniconda3", "my_env": "superlearner" }, "superlearner": { "train_test_data": "scripts/prep_06_output_final_train.csv", "predict_data": "scripts/prep_06_output_final_predict", "num_inputs": "25", "superlearner_conf": "sample_inputs/superlearner_conf_TTR_mms_log10_neg.py", "work_dir_base": "ml_models/sl_", "hpo": "true", "cross_val_score": "false", "smogn": "false", "onnx": "false", "backend": "loky", "n_jobs": "8", "num_inst": "10", "fpi_corr_cutoff": "50" } }'
390 changes: 195 additions & 195 deletions input_data/ICON-ModEx_Data.csv
View file
Open in desktop

Large diffs are not rendered by default.

Binary file added ml_models/sl_0/SuperLearners.pkl
View file
Open in desktop
Binary file not shown.
3 changes: 3 additions & 0 deletions ml_models/sl_0/classical-metrics.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"Normalized_Respiration_Rate_mg_DO_per_H_per_L_sediment": 0.9215824536204621
}
2 changes: 2 additions & 0 deletions ml_models/sl_0/fpi.std.err
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/home/sfgary/.miniconda3/envs/superlearner/lib/python3.9/site-packages/skopt/space/space.py:110: UserWarning: Dimension [-0.99, 0.99, 'uniform'] was inferred to Real(low=-0.99, high=0.99, prior='uniform', transform='identity'). In upcoming versions of scikit-optimize, it will be inferred to Categorical(categories=(-0.99, 0.99, 'uniform'), prior=None). See the documentation of the check_dimension function for the upcoming API.
warnings.warn(
57 changes: 57 additions & 0 deletions ml_models/sl_0/fpi.std.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
Parsing SuperLearner FPI arguments...
--model_dir
--predict_var
--num_inputs
--corr_cutoff
--predict_data
Submodels within SuperLearner and their weights:
['nusvr-rbf', 'nusvr-lin', 'nusvr-poly', 'nusvr-sig', 'knn-uni', 'knn-dist', 'pls', 'mlp', 'ridge', 'lasso', 'linear', 'enet', 'huber', 'xgb', 'etr']
[0. 0. 0.01809048 1.03373511 0.07584601 0.
0.00352946 0. 0.04230056 0.02854969 0. 0.
0. 0.80105367 0. ]
Did not detect any one-hot features.
Started with 25 features.
Finishing with 18 features in 3 groups.
pre_mm_cyr,run_mm_cyr,pre_mm_cdi,tmp_dc_cdi;RA_dm,RA_SO,RA_ms_av,RA_ms_di;Mean_DO_mg_per_L,Mean_DO_percent_saturation,crp_pc_cse,ire_pc_cse,for_pc_cse,gwt_cm_cav,slp_dg_cav,ele_mt_cav,tmp_dc_cyr,Mean_Temp_Deg_C
Loading model for job: 0
FPI on stacked ensemble...
FPI on ML model: nusvr-sig...
FPI on ML model: xgb...
Avg_Ratiostack0 Std_Ratiostack0
Feature
pre_mm_cyr,run_mm_cyr,pre_mm_cdi,tmp_dc_cdi 3.356069 0.175626
RA_dm,RA_SO,RA_ms_av,RA_ms_di 3.706418 0.126283
Mean_DO_mg_per_L,Mean_DO_percent_saturation,crp... 7.782892 0.272012
pH 1.588919 0.133131
prm_pc_cse 1.032120 0.014893
gla_pc_cse 0.999675 0.001301
pst_pc_cse 1.398878 0.100030
ppd_pk_cav 1.634504 0.145447
sgr_dk_rav 1.697786 0.180473
dor_pc_pva 1.585081 0.168342
--------nusvr-sig---------
Feature
pre_mm_cyr,run_mm_cyr,pre_mm_cdi,tmp_dc_cdi 1.001265
RA_dm,RA_SO,RA_ms_av,RA_ms_di 0.999994
Mean_DO_mg_per_L,Mean_DO_percent_saturation,crp_pc_cse,ire_pc_cse,for_pc_cse,gwt_cm_cav,slp_dg_cav,ele_mt_cav,tmp_dc_cyr,Mean_Temp_Deg_C 1.002887
pH 1.000903
prm_pc_cse 1.000000
gla_pc_cse 1.000000
pst_pc_cse 0.999875
ppd_pk_cav 1.000044
sgr_dk_rav 0.999985
dor_pc_pva 0.999964
Name: Avg_Rationusvr-sig0, dtype: float64
--------xgb---------
Feature
pre_mm_cyr,run_mm_cyr,pre_mm_cdi,tmp_dc_cdi 3.693109
RA_dm,RA_SO,RA_ms_av,RA_ms_di 3.871569
Mean_DO_mg_per_L,Mean_DO_percent_saturation,crp_pc_cse,ire_pc_cse,for_pc_cse,gwt_cm_cav,slp_dg_cav,ele_mt_cav,tmp_dc_cyr,Mean_Temp_Deg_C 8.437513
pH 1.699551
prm_pc_cse 0.999957
gla_pc_cse 1.000000
pst_pc_cse 1.400325
ppd_pk_cav 1.570874
sgr_dk_rav 1.888092
dor_pc_pva 1.529334
Name: Avg_Ratioxgb0, dtype: float32
4 changes: 4 additions & 0 deletions ml_models/sl_0/hold-out-metrics-high-low-split.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"r2_high": 0.5261037466185012,
"r2_low": 0.08042113189474676
}
5 changes: 5 additions & 0 deletions ml_models/sl_0/hold-out-metrics.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"Normalized_Respiration_Rate_mg_DO_per_H_per_L_sediment": 0.6425611731797971,
"Normalized_Respiration_Rate_mg_DO_per_H_per_L_sediment_high": -0.4211458489683779,
"Normalized_Respiration_Rate_mg_DO_per_H_per_L_sediment_low": -66.6600701969094
}
11 changes: 11 additions & 0 deletions ml_models/sl_0/model_fpi_results_df
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Feature,Avg_Rationusvr-sig0,Std_Rationusvr-sig0,Avg_Ratioxgb0,Std_Ratioxgb0
"pre_mm_cyr,run_mm_cyr,pre_mm_cdi,tmp_dc_cdi",1.0012650020541511,0.0003512435208026101,3.6931086,0.18488343
"RA_dm,RA_SO,RA_ms_av,RA_ms_di",0.9999939262051148,0.0001522039136923974,3.8715687,0.20636418
"Mean_DO_mg_per_L,Mean_DO_percent_saturation,crp_pc_cse,ire_pc_cse,for_pc_cse,gwt_cm_cav,slp_dg_cav,ele_mt_cav,tmp_dc_cyr,Mean_Temp_Deg_C",1.0028872512228855,0.0006391530753469736,8.437513,0.22184964
pH,1.0009032867746113,0.000182478906507304,1.6995512,0.23127481
prm_pc_cse,1.0,0.0,0.9999567,0.00010276365
gla_pc_cse,1.0,0.0,1.0,0.0
pst_pc_cse,0.9998754869832014,7.295768397832177e-05,1.4003246,0.08893507
ppd_pk_cav,1.000044433962566,2.4827745460881275e-05,1.5708745,0.11568346
sgr_dk_rav,0.9999849943585395,3.563770463683232e-06,1.8880919,0.20384881
dor_pc_pva,0.9999644831687743,2.4850550852229005e-05,1.5293338,0.19650507
Loading