change src/pipeline folder structure (#149)

* feat: change conf folder structure * feat: change docs readme * feat: change pipelines structure * feat: remove feat_pre_process * feat: update changelog info and version * Update CHANGELOG for version 1.1.0
JoseRZapata · Feb 27, 2025 · c83ce9c · c83ce9c
1 parent 2f3efb6
commit c83ce9c
Show file tree

Hide file tree

Showing 18 changed files with 37 additions and 15 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Released]
 
+## [1.1.0] - 2024-02-26
+
+Changed pipeline folder structure in src to match MLOps description by Google
+
+
+### Added 🚀
+
+- google mlops link reference
+
+### Changed 💥
+
+- File restructuring:
+
+{{cookiecutter.repo_name}}/src/pipelines: change folder structure
+conf/: change folder names
+
 ## [1.0.1] - 2024-02-14
 
 The Main change in this to configure vscode settings for run RUFF native server and updates dependencies and pre-commit hooks.

diff --git a/README.md b/README.md
@@ -133,7 +133,8 @@ cruft update
 
 Folder structure for data science projects  [why?](https://towardsdatascience.com/the-importance-of-layered-thinking-in-data-engineering-a09f685edc71)
 
-[Data structure]
+- [Data structure]
+- [Pipelines based on mlops by google](https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning#mlops_level_1_ml_pipeline_automation)
 
 ```bash
 .
@@ -190,7 +191,6 @@ Folder structure for data science projects  [why?](https://towardsdatascience.co
 │   │   ├── feat_imputation             # feature engineering imputation    
 │   │   ├── feat_new_features           # feature engineering new features
 │   │   ├── feat_pipelines              # feature engineering pipelines
-│   │   ├── feat_preprocess_strings     # feature engineering pre process strings
 │   │   ├── feat_scaling                # feature engineering scaling data
 │   │   ├── feat_selection              # feature engineering feature selection
 │   │   ├── feat_strings                # feature engineering strings
@@ -201,11 +201,13 @@ Folder structure for data science projects  [why?](https://towardsdatascience.co
 │   │   ├── model_validation            # model validation
 │   │   └── reports                     # reports
 │   ├── pipelines
-│   │   ├── data_etl                    # data extraction, transformation, and loading
-│   │   ├── feature_engineering         # prepare data for modeling
-│   │   ├── model_evaluation            # evaluate model performance
-│   │   ├── model_prediction            # model predictions
-│   │   └── model_train                 # train models    
+│   │   ├── data_extraction             # data extraction loading
+│   │   ├── data_validation             # data validation
+│   │   ├── data_preparation            # prepare data for modeling (cleaning, feature engineering)
+│   │   ├── model_train                 # train models
+│   │   ├── model_eval                  # evaluate model performance using test set
+│   │   ├── model_validation            # compare model vs baseline
+│   │   └── model_serving               # deploy model to serve predictions
 ├── tests                               # test code for your project
 │   └── test_mock.py                    # example test file
 ├── .editorconfig                       # editor configuration

diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [project]
 name = "data-science-project-template"
-version = "1.0.1"
+version = "1.1.0"
 description = "A modern template for data science projects with all the necessary tools for experiment, development, testing, and deployment. From notebooks to production."
 authors = [
     {name = "Jose R. Zapata", email = "https://joserzapata.github.io/"},

diff --git a/{{cookiecutter.repo_name}}/README.md b/{{cookiecutter.repo_name}}/README.md
@@ -61,6 +61,9 @@ uv add --group dev plotly
 
 ## 🗃️ Project structure
 
+- [Data structure]
+- [Pipelines based on mlops by google](https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning#mlops_level_1_ml_pipeline_automation)
+
 ```bash
 .
 ├── codecov.yml                         # configuration for codecov
@@ -113,7 +116,6 @@ uv add --group dev plotly
 │   │   ├── feat_imputation             # feature engineering imputation    
 │   │   ├── feat_new_features           # feature engineering new features
 │   │   ├── feat_pipelines              # feature engineering pipelines
-│   │   ├── feat_preprocess_strings     # feature engineering pre process strings
 │   │   ├── feat_scaling                # feature engineering scaling data
 │   │   ├── feat_selection              # feature engineering feature selection
 │   │   ├── feat_strings                # feature engineering strings
@@ -124,11 +126,13 @@ uv add --group dev plotly
 │   │   ├── model_validation            # model validation
 │   │   └── reports                     # reports
 │   ├── pipelines
-│   │   ├── data_etl                    # data extraction, transformation, and loading
-│   │   ├── feature_engineering         # prepare data for modeling
-│   │   ├── model_evaluation            # evaluate model performance
-│   │   ├── model_prediction            # model predictions
-│   │   └── model_train                 # train models    
+│   │   ├── data_extraction             # data extraction loading
+│   │   ├── data_validation             # data validation
+│   │   ├── data_preparation            # prepare data for modeling (cleaning, feature engineering)
+│   │   ├── model_train                 # train models
+│   │   ├── model_eval                  # evaluate model performance using test set
+│   │   ├── model_validation            # compare model vs baseline
+│   │   └── model_serving               # deploy model to serve predictions
 ├── tests                               # test code for your project
 └── .vscode                             # vscode configuration
     ├── extensions.json                 # list of recommended extensions
@@ -149,7 +153,7 @@ This project was generated from [@JoseRZapata]'s [data science project template]
 [coverage.py]: https://coverage.readthedocs.io/
 [Cruft]: https://cruft.github.io/cruft/
 [data science project template]: https://github.com/JoseRZapata/data-science-project-template
-[Data structure]: {{cookiecutter.repo_name}}/data/README.md
+[Data structure]: https://github.com/JoseRZapata/data-science-project-template/blob/main/{{cookiecutter.repo_name}}/data/README.md
 [hydra]: https://hydra.cc/
 [Mypy]: http://mypy-lang.org/
 [Notebook template]: {{cookiecutter.repo_name}}/notebooks/notebook_template.ipynb

diff --git a/...utter.repo_name}}/conf/deploy/__init__.py → ...o_name}}/conf/data_extraction/__init__.py b/...utter.repo_name}}/conf/deploy/__init__.py → ...o_name}}/conf/data_extraction/__init__.py
diff --git a/...iecutter.repo_name}}/conf/etl/__init__.py → ..._name}}/conf/data_preparation/__init__.py b/...iecutter.repo_name}}/conf/etl/__init__.py → ..._name}}/conf/data_preparation/__init__.py
diff --git a/...r.repo_name}}/conf/evaluation/__init__.py → ...o_name}}/conf/data_validation/__init__.py b/...r.repo_name}}/conf/evaluation/__init__.py → ...o_name}}/conf/data_validation/__init__.py
diff --git a/...me}}/conf/feature_engineering/__init__.py → ...r.repo_name}}/conf/model_eval/__init__.py b/...me}}/conf/feature_engineering/__init__.py → ...r.repo_name}}/conf/model_eval/__init__.py
diff --git a/...cutter.repo_name}}/conf/model/__init__.py → ...epo_name}}/conf/model_serving/__init__.py b/...cutter.repo_name}}/conf/model/__init__.py → ...epo_name}}/conf/model_serving/__init__.py
diff --git a/...r.repo_name}}/conf/prediction/__init__.py → ....repo_name}}/conf/model_train/__init__.py b/...r.repo_name}}/conf/prediction/__init__.py → ....repo_name}}/conf/model_train/__init__.py
diff --git a/...cutter.repo_name}}/conf/train/__init__.py → ..._name}}/conf/model_validation/__init__.py b/...cutter.repo_name}}/conf/train/__init__.py → ..._name}}/conf/model_validation/__init__.py
diff --git a/.../libs/feat_preprocess_strings/__init__.py → ...src/pipelines/data_extraction/__init__.py b/.../libs/feat_preprocess_strings/__init__.py → ...src/pipelines/data_extraction/__init__.py
diff --git a/...o_name}}/src/pipelines/deploy/__init__.py → ...rc/pipelines/data_preparation/__init__.py b/...o_name}}/src/pipelines/deploy/__init__.py → ...rc/pipelines/data_preparation/__init__.py
diff --git a/...repo_name}}/src/pipelines/etl/__init__.py → ...src/pipelines/data_validation/__init__.py b/...repo_name}}/src/pipelines/etl/__init__.py → ...src/pipelines/data_validation/__init__.py
diff --git a/...me}}/src/pipelines/evaluation/__init__.py → ...me}}/src/pipelines/model_eval/__init__.py b/...me}}/src/pipelines/evaluation/__init__.py → ...me}}/src/pipelines/model_eval/__init__.py
diff --git a/...e}}/src/pipelines/feature_eng/__init__.py → ...}/src/pipelines/model_serving/__init__.py b/...e}}/src/pipelines/feature_eng/__init__.py → ...}/src/pipelines/model_serving/__init__.py
diff --git a/...me}}/src/pipelines/prediction/__init__.py → ...e}}/src/pipelines/model_train/__init__.py b/...me}}/src/pipelines/prediction/__init__.py → ...e}}/src/pipelines/model_train/__init__.py
diff --git a/...po_name}}/src/pipelines/train/__init__.py → ...rc/pipelines/model_validation/__init__.py b/...po_name}}/src/pipelines/train/__init__.py → ...rc/pipelines/model_validation/__init__.py