Skip to content

Refactor ML validation and Torch lantern installation #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PondiB merged 5 commits into from
May 28, 2026
Merged

Refactor ML validation and Torch lantern installation #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cda5428
fix torch Lantern install/verify and pre-copy cubes before cube_regul…
PondiB May 6, 2026
f043b15
Reorganize ML demos (2026 UC2 + SW paper), trim LPS notebooks, and ex…
PondiB May 7, 2026
86f0d20
Merge pull request #130 from PondiB/dev
PondiB May 7, 2026
93e3377
Refactor ml_validate and ml_validate_kfold to return enriched models …
PondiB May 27, 2026
cade8c7
Refactor ml_validate and ml_validate_kfold to return enriched models …
PondiB May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/docker-build-verify.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ on:
- "inst/**"

concurrency:
group: docker-verify-${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
group: docker-verify-${{ github.workflow }}-${{ github.event.pull_request.number
|| github.sha }}
cancel-in-progress: true

jobs:
Expand Down
9 changes: 6 additions & 3 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,12 @@ RUN R -e "install.packages(c('devtools'), repos='https://cloud.r-project.org')"
# Install basic packages first (these are dependencies for others)
RUN R -e "install.packages(c('gdalcubes','plumber','useful','s2','sf','rstac','geojsonsf', 'jsonlite', 'base64enc', 'ids', 'callr' , 'sits'), repos='https://cloud.r-project.org')"

# Install torch's Lantern library (required for deep learning models like TempCNN)
# TORCH_INSTALL=1 auto-confirms the interactive prompt
RUN TORCH_INSTALL=1 R -e "torch::install_torch()"
# Install torch's Lantern library (required for deep learning models like TempCNN).
# TORCH_INSTALL=1 auto-confirms the interactive prompt.
# The second R call fails the build immediately if install_torch() left Lantern
# absent — without it, install failures are silent and only surface at runtime.
RUN TORCH_INSTALL=1 R -e "torch::install_torch()" && \
R -e "t <- torch::torch_tensor(1L); stopifnot(as.numeric(t) == 1L); message('Lantern OK')"


# create directories
Expand Down
26 changes: 26 additions & 0 deletions docker/plumber.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,32 @@
library(openeocraft)
library(plumber)

# Ensure torch Lantern binary is present before any ML process tries to use it.
# install_torch() is a no-op when Lantern is already installed; it only runs the
# download when the binary is missing (first local run, or a broken Docker build).
if (requireNamespace("torch", quietly = TRUE)) {
lantern_ok <- tryCatch({
torch::torch_tensor(1L)
TRUE
}, error = function(e) FALSE)

if (!lantern_ok) {
message("[startup] Lantern binary not found — running install_torch()...")
tryCatch(
torch::install_torch(),
error = function(e) {
warning(
"[startup] install_torch() failed: ", conditionMessage(e),
"\nTorch-based processes (TempCNN, TAE, LightTAE) will not work.",
call. = FALSE
)
}
)
} else {
message("[startup] torch Lantern OK.")
}
}

# Increase request body size limit (100MB) to handle large payloads
# such as serialized training datasets in process graphs
options(plumber.maxRequestSize = 1024 * 1024 * 100)
Expand Down
5 changes: 5 additions & 0 deletions docker/server.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# Suppress torch's interactive "Do you want to continue?" prompt before any
# package load triggers torch's .onLoad hook (via openeocraft -> sits -> torch).
# Must be set here, before library() calls, for it to take effect in time.
Sys.setenv(TORCH_INSTALL = "1")

# TO-DO: show R errors into terminal
# Determine plumber file path - works both locally and in Docker
docker_path <- "/opt/dockerfiles/docker/plumber.R"
Expand Down
16 changes: 16 additions & 0 deletions docker/verify-r-deps.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,19 @@ if (!all(ok)) {
}

message("Docker R dependency check OK (", length(pkgs), " packages).")

# requireNamespace("torch") succeeds even when the Lantern C binary is absent —
# it only loads the R namespace. Verify Lantern is actually present by calling
# a real tensor operation, which triggers the binary load.
message("Verifying torch Lantern binary loads...")
tryCatch({
t <- torch::torch_tensor(1L)
stopifnot(as.numeric(t) == 1L)
message("torch Lantern OK.")
}, error = function(e) {
stop(
"torch Lantern binary failed to load: ", conditionMessage(e),
"\nRun install_torch() to install the missing binary.",
call. = FALSE
)
})
183 changes: 22 additions & 161 deletions inst/demo-lps-2025/00_ml_month.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"id": "0751accd",
"metadata": {},
"outputs": [],
Expand All @@ -32,21 +32,10 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"id": "10662c95",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<Connection to 'http://127.0.0.1:8000/' with BasicBearerAuth>"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"connection = openeo.connect(\"http://127.0.0.1:8000\")\n",
"connection.authenticate_basic(\"user\", \"password\")"
Expand All @@ -62,20 +51,10 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"id": "76f35649",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Available collections: ['mpc-landsat-c2-l2', 'mpc-sentinel-2-l2a', 'mpc-sentinel-1-grd', 'mpc-sentinel-1-rtc', 'aws-sentinel-2-l2a', 'aws-landsat-c2-l2', 'cdse-sentinel-2-l2a']\n",
"\n",
"Available processes: ['load_collection', 'mlm_class_random_forest', 'mlm_class_svm', 'mlm_class_xgboost', 'ml_class_mlp', 'mlm_class_tempcnn', 'mlm_class_tae', 'mlm_class_lighttae', 'ml_fit', 'ml_predict', 'ml_validate', 'ml_validate_kfold', 'ml_tune_grid', 'ml_tune_random', 'ml_predict_probabilities', 'ml_uncertainty_class', 'ml_smooth_class', 'ml_label_class', 'cube_regularize', 'ndvi', 'merge_cubes', 'filter_bands', 'save_result', 'load_result', 'export_cube', 'import_cube', 'export_ml_model', 'import_ml_model', 'save_ml_model', 'load_stac_ml', 'load_ml_model']\n"
]
}
],
"outputs": [],
"source": [
"print(\"Available collections:\", connection.list_collection_ids())\n",
"print(\"\\nAvailable processes:\", [p[\"id\"] for p in connection.list_processes()])"
Expand All @@ -91,7 +70,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"id": "24c76f46",
"metadata": {},
"outputs": [],
Expand All @@ -116,13 +95,13 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"id": "98ebca43",
"metadata": {},
"outputs": [],
"source": [
"datacube = connection.load_collection(\n",
" \"aws-sentinel-2-l2a\",\n",
" \"mpc-sentinel-2-l2a\",\n",
" spatial_extent=bbox,\n",
" temporal_extent=temporal_extent\n",
")"
Expand All @@ -138,7 +117,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"id": "d94d6c9e",
"metadata": {},
"outputs": [],
Expand All @@ -163,7 +142,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"id": "1fef4727",
"metadata": {},
"outputs": [],
Expand All @@ -181,7 +160,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"id": "b26c07aa",
"metadata": {},
"outputs": [],
Expand All @@ -199,7 +178,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"id": "2da13a45",
"metadata": {},
"outputs": [],
Expand All @@ -221,7 +200,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"id": "31cc072e",
"metadata": {},
"outputs": [],
Expand All @@ -246,7 +225,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": null,
"id": "daa43808",
"metadata": {},
"outputs": [],
Expand All @@ -266,42 +245,10 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": null,
"id": "13611ba7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <script>\n",
" if (!window.customElements || !window.customElements.get('openeo-model-builder')) {\n",
" var el = document.createElement('script');\n",
" el.src = \"https://cdn.jsdelivr.net/npm/@openeo/vue-components@2/assets/openeo.min.js\";\n",
" document.head.appendChild(el);\n",
"\n",
" var font = document.createElement('font');\n",
" font.as = \"font\";\n",
" font.type = \"font/woff2\";\n",
" font.crossOrigin = true;\n",
" font.href = \"https://use.fontawesome.com/releases/v5.13.0/webfonts/fa-solid-900.woff2\"\n",
" document.head.appendChild(font);\n",
" }\n",
" </script>\n",
" <openeo-model-builder>\n",
" <script type=\"application/json\">{\"id\": \"4cc9ca2532094316b844d16ab271de4e\", \"explicit-zoom\": true, \"height\": \"400px\", \"value\": {\"process_graph\": {\"mlmclasstempcnn1\": {\"process_id\": \"mlm_class_tempcnn\", \"arguments\": {\"batch_size\": 64, \"cnn_dropout_rates\": [0.2, 0.2, 0.2], \"cnn_kernels\": [7, 7, 7], \"cnn_layers\": [256, 256, 256], \"dense_layer_dropout_rate\": 0.5, \"dense_layer_nodes\": 256, \"epochs\": 100, \"learning_rate\": 0.0005, \"optimizer\": \"adam\", \"seed\": 42, \"verbose\": false}}, \"mlfit1\": {\"process_id\": \"ml_fit\", \"arguments\": {\"model\": {\"from_node\": \"mlmclasstempcnn1\"}, \"target\": \"label\", \"training_set\": \"https://github.com/Open-Earth-Monitor/openeocraft/raw/main/inst/demo-paper-2025/data/samples_deforestation_rondonia.rds\"}}, \"savemlmodel1\": {\"process_id\": \"save_ml_model\", \"arguments\": {\"data\": {\"from_node\": \"mlfit1\"}, \"name\": \"tempcnn_rondonia\", \"options\": {}}, \"result\": true}}}}</script>\n",
" </openeo-model-builder>\n",
" "
],
"text/plain": [
"<openeo.rest.ml_extension.MLModel at 0x114ba1750>"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"tempcnn_model.save_ml_model(name=\"tempcnn_rondonia\")"
]
Expand All @@ -318,7 +265,7 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": null,
"id": "1fb2d7cb",
"metadata": {},
"outputs": [],
Expand All @@ -328,49 +275,17 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": null,
"id": "d49d7f98",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <script>\n",
" if (!window.customElements || !window.customElements.get('openeo-model-builder')) {\n",
" var el = document.createElement('script');\n",
" el.src = \"https://cdn.jsdelivr.net/npm/@openeo/vue-components@2/assets/openeo.min.js\";\n",
" document.head.appendChild(el);\n",
"\n",
" var font = document.createElement('font');\n",
" font.as = \"font\";\n",
" font.type = \"font/woff2\";\n",
" font.crossOrigin = true;\n",
" font.href = \"https://use.fontawesome.com/releases/v5.13.0/webfonts/fa-solid-900.woff2\"\n",
" document.head.appendChild(font);\n",
" }\n",
" </script>\n",
" <openeo-model-builder>\n",
" <script type=\"application/json\">{\"id\": \"b1eba19a6b924eea92eeeb74fecfb2f1\", \"explicit-zoom\": true, \"height\": \"400px\", \"value\": {\"process_graph\": {\"loadcollection1\": {\"process_id\": \"load_collection\", \"arguments\": {\"id\": \"aws-sentinel-2-l2a\", \"spatial_extent\": {\"west\": -63.33, \"south\": -12.03, \"east\": -62.43, \"north\": -11.13, \"crs\": 4326}, \"temporal_extent\": [\"2022-01-01\", \"2022-12-31\"]}}, \"cuberegularize1\": {\"process_id\": \"cube_regularize\", \"arguments\": {\"data\": {\"from_node\": \"loadcollection1\"}, \"period\": \"P16D\", \"resolution\": 300}}, \"ndvi1\": {\"process_id\": \"ndvi\", \"arguments\": {\"data\": {\"from_node\": \"cuberegularize1\"}, \"nir\": \"B08\", \"red\": \"B04\", \"target_band\": \"NDVI\"}}, \"mlmclasstempcnn1\": {\"process_id\": \"mlm_class_tempcnn\", \"arguments\": {\"batch_size\": 64, \"cnn_dropout_rates\": [0.2, 0.2, 0.2], \"cnn_kernels\": [7, 7, 7], \"cnn_layers\": [256, 256, 256], \"dense_layer_dropout_rate\": 0.5, \"dense_layer_nodes\": 256, \"epochs\": 100, \"learning_rate\": 0.0005, \"optimizer\": \"adam\", \"seed\": 42, \"verbose\": false}}, \"mlfit1\": {\"process_id\": \"ml_fit\", \"arguments\": {\"model\": {\"from_node\": \"mlmclasstempcnn1\"}, \"target\": \"label\", \"training_set\": \"https://github.com/Open-Earth-Monitor/openeocraft/raw/main/inst/demo-paper-2025/data/samples_deforestation_rondonia.rds\"}}, \"mlpredict1\": {\"process_id\": \"ml_predict\", \"arguments\": {\"data\": {\"from_node\": \"ndvi1\"}, \"model\": {\"from_node\": \"mlfit1\"}}}, \"saveresult1\": {\"process_id\": \"save_result\", \"arguments\": {\"data\": {\"from_node\": \"mlpredict1\"}, \"format\": \"GeoTiff\", \"options\": {}}, \"result\": true}}}}</script>\n",
" </openeo-model-builder>\n",
" "
],
"text/plain": [
"<openeo.rest.result.SaveResult at 0x10a230b20>"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"result"
]
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": null,
"id": "e9777398",
"metadata": {},
"outputs": [],
Expand All @@ -383,64 +298,10 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": null,
"id": "db973d75",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:00:00 Job '444dbf2f651d5451cb0085cd5f7bbd3c': send 'start'\n",
"0:00:03 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:00:08 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:00:15 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:00:23 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:00:32 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:00:45 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:01:00 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:01:19 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:01:43 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:02:13 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:02:50 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:03:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:04:35 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:05:35 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:06:35 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:07:35 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:08:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:09:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:10:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:11:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:12:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:13:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:14:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:15:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:16:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:17:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:18:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:19:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:20:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:21:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:22:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:23:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:24:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:25:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': running (progress N/A)\n",
"0:26:36 Job '444dbf2f651d5451cb0085cd5f7bbd3c': finished (progress N/A)\n"
]
},
{
"data": {
"text/plain": [
"[PosixPath('data/output_month/openeocraft_merged.tif'),\n",
" PosixPath('data/output_month/job-results.json')]"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"job.start_and_wait()\n",
"results = job.get_results()\n",
Expand Down
Loading