Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/terraform-validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
strategy:
fail-fast: false
matrix:
cloud: [aws, azure, gcp]
cloud: [aws, azure, gcp, openstack]
steps:
- name: Checkout repository
uses: actions/checkout@v6
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ crash.log
terraform/aws/terraform.tfvars
terraform/azure/terraform.tfvars
terraform/gcp/terraform.tfvars
terraform/openstack/terraform.tfvars

# inst/demo-lps-2025/00_ml_month.ipynb — openEO job downloads
inst/demo-lps-2025/data/output-month/
Expand Down
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ URL: https://github.com/Open-Earth-Monitor/openeocraft
BugReports: https://github.com/Open-Earth-Monitor/openeocraft/issues
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.3
Depends:
R (>= 4.5.0)
Imports:
Expand Down Expand Up @@ -58,3 +57,4 @@ Collate:
'pgraphs.R'
'mock.R'
'run.R'
Config/roxygen2/version: 8.0.0
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,10 @@ docker run -d --name openeocraft --restart unless-stopped \

#### Terraform (recommended)

For repeatable cloud provisioning, use the Terraform stacks in [`terraform/README.md`](terraform/README.md). Defaults target **GPU with at least 32 GB RAM** on AWS (`g4dn.2xlarge`), Azure (`Standard_NC8as_T4_v3`), and GCP (`g2-standard-8`), with NVIDIA drivers and `docker run --gpus all`. See each cloud README for quota, auth, and CPU fallbacks.
For repeatable cloud provisioning, use the Terraform stacks in [`terraform/README.md`](terraform/README.md). Defaults target **GPU with at least 32 GB RAM** on AWS, Azure, and GCP; OpenStack uses a configurable `flavor_name` with a **32 GB RAM minimum**. See each cloud README for quota, auth, and CPU fallbacks.

```bash
cd terraform/aws # or azure/ or gcp/
cd terraform/aws # or azure/, gcp/, or openstack/
cp terraform.tfvars.example terraform.tfvars
# Set credentials, key pair / SSH key, ssh_cidr_blocks, api_cidr_blocks
terraform init
Expand Down
102 changes: 102 additions & 0 deletions inst/examples/06_ml_tempcnn_tune_grid.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# TempCNN fine-tuning (grid search) + inference via the openEO R client.
#
# Prerequisites:
# - OpenEOcraft backend running locally (e.g. source docker/plumber.R in R/RStudio,
# or docker compose up)
# - openeo R package installed
#
# Run from repo root:
# Rscript inst/examples/06_ml_tempcnn_tune_grid.R
#
# Default connection below matches docker/plumber.R demo credentials (user brian).
# With a local plumber process, job artefacts are written under the backend
# workdir, e.g. <workdir>/workspace/<user>/jobs/<job_id>/ (tuning_results.json,
# saved model, GeoTIFF). Set the workdir when starting the API if needed.
#
# Cube bands must match the training RDS timeline and features (P16D; add NDVI
# when samples were built with an NDVI band). See inst/demo-lps-2025/00_ml_month.ipynb.
# Grid search uses a small param_grid for demo wall time; see inst/demo-lps-2025/.

library(openeo)

con <- connect(
host = "http://127.0.0.1:8000",
user = "brian",
password = "123456"
)

p <- processes()

deforestation_data <-
"https://github.com/e-sensing/sitsdata/raw/main/data/samples_deforestation_rondonia.rds"

tempcnn_model_init <- p$mlm_class_tempcnn(
optimizer = "adam",
learning_rate = 0.0005,
seed = 42,
epochs = 20,
batch_size = 64,
verbose = TRUE
)

param_grid <- list(
learning_rate = c(0.0005, 0.0001),
epochs = c(20, 40)
)

tempcnn_tuned <- p$ml_tune_grid(
model = tempcnn_model_init,
training_data = deforestation_data,
target = "label",
parameters = param_grid,
scoring = "accuracy",
cv = 0,
seed = 42
)

tempcnn_model <- p$save_ml_model(
data = tempcnn_tuned,
name = "tempcnn_rondonia_tuned_v1",
return_model = TRUE
)

datacube <- p$load_collection(
id = "mpc-sentinel-2-l2a",
spatial_extent = list(
west = -63.50,
south = -8.92,
east = -63.35,
north = -8.78
),
temporal_extent = c("2022-01-01", "2022-12-31"),
bands = list(
"b02", "b03", "b04", "b05", "b06", "b07", "b08",
"b11", "b12", "b8a"
)
)

datacube <- p$cube_regularize(
data = datacube,
period = "P16D",
resolution = 30
)

datacube <- p$ndvi(
data = datacube,
red = "B04",
nir = "B08",
target_band = "NDVI"
)

data <- p$ml_predict(data = datacube, model = tempcnn_model)

ml_job <- p$save_result(data = data, format = "GTiff")

job <- create_job(
graph = ml_job,
title = "TempCNN fine-tuning + inference",
description = "Grid-search TempCNN on Rondonia samples; predict GeoTIFF"
)
job <- start_job(job)

describe_job(job)
8 changes: 5 additions & 3 deletions terraform/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# OpenEOcraft Terraform (multi-cloud)

Repeatable provisioning for OpenEOcraft on **AWS**, **Azure**, and **Google Cloud**. Each stack launches a single **x86_64** Ubuntu 22.04 VM, installs Docker via cloud-init, optionally installs the **NVIDIA driver + NVIDIA Container Toolkit**, pulls the OpenEOcraft image, and starts the API on port **8000**.
Repeatable provisioning for OpenEOcraft on **AWS**, **Azure**, **Google Cloud**, and **OpenStack**. Each stack launches a single **x86_64** Ubuntu 22.04 VM, installs Docker via cloud-init, optionally installs the **NVIDIA driver + NVIDIA Container Toolkit**, pulls the OpenEOcraft image, and starts the API on port **8000**.

All defaults target **at least 32 GB instance RAM** (30 GB container limit, leaving headroom for the OS).

Expand All @@ -9,6 +9,7 @@ All defaults target **at least 32 GB instance RAM** (30 GB container limit, leav
| AWS | [`aws/`](aws/) | `g4dn.2xlarge` (32 GB, 1× T4) | `m5.2xlarge` (32 GB) |
| Azure | [`azure/`](azure/) | `Standard_NC8as_T4_v3` (56 GB, 1× T4) | `Standard_D8s_v5` (32 GB) |
| GCP | [`gcp/`](gcp/) | `g2-standard-8` (32 GB, 1× L4) | `n2-standard-8` (32 GB) |
| OpenStack | [`openstack/`](openstack/) | cloud-specific GPU flavor | `flavor_name` ≥ 32 GB RAM (default `m1.8xlarge`) |

Bootstrap scripts live in [`shared/cloud_init.tftpl`](shared/cloud_init.tftpl) and are shared across clouds. Workspace disks are mounted only via stable cloud-specific device paths (never heuristic disk guessing); if the data disk is not ready, bootstrap falls back to a directory on the root filesystem.

Expand All @@ -17,7 +18,7 @@ Bootstrap scripts live in [`shared/cloud_init.tftpl`](shared/cloud_init.tftpl) a
Pick a cloud, copy the example variables, edit credentials and CIDR blocks, then apply:

```bash
cd terraform/aws # or azure/ or gcp/
cd terraform/aws # or azure/, gcp/, or openstack/
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars (key pair / SSH key, your public IP /32)
terraform init
Expand All @@ -34,14 +35,15 @@ Allow **15–20 minutes** on first boot when `enable_gpu = true`.
| ----------- | ----- |
| [Terraform](https://developer.hashicorp.com/terraform/install) >= 1.5 | |
| Cloud account with billing enabled | GPU instances require quota in most accounts |
| SSH access | AWS: EC2 key pair name; Azure/GCP: SSH public key |
| SSH access | AWS/OpenStack: key pair name; Azure/GCP: SSH public key |
| Your public IP as CIDR | `curl -s ifconfig.me` → `x.x.x.x/32` for SSH and API rules |

## Cloud-specific docs

- [AWS EC2](aws/README.md)
- [Azure VM](azure/README.md)
- [Google Compute Engine](gcp/README.md)
- [OpenStack Nova](openstack/README.md)

## What each stack creates

Expand Down
24 changes: 24 additions & 0 deletions terraform/openstack/.terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

104 changes: 104 additions & 0 deletions terraform/openstack/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# OpenEOcraft on OpenStack (Terraform)

This stack provisions a single **x86_64** Ubuntu 22.04 Nova instance on a dedicated tenant network with a floating IP, installs Docker via cloud-init, optionally installs the **NVIDIA driver + NVIDIA Container Toolkit** (GPU mode), pulls the OpenEOcraft image, and starts the API on port **8000**.

**Default target:** CPU flavor with **≥ 32 GB RAM** (`flavor_name`, default `m1.8xlarge` — verify on your cloud with `openstack flavor list`).

## Prerequisites

| Requirement | Notes |
| ----------- | ----- |
| [Terraform](https://developer.hashicorp.com/terraform/install) >= 1.5 | |
| OpenStack project with compute + networking + block storage | |
| Nova **key pair** | `openstack keypair create …` |
| Flavor with **≥ 32 GB RAM** | `openstack flavor list` — names vary by cloud |
| External network for floating IPs | Often `public` or `ext-net` |
| Your public IP as CIDR | `curl -s ifconfig.me` → `x.x.x.x/32` |

### OpenStack authentication

Set standard OpenStack environment variables (from `openstack rc` or your cloud dashboard):

```bash
export OS_AUTH_URL=...
export OS_USERNAME=...
export OS_PASSWORD=...
export OS_PROJECT_NAME=...
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_REGION_NAME=...
openstack token issue
```

Or use `clouds.yaml` / application credentials as supported by the [OpenStack Terraform provider](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs).

## Quick start

```bash
cd terraform/openstack
cp terraform.tfvars.example terraform.tfvars
```

Edit `terraform.tfvars`:

- `key_pair_name` — your Nova key pair
- `flavor_name` — a flavor with **≥ 32768 MB RAM** on your cloud
- `external_network_name` / `floating_ip_pool` — your provider external network
- `ssh_cidr_blocks` / `api_cidr_blocks` — your public IP `/32`

```bash
terraform init
terraform plan
terraform apply
```

After apply:

```bash
terraform output api_url
```

**Bootstrap time:** allow **15–20 minutes** on first boot when `enable_gpu = true`.

## What gets created

| Resource | Purpose |
| -------- | ------- |
| Network, subnet, router | Tenant network with external gateway |
| Security group | SSH (22) and OpenEOcraft API (8000) from your CIDR lists |
| Nova instance | Ubuntu 22.04, Docker, optional NVIDIA stack, OpenEOcraft container |
| Cinder volume (optional) | Persistent `/var/openeo/workspace` on `/dev/vdb` (default 200 GB) |
| Floating IP | Public access to the API |

## Variables

See [`variables.tf`](variables.tf). Key settings:

| Variable | Default | Notes |
| -------- | ------- | ----- |
| `flavor_name` | `m1.8xlarge` | Must exist on your cloud with **≥ 32 GB RAM** |
| `enable_gpu` | `false` | Set `true` only with a GPU flavor; names are cloud-specific |
| `docker_cpus` / `docker_memory_gb` | `8` / `30` | Container limits |
| `workspace_volume_size_gb` | `200` | Set to `0` to skip separate Cinder volume |
| `external_network_name` | `public` | Provider external network for router + floating IP |

### GPU (optional)

GPU flavor names differ widely between OpenStack deployments. Copy [`terraform.gpu.tfvars.example`](terraform.gpu.tfvars.example), set `flavor_name` to a GPU flavor from your cloud, and set `enable_gpu = true`.

## Verify after apply

```bash
ssh ubuntu@$(terraform output -raw public_ip)

nvidia-smi # when enable_gpu = true
sudo docker ps
sudo docker logs openeocraft --tail 50
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/
```

## Teardown

```bash
terraform destroy
```
Loading
Loading