diff --git a/.github/workflows/check_license_header.yml b/.github/workflows/check_license_header.yml index 5ded52c..394c232 100644 --- a/.github/workflows/check_license_header.yml +++ b/.github/workflows/check_license_header.yml @@ -33,8 +33,8 @@ jobs: FAILED=0 for FILE in $CHANGED_FILES; do - # Skip markdown and json files - if [[ "$FILE" == *.md ]] || [[ "$FILE" == *.json ]]; then + # Skip markdown, json and csv files + if [[ "$FILE" == *.md ]] || [[ "$FILE" == *.json ]] || [[ "$FILE" == *.csv ]]; then echo "Skipping: $FILE" continue fi diff --git a/hadoop-spark-oci-sample/native/.gitignore b/hadoop-spark-oci-sample/native/.gitignore new file mode 100644 index 0000000..d39a417 --- /dev/null +++ b/hadoop-spark-oci-sample/native/.gitignore @@ -0,0 +1,23 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +# Local Terraform state and provider plugins +.terraform/ +.terraform.lock.hcl +*.tfstate +*.tfstate.* +*.tfstate.backup +crash.log +crash.*.log + +# Sensitive variable values +*.tfvars +!*.tfvars.example + +# CLI override files +override.tf +override.tf.json +*_override.tf +*_override.tf.json + +# Plan output +*.tfplan diff --git a/hadoop-spark-oci-sample/native/README.md b/hadoop-spark-oci-sample/native/README.md new file mode 100644 index 0000000..28f1a03 --- /dev/null +++ b/hadoop-spark-oci-sample/native/README.md @@ -0,0 +1,290 @@ +# Spark + Hadoop on OCI — Terraform stack + +A configurable Terraform deployment that stands up a complete Spark + Hadoop +platform on Oracle Cloud Infrastructure: + +- **OCI Big Data Service** — managed Hadoop cluster (HDFS, YARN, Spark, Hive, …) +- **OCI Data Flow** — serverless Spark applications, optional warm pool +- **Object Storage** — buckets for Spark scripts, logs and warehouse output +- **VCN** — created on demand, or you can reuse an existing one +- **IAM** — dynamic group + policy so Data Flow runs can talk to your buckets + +The stack runs from the Terraform CLI **and** from the OCI Resource Manager +(`schema.yaml` ships in the repo). + +--- + +## Repository layout + +``` +. +├── provider.tf # OCI + random providers +├── variables.tf # All input variables +├── locals.tf # Derived values +├── network.tf # VCN, subnets, gateways, security lists +├── object_storage.tf # Buckets for Data Flow +├── iam.tf # Dynamic group + policies for Data Flow runs +├── bds.tf # Big Data Service (Hadoop) cluster +├── dataflow.tf # Data Flow applications + warm pool +├── operator.tf # Operator VM + OCI Bastion + instance-principal IAM +├── outputs.tf +├── schema.yaml # Resource Manager form schema +├── terraform.tfvars.example # Three showcase profiles +├── templates/ +│ └── operator_init.sh.tftpl # Operator cloud-init (tooling + deployment.env) +├── examples/ +│ ├── pi.py # Bundled sample PySpark job +│ ├── demo.sql # Bundled sample Spark SQL job +│ └── bootstrap.sh # Example BDS bootstrap script +└── use-cases/ # End-to-end scenario walkthroughs (see below) + └── lib.sh # Capability-check helpers sourced by the run scripts +``` + +--- + +## Example use cases + +The [`use-cases/`](use-cases/) folder contains self-contained, end-to-end +walkthroughs that show what the stack can do. The intended flow is to **deploy +via Resource Manager with the operator VM enabled**, open an **OCI Bastion** +session into the operator, and run the scripts there — each script self-checks +what the stack actually deployed and tells you which form field to change if a +use case isn't supported. Start with [`use-cases/README.md`](use-cases/README.md). + +| Use case | Showcases | +|----------|-----------| +| [01 — Serverless ETL](use-cases/01-serverless-etl/) | Cheapest Spark on OCI: CSV → Parquet via Data Flow, no cluster | +| [02 — Hadoop cluster analytics](use-cases/02-hadoop-cluster-analytics/) | `spark-submit` on YARN + HDFS on a managed BDS cluster | +| [03 — Warm-pool low latency](use-cases/03-warm-pool-low-latency/) | Repeated/scheduled jobs that start in seconds via a warm pool | +| [04 — Secure HA production](use-cases/04-secure-ha-production/) | Kerberos + Ranger, HA, elastic compute-only workers, bootstrap tuning | + +--- + +## Operator VM + OCI Bastion + +Set **`deploy_operator = true`** (form: *Deploy operator VM behind OCI Bastion*) +to add a small jump/control host in the private subnet, reachable only through +the **OCI Bastion** service — no public IP, no internet-facing SSH. It is the +recommended way to drive the use cases. + +What it gives you: + +- The use-case scripts staged on the box — Terraform uploads `use-cases/` to the + scripts bucket, and the operator pulls them at boot with instance-principal + auth — plus a `deployment.env` descriptor of what the stack deployed (so the + scripts can precheck capabilities). This works from Resource Manager because + the operator self-pulls; nothing is pushed from the apply host. +- **Instance-principal auth** — a dynamic group + policy let the VM submit Data + Flow runs and use Object Storage with no API keys on the host. +- A ready-made connect command in the `operator_bastion_session_hint` output. + +Connect: + +```bash +terraform output -raw operator_bastion_session_hint # prints the session command +# run it, wait for SUCCEEDED, then SSH via the bastion (use -A for BDS use cases) +``` + +If you can't create tenancy-level IAM (`create_iam_resources = false`), +pre-create the operator's dynamic group and policy out of band: + +- Dynamic group matching rule: `ALL {instance.id = ''}` +- Policy statements (in the deployment compartment): + - `Allow dynamic-group to manage dataflow-family in compartment id ` + - `Allow dynamic-group to read buckets in compartment id ` + - `Allow dynamic-group to manage objects in compartment id ` + - `Allow dynamic-group to read objectstorage-namespaces in tenancy` + +--- + +## Running from the Terraform CLI + +### Prerequisites + +- Terraform `>= 1.3` +- An OCI API key configured (`~/.oci/config` or env vars) +- IAM rights to create VCN, BDS, Data Flow, Object Storage **and** to create + a tenancy-level dynamic group + policy (or pre-create them out of band) + +### Steps + +```bash +cp terraform.tfvars.example terraform.tfvars +# edit terraform.tfvars — set tenancy_ocid, compartment_ocid, region, +# ssh_public_key, bds_cluster_admin_password + +terraform init +terraform plan +terraform apply +``` + +To tear it all down: + +```bash +terraform destroy +``` + +--- + +## Running from OCI Resource Manager + +1. Zip the repo (exclude `.terraform/`, `*.tfstate*`): + + ```bash + zip -r spark-hadoop-stack.zip . \ + -x '.git/*' -x '.terraform/*' -x '*.tfstate*' -x 'terraform.tfvars' + ``` + +2. In the OCI Console go to **Developer Services → Resource Manager → Stacks + → Create stack**. +3. Source: **My configuration → .zip file**, upload `spark-hadoop-stack.zip`. +4. Resource Manager reads `schema.yaml` and renders the input form. Fill it + in. +5. Run **Plan**, then **Apply**. + +`schema.yaml` hides the CLI-only auth variables, groups the inputs into +logical panels (Networking, BDS Master / Utility / Workers, Data Flow, Warm +Pool …), and conditionally shows the "existing VCN/subnet" pickers only when +**Create VCN** is unticked. + +--- + +## What gets deployed + +### Networking + +| `create_vcn` | Result | +| --- | --- | +| `true` (default) | A new VCN (`10.0.0.0/16` by default) with a public + private subnet, Internet Gateway, NAT Gateway, Service Gateway, route tables and security lists. | +| `false` | The stack reuses the VCN + subnets whose OCIDs you supply (`existing_vcn_id`, `existing_private_subnet_id`, `existing_public_subnet_id`). Nothing network-level is created or modified. | + +### Big Data Service (Hadoop) + +Toggled by `deploy_bds`. Configurable knobs: + +- **Cluster version** — `ODH0.9` … `ODH2.1` +- **Cluster profile** — `HADOOP`, `HADOOP_EXTENDED`, `SPARK`, `HIVE`, + `HBASE`, `TRINO`, `KAFKA`, `DATAFLOW`, `DATA_SCIENCE`, `AIRFLOW` +- **High availability** — 1+1 or 2+2 master + utility nodes +- **Security** — Kerberos + Ranger toggle (`bds_is_secure`) +- **Per node-group sizing** — shape, OCPUs, memory, block volume, count for + master / utility / worker / compute-only worker pools +- **Bootstrap script** — `bds_bootstrap_script_url` points at an Object + Storage URL; BDS runs the script on every node at creation time. Use it + to push custom `core-site.xml`, `hdfs-site.xml`, `yarn-site.xml`, + `spark-defaults.conf`, install extra packages, drop keytabs, etc. + `examples/bootstrap.sh` is a starting template. + +### Data Flow (Spark) + +Toggled by `deploy_dataflow`. The list `dataflow_applications` accepts any +number of applications — each one is a separate showcase of Spark on +OCI. Each entry exposes: + +- `language` — `PYTHON` / `JAVA` / `SCALA` / `SQL` +- `spark_version` — e.g. `3.5.0`, `3.2.1` +- `file_uri` — pointer to the entry-point script/jar in Object Storage. + If left empty, the stack falls back to the bundled sample for the + language (`examples/pi.py` for PYTHON, `examples/demo.sql` for SQL), + which it uploads into the scripts bucket for you. +- `driver_shape`, `driver_ocpus`, `driver_memory_gbs` +- `executor_shape`, `executor_ocpus`, `executor_memory_gbs`, `num_executors` +- `configuration` — a map of Spark properties (e.g. `spark.sql.shuffle.partitions`, + `spark.dynamicAllocation.enabled`, …). This is the per-application Spark + config knob. + +Optional **warm pool** (`dataflow_create_pool`) keeps a small set of +executors warm so Data Flow runs start in seconds instead of ~1 minute. + +### Object Storage + +Three buckets are created (each independently toggle-able): + +- `-dataflow-scripts` — holds Spark entry points +- `-dataflow-logs` — Data Flow stdout/stderr + Spark event logs +- `-dataflow-warehouse` — Spark SQL output (Parquet/etc.) + +### IAM + +Toggled by `create_iam_resources` (default `true`). When enabled the stack +creates: + +- **For Data Flow** (`deploy_dataflow`): a **dynamic group** matching + `resource.type='dataflowrun'` in the target compartment, plus a **policy** + granting that group read/write on Object Storage in the compartment. +- **For BDS** (`deploy_bds`): a **policy** granting the `bdsprod` service + principal access to the VCN/subnet so it can attach cluster nodes. Without + this, cluster creation fails with *"not enough permissions to access subnet + or vcn details"*. The policy is scoped to `compartment_ocid` by default; set + `bds_network_compartment_ocid` when reusing an existing VCN/subnet that lives + in a different compartment. + +Dynamic groups and policies live at the tenancy root, so the caller needs +IAM admin rights on the tenancy. If you cannot grant those, set +`create_iam_resources = false` and pre-create the resources out of band: + +```text +# Data Flow dynamic group — name it whatever you like +Matching rule: + ALL {resource.type='dataflowrun', resource.compartment.id=''} + +# Data Flow policy — attach in the tenancy +Allow dynamic-group to read buckets in compartment id +Allow dynamic-group to manage objects in compartment id +Allow dynamic-group to read objectstorage-namespaces in tenancy + +# BDS network policy — attach where the VCN/subnet lives () +Allow service bdsprod to {VNIC_READ, VNIC_ATTACH, VNIC_DETACH, VNIC_CREATE, VNIC_DELETE, VNIC_ATTACHMENT_READ, SUBNET_READ, VCN_READ, SUBNET_ATTACH, SUBNET_DETACH, INSTANCE_ATTACH_SECONDARY_VNIC, INSTANCE_DETACH_SECONDARY_VNIC} in compartment id +``` + +--- + +## Showcase profiles + +`terraform.tfvars.example` includes three commented-out profiles: + +| Profile | Use case | +| --- | --- | +| **Minimal** | Data Flow only, no Hadoop cluster — quick Spark try-out. | +| **Standard** | Small Hadoop cluster + a couple of Spark apps + warm pool. Two Data Flow apps: one with default Spark config, one with adaptive query execution + Kryo + dynamic allocation. | +| **Production-ish** | HA Hadoop with compute-only workers, secure cluster, custom bootstrap script, larger warm pool. | + +Mix and match: any field can be overridden independently. + +--- + +## Customising Hadoop and Spark + +| Knob | Where | +| --- | --- | +| Per-application Spark properties (Data Flow) | `dataflow_applications[*].configuration` map | +| Cluster-wide Hadoop / Spark config (BDS) | `bds_bootstrap_script_url` — edits `core-site.xml`, `hdfs-site.xml`, `yarn-site.xml`, `spark-defaults.conf` directly on the nodes. See `examples/bootstrap.sh`. | +| Per-node sizing (BDS) | `bds_*_shape`, `bds_*_ocpus`, `bds_*_memory_gbs`, `bds_*_block_volume_gbs` | +| Compute / storage separation | Add compute-only workers via `bds_compute_only_worker_count` | +| Cluster role | `bds_cluster_profile` (full Hadoop vs Spark-only vs Hive vs HBase vs Trino vs …) | +| Kerberos / Ranger | `bds_is_secure` | +| Warm Spark executors | `dataflow_create_pool`, `dataflow_pool_*` | + +--- + +## Outputs + +After `apply`: + +- `bds_cluster_id`, `bds_master_node_ips`, `bds_utility_node_ips` — + SSH into the utility node to reach Ambari / Cloudera Manager / Hue. +- `dataflow_application_ids` — submit runs with + `oci data-flow run create --application-id ` or the console. +- `scripts_bucket_uri`, `logs_bucket_name`, `warehouse_bucket_name` — + upload your own scripts and consume logs / results. + +--- + +## Notes + +- Some IAM resources (`oci_identity_dynamic_group`, `oci_identity_policy`) + are created at the **tenancy** level. The principal running this stack + needs `manage dynamic-groups` + `manage policies` in the tenancy. +- BDS provisioning takes ~30 minutes. +- Data Flow application creation is fast; cost is incurred only when a run + is submitted (or when a warm pool is running). diff --git a/hadoop-spark-oci-sample/native/bds.tf b/hadoop-spark-oci-sample/native/bds.tf new file mode 100644 index 0000000..363dd0b --- /dev/null +++ b/hadoop-spark-oci-sample/native/bds.tf @@ -0,0 +1,134 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# OCI Big Data Service (managed Hadoop) cluster +############################################################################### + +resource "oci_bds_bds_instance" "this" { + count = var.deploy_bds ? 1 : 0 + + compartment_id = var.compartment_ocid + display_name = local.bds_display_name + cluster_version = var.bds_cluster_version + cluster_profile = var.bds_cluster_profile + cluster_admin_password = base64encode(var.bds_cluster_admin_password) + cluster_public_key = var.ssh_public_key + + is_high_availability = var.bds_is_high_availability + is_secure = var.bds_is_secure + + # ----- Master nodes --------------------------------------------------------- + master_node { + shape = var.bds_master_shape + subnet_id = local.private_subnet_id + block_volume_size_in_gbs = var.bds_master_block_volume_gbs + number_of_nodes = var.bds_is_high_availability ? 2 : 1 + + dynamic "shape_config" { + for_each = can(regex("Flex$", var.bds_master_shape)) ? [1] : [] + content { + ocpus = var.bds_master_ocpus + memory_in_gbs = var.bds_master_memory_gbs + } + } + } + + # ----- Utility nodes (Ambari / Hue / Cloudera Manager / Zeppelin) ----------- + util_node { + shape = var.bds_utility_shape + subnet_id = local.private_subnet_id + block_volume_size_in_gbs = var.bds_utility_block_volume_gbs + number_of_nodes = var.bds_is_high_availability ? 2 : 1 + + dynamic "shape_config" { + for_each = can(regex("Flex$", var.bds_utility_shape)) ? [1] : [] + content { + ocpus = var.bds_utility_ocpus + memory_in_gbs = var.bds_utility_memory_gbs + } + } + } + + # ----- Worker (data) nodes -------------------------------------------------- + worker_node { + shape = var.bds_worker_shape + subnet_id = local.private_subnet_id + block_volume_size_in_gbs = var.bds_worker_block_volume_gbs + number_of_nodes = var.bds_worker_count + + dynamic "shape_config" { + for_each = can(regex("Flex$", var.bds_worker_shape)) ? [1] : [] + content { + ocpus = var.bds_worker_ocpus + memory_in_gbs = var.bds_worker_memory_gbs + } + } + } + + # ----- Compute-only worker nodes (optional, for elastic Spark/YARN) --------- + dynamic "compute_only_worker_node" { + for_each = var.bds_compute_only_worker_count > 0 ? [1] : [] + content { + shape = var.bds_compute_only_worker_shape + subnet_id = local.private_subnet_id + block_volume_size_in_gbs = 150 + number_of_nodes = var.bds_compute_only_worker_count + + dynamic "shape_config" { + for_each = can(regex("Flex$", var.bds_compute_only_worker_shape)) ? [1] : [] + content { + ocpus = var.bds_compute_only_worker_ocpus + memory_in_gbs = var.bds_compute_only_worker_memory_gbs + } + } + } + } + + # Optional bootstrap script — user-supplied Object Storage URL with a shell + # script that BDS runs on every node at cluster creation. This is the hook + # for customising core-site.xml, yarn-site.xml, hdfs-site.xml, + # spark-defaults.conf, etc. + bootstrap_script_url = var.bds_bootstrap_script_url + + # cidr_block is the Oracle-managed network BDS creates for the cluster — it + # must NOT overlap the customer subnet the nodes attach to (var.vcn_cidr_block + # would, since the subnet lives inside it). + network_config { + cidr_block = var.bds_oracle_network_cidr + is_nat_gateway_required = var.create_vcn ? false : true + } + + freeform_tags = var.freeform_tags + + # The bdsprod service principal must be able to read the VCN/subnet before + # the cluster attaches nodes, otherwise creation fails with "not enough + # permissions to access subnet or vcn details". + depends_on = [oci_identity_policy.bds_network] + + lifecycle { + ignore_changes = [ + # BDS may inject extra services post-create depending on cluster profile. + cluster_version, + ] + + precondition { + condition = !var.deploy_bds || length(var.ssh_public_key) > 0 + error_message = "ssh_public_key must be set when deploy_bds = true." + } + + precondition { + condition = !var.deploy_bds || length(var.bds_cluster_admin_password) >= 8 + error_message = "bds_cluster_admin_password must be at least 8 characters when deploy_bds = true." + } + + precondition { + condition = !var.deploy_bds || var.bds_is_high_availability == var.bds_is_secure + error_message = "OCI BDS requires bds_is_high_availability and bds_is_secure to be logically equivalent: either both true (HA + Kerberos) or both false." + } + + precondition { + condition = !var.deploy_bds || !var.create_vcn || var.bds_oracle_network_cidr != var.vcn_cidr_block + error_message = "bds_oracle_network_cidr must not overlap the VCN/subnet. It is set equal to vcn_cidr_block; pick a non-overlapping range (default 172.16.0.0/16)." + } + } +} diff --git a/hadoop-spark-oci-sample/native/cleanup.tf b/hadoop-spark-oci-sample/native/cleanup.tf new file mode 100644 index 0000000..c810665 --- /dev/null +++ b/hadoop-spark-oci-sample/native/cleanup.tf @@ -0,0 +1,66 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Teardown helpers +# +# Running the use cases creates state that Terraform does NOT track and that +# blocks `destroy`: +# * Object Storage objects — uploaded job scripts, Data Flow run logs, and +# job output. A bucket cannot be deleted while it still holds objects. +# * The Data Flow warm pool ends up running, and a pool cannot be deleted +# unless it is stopped first. +# +# These destroy-time hooks stop the pool and empty the buckets just before +# Terraform deletes them. They run `oci` on the machine performing the destroy +# (your workstation for a CLI destroy, the Resource Manager runner for an RM +# destroy). Each command is `|| true`, so if that host has no OCI CLI auth the +# hook simply no-ops and `destroy` still proceeds — in that case run +# `use-cases/cleanup.sh` from the operator (instance-principal auth) before +# destroying. +############################################################################### + +resource "null_resource" "pool_stop_on_destroy" { + count = var.deploy_dataflow && var.dataflow_create_pool ? 1 : 0 + + triggers = { + pool_id = oci_dataflow_pool.this[0].id + region = var.region + } + + provisioner "local-exec" { + when = destroy + command = "oci data-flow pool stop --pool-id ${self.triggers.pool_id} --region ${self.triggers.region} --wait-for-state SUCCEEDED || true" + } + + depends_on = [oci_dataflow_pool.this] +} + +resource "null_resource" "buckets_empty_on_destroy" { + count = var.deploy_dataflow ? 1 : 0 + + triggers = { + namespace = local.os_namespace + region = var.region + buckets = join(" ", compact([ + var.dataflow_create_scripts_bucket ? local.scripts_bucket_name : "", + var.dataflow_create_logs_bucket ? local.logs_bucket_name : "", + var.dataflow_create_warehouse_bucket ? local.warehouse_bucket_name : "", + ])) + } + + provisioner "local-exec" { + when = destroy + command = <<-EOT + for b in ${self.triggers.buckets}; do + echo "emptying bucket $b" + oci os object bulk-delete -bn "$b" --namespace ${self.triggers.namespace} --region ${self.triggers.region} --force || true + done + EOT + } + + depends_on = [ + oci_objectstorage_bucket.scripts, + oci_objectstorage_bucket.logs, + oci_objectstorage_bucket.warehouse, + ] +} diff --git a/hadoop-spark-oci-sample/native/dataflow.tf b/hadoop-spark-oci-sample/native/dataflow.tf new file mode 100644 index 0000000..ac24c61 --- /dev/null +++ b/hadoop-spark-oci-sample/native/dataflow.tf @@ -0,0 +1,107 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# OCI Data Flow — managed Spark applications + optional warm pool +############################################################################### + +# Optional warm pool: keeps a small set of executors hot so Data Flow runs +# start in seconds instead of ~1 minute. Apps reference it via pool_id. +resource "oci_dataflow_pool" "this" { + count = var.deploy_dataflow && var.dataflow_create_pool ? 1 : 0 + + compartment_id = var.compartment_ocid + display_name = "${var.resource_prefix}-dataflow-pool" + + configurations { + shape = var.dataflow_pool_shape + min = var.dataflow_pool_min_executors + max = var.dataflow_pool_max_executors + + shape_config { + ocpus = var.dataflow_pool_ocpus + memory_in_gbs = var.dataflow_pool_memory_gbs + } + } + + freeform_tags = var.freeform_tags +} + +locals { + # Resolve the effective file_uri for each application. If the user did not + # supply one, fall back to the sample script we uploaded for that language. + dataflow_apps = var.deploy_dataflow ? { + for app in var.dataflow_applications : + app.name => merge(app, { + resolved_file_uri = ( + length(app.file_uri) > 0 + ? app.file_uri + : ( + contains(keys(local.sample_scripts), app.language) && var.dataflow_create_scripts_bucket && var.dataflow_upload_sample_scripts + ? "${local.scripts_bucket_uri}${local.sample_scripts[app.language].filename}" + : "" + ) + ) + }) + } : {} +} + +resource "oci_dataflow_application" "this" { + for_each = local.dataflow_apps + + compartment_id = var.compartment_ocid + display_name = "${var.resource_prefix}-${each.value.name}" + description = "Showcase Spark application: ${each.value.name}" + + language = each.value.language + spark_version = each.value.spark_version + file_uri = each.value.resolved_file_uri + class_name = each.value.class_name + arguments = each.value.arguments + + driver_shape = each.value.driver_shape + executor_shape = each.value.executor_shape + num_executors = each.value.num_executors + + dynamic "driver_shape_config" { + for_each = can(regex("Flex$", each.value.driver_shape)) ? [1] : [] + content { + ocpus = each.value.driver_ocpus + memory_in_gbs = each.value.driver_memory_gbs + } + } + + dynamic "executor_shape_config" { + for_each = can(regex("Flex$", each.value.executor_shape)) ? [1] : [] + content { + ocpus = each.value.executor_ocpus + memory_in_gbs = each.value.executor_memory_gbs + } + } + + configuration = each.value.configuration + + logs_bucket_uri = local.logs_bucket_uri + warehouse_bucket_uri = local.warehouse_bucket_uri + + pool_id = var.dataflow_create_pool ? oci_dataflow_pool.this[0].id : null + + type = "BATCH" + freeform_tags = var.freeform_tags + + lifecycle { + precondition { + condition = length(each.value.resolved_file_uri) > 0 + error_message = "Application '${each.value.name}' has no file_uri and no bundled sample is available for language ${each.value.language}. Either set file_uri or enable dataflow_upload_sample_scripts." + } + + precondition { + condition = !contains(["JAVA", "SCALA"], each.value.language) || length(each.value.class_name) > 0 + error_message = "Application '${each.value.name}' uses ${each.value.language} but class_name is empty." + } + } + + depends_on = [ + oci_objectstorage_object.sample_scripts, + oci_identity_policy.dataflow, + ] +} diff --git a/hadoop-spark-oci-sample/native/examples/bootstrap.sh b/hadoop-spark-oci-sample/native/examples/bootstrap.sh new file mode 100644 index 0000000..f200800 --- /dev/null +++ b/hadoop-spark-oci-sample/native/examples/bootstrap.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +# Example BDS bootstrap script. Upload this to Object Storage and point +# var.bds_bootstrap_script_url at it. +# +# BDS executes this on every node at cluster creation. Use it to customise +# Hadoop / Spark configuration files, install OS packages, deploy keytabs, +# etc. +# +# Useful environment variables provided by BDS: +# $HOSTTYPE — MASTER | UTILITY | WORKER | COMPUTE_ONLY_WORKER +# $CLUSTER_VERSION +# $CLUSTER_PROFILE +# +# All output goes to /var/logs/oracle/bds/bootstrap.log on the node. + +set -euo pipefail + +echo "Bootstrap starting on $(hostname) — type=${HOSTTYPE:-unknown}" + +# --------------------------------------------------------------------------- +# Spark tuning — applied on master + worker nodes +# --------------------------------------------------------------------------- +if [[ "${HOSTTYPE:-}" == "MASTER" || "${HOSTTYPE:-}" == "WORKER" || "${HOSTTYPE:-}" == "COMPUTE_ONLY_WORKER" ]]; then + SPARK_DEFAULTS=/etc/spark3/conf/spark-defaults.conf + if [[ -f "$SPARK_DEFAULTS" ]]; then + { + echo "" + echo "# Added by stack bootstrap" + echo "spark.sql.adaptive.enabled true" + echo "spark.sql.adaptive.coalescePartitions.enabled true" + echo "spark.serializer org.apache.spark.serializer.KryoSerializer" + echo "spark.shuffle.service.enabled true" + echo "spark.dynamicAllocation.enabled true" + } >> "$SPARK_DEFAULTS" + fi +fi + +# --------------------------------------------------------------------------- +# YARN tuning — applied on master nodes +# --------------------------------------------------------------------------- +if [[ "${HOSTTYPE:-}" == "MASTER" ]]; then + YARN_SITE=/etc/hadoop/conf/yarn-site.xml + if [[ -f "$YARN_SITE" ]]; then + echo "yarn-site.xml already managed by Ambari/Cloudera Manager; edit there instead." + fi +fi + +echo "Bootstrap finished on $(hostname)" diff --git a/hadoop-spark-oci-sample/native/examples/demo.sql b/hadoop-spark-oci-sample/native/examples/demo.sql new file mode 100644 index 0000000..c54d74d --- /dev/null +++ b/hadoop-spark-oci-sample/native/examples/demo.sql @@ -0,0 +1,21 @@ +-- Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +-- The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +-- Showcase Data Flow SQL job. Reads a CSV from Object Storage, aggregates, writes Parquet. +-- Replace the source URI with one of your own (or upload a CSV to the scripts bucket). + +CREATE TABLE IF NOT EXISTS sales USING CSV +OPTIONS ( + path 'oci://@/sales.csv', + header 'true', + inferSchema 'true' +); + +SELECT + region, + product_category, + COUNT(*) AS order_count, + SUM(amount) AS total_revenue +FROM sales +GROUP BY region, product_category +ORDER BY total_revenue DESC; diff --git a/hadoop-spark-oci-sample/native/examples/pi.py b/hadoop-spark-oci-sample/native/examples/pi.py new file mode 100644 index 0000000..e717b4e --- /dev/null +++ b/hadoop-spark-oci-sample/native/examples/pi.py @@ -0,0 +1,33 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +"""Estimate Pi using Monte Carlo. Bundled showcase Spark job for OCI Data Flow.""" + +import sys +from random import random +from operator import add + +from pyspark.sql import SparkSession + + +def main(): + partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 100 + n = 100_000 * partitions + + spark = SparkSession.builder.appName("DataFlowPi").getOrCreate() + + def inside(_): + x = random() * 2 - 1 + y = random() * 2 - 1 + return 1 if x * x + y * y <= 1 else 0 + + count = ( + spark.sparkContext.parallelize(range(1, n + 1), partitions).map(inside).reduce(add) + ) + + print(f"Pi is roughly {4.0 * count / n}") + spark.stop() + + +if __name__ == "__main__": + main() diff --git a/hadoop-spark-oci-sample/native/iam.tf b/hadoop-spark-oci-sample/native/iam.tf new file mode 100644 index 0000000..1dfcdb1 --- /dev/null +++ b/hadoop-spark-oci-sample/native/iam.tf @@ -0,0 +1,83 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# IAM — Tenancy-level resources the stack needs: +# +# * Data Flow: a dynamic group + policy so Data Flow runs (resource +# principals) can read scripts and read/write logs & warehouse buckets in +# this compartment. +# * BDS: a policy granting the bdsprod service principal access to the +# VCN/subnet so it can attach cluster nodes. Without it cluster creation +# fails with "not enough permissions to access subnet or vcn details". +# +# Dynamic groups and policies live in the tenancy root, so the caller needs +# IAM admin rights on the tenancy. If that is not the case, set +# create_iam_resources = false and pre-create the resources out of band — see +# README.md for the exact matching rule and statements. +############################################################################### + +locals { + create_dataflow_iam = var.deploy_dataflow && var.create_iam_resources + create_bds_iam = var.deploy_bds && var.create_iam_resources + create_operator_iam = var.deploy_operator && var.create_iam_resources + create_iam = local.create_dataflow_iam || local.create_bds_iam || local.create_operator_iam + + # Compartment that owns the network the BDS cluster attaches to. Defaults to + # the deployment compartment (where create_vcn = true puts the VCN). + bds_network_compartment = coalesce(var.bds_network_compartment_ocid, var.compartment_ocid) +} + +resource "random_string" "iam_suffix" { + count = local.create_iam ? 1 : 0 + + length = 6 + upper = false + special = false +} + +resource "oci_identity_dynamic_group" "dataflow" { + count = local.create_dataflow_iam ? 1 : 0 + provider = oci.home + + compartment_id = var.tenancy_ocid + name = "${var.resource_prefix}-dataflow-dg-${random_string.iam_suffix[0].result}" + description = "Dynamic group containing Data Flow runs for ${var.resource_prefix}" + matching_rule = "ALL {resource.type='dataflowrun', resource.compartment.id='${var.compartment_ocid}'}" + freeform_tags = var.freeform_tags +} + +resource "oci_identity_policy" "dataflow" { + count = local.create_dataflow_iam ? 1 : 0 + provider = oci.home + + compartment_id = var.tenancy_ocid + name = "${var.resource_prefix}-dataflow-policy-${random_string.iam_suffix[0].result}" + description = "Allow Data Flow runs to read/write the stack's Object Storage buckets." + + statements = [ + "Allow dynamic-group ${oci_identity_dynamic_group.dataflow[0].name} to read buckets in compartment id ${var.compartment_ocid}", + "Allow dynamic-group ${oci_identity_dynamic_group.dataflow[0].name} to manage objects in compartment id ${var.compartment_ocid}", + "Allow dynamic-group ${oci_identity_dynamic_group.dataflow[0].name} to read objectstorage-namespaces in tenancy", + ] + + freeform_tags = var.freeform_tags +} + +# Grants the Big Data Service (bdsprod) service principal the network access it +# needs to create VNICs and attach cluster nodes to the subnet. This must exist +# before the cluster is created, otherwise BDS reports "not enough permissions +# to access subnet or vcn details". The service name "bdsprod" is fixed by OCI. +resource "oci_identity_policy" "bds_network" { + count = local.create_bds_iam ? 1 : 0 + provider = oci.home + + compartment_id = local.bds_network_compartment + name = "${var.resource_prefix}-bds-network-policy-${random_string.iam_suffix[0].result}" + description = "Allow the BDS (bdsprod) service principal to attach clusters to the VCN/subnet." + + statements = [ + "Allow service bdsprod to {VNIC_READ, VNIC_ATTACH, VNIC_DETACH, VNIC_CREATE, VNIC_DELETE, VNIC_ATTACHMENT_READ, SUBNET_READ, VCN_READ, SUBNET_ATTACH, SUBNET_DETACH, INSTANCE_ATTACH_SECONDARY_VNIC, INSTANCE_DETACH_SECONDARY_VNIC} in compartment id ${local.bds_network_compartment}", + ] + + freeform_tags = var.freeform_tags +} diff --git a/hadoop-spark-oci-sample/native/locals.tf b/hadoop-spark-oci-sample/native/locals.tf new file mode 100644 index 0000000..20f8f9b --- /dev/null +++ b/hadoop-spark-oci-sample/native/locals.tf @@ -0,0 +1,62 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +data "oci_identity_availability_domains" "ads" { + compartment_id = var.tenancy_ocid +} + +data "oci_objectstorage_namespace" "ns" { + compartment_id = var.compartment_ocid +} + +locals { + ad_name = data.oci_identity_availability_domains.ads.availability_domains[0].name + + os_namespace = data.oci_objectstorage_namespace.ns.namespace + + vcn_id = var.create_vcn ? oci_core_vcn.this[0].id : var.existing_vcn_id + private_subnet_id = var.create_vcn ? oci_core_subnet.private[0].id : var.existing_private_subnet_id + public_subnet_id = ( + var.create_vcn + ? oci_core_subnet.public[0].id + : var.existing_public_subnet_id + ) + + bds_display_name = coalesce(var.bds_display_name, "${var.resource_prefix}-hadoop") + logs_bucket_name = "${var.resource_prefix}-dataflow-logs" + warehouse_bucket_name = "${var.resource_prefix}-dataflow-warehouse" + scripts_bucket_name = "${var.resource_prefix}-dataflow-scripts" + + # Sample scripts shipped with the stack, keyed by language. When a user + # does not provide a file_uri for an application we fall back to one of + # these. The path is relative to the module root and is uploaded into the + # scripts bucket when dataflow_upload_sample_scripts = true. + sample_scripts = { + PYTHON = { + filename = "pi.py" + source = "${path.module}/examples/pi.py" + } + SQL = { + filename = "demo.sql" + source = "${path.module}/examples/demo.sql" + } + } + + scripts_bucket_uri = ( + var.deploy_dataflow && var.dataflow_create_scripts_bucket + ? "oci://${local.scripts_bucket_name}@${local.os_namespace}/" + : "" + ) + + logs_bucket_uri = ( + var.deploy_dataflow && var.dataflow_create_logs_bucket + ? "oci://${local.logs_bucket_name}@${local.os_namespace}/" + : "" + ) + + warehouse_bucket_uri = ( + var.deploy_dataflow && var.dataflow_create_warehouse_bucket + ? "oci://${local.warehouse_bucket_name}@${local.os_namespace}/" + : "" + ) +} diff --git a/hadoop-spark-oci-sample/native/network.tf b/hadoop-spark-oci-sample/native/network.tf new file mode 100644 index 0000000..6f2bf10 --- /dev/null +++ b/hadoop-spark-oci-sample/native/network.tf @@ -0,0 +1,198 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Network — only created when var.create_vcn = true +############################################################################### + +resource "oci_core_vcn" "this" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + cidr_blocks = [var.vcn_cidr_block] + display_name = "${var.resource_prefix}-vcn" + dns_label = var.vcn_dns_label + freeform_tags = var.freeform_tags +} + +resource "oci_core_internet_gateway" "igw" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-igw" + enabled = true + freeform_tags = var.freeform_tags +} + +resource "oci_core_nat_gateway" "natgw" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-natgw" + freeform_tags = var.freeform_tags +} + +resource "oci_core_service_gateway" "sgw" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-sgw" + + services { + service_id = data.oci_core_services.all[0].services[0].id + } + + freeform_tags = var.freeform_tags +} + +data "oci_core_services" "all" { + count = var.create_vcn ? 1 : 0 + + filter { + name = "name" + values = ["All .* Services In Oracle Services Network"] + regex = true + } +} + +resource "oci_core_route_table" "public" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-public-rt" + + route_rules { + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + network_entity_id = oci_core_internet_gateway.igw[0].id + } + + freeform_tags = var.freeform_tags +} + +resource "oci_core_route_table" "private" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-private-rt" + + # Default route via NAT for outbound internet (yum / pip / etc.) + route_rules { + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + network_entity_id = oci_core_nat_gateway.natgw[0].id + } + + # Oracle services (Object Storage etc.) via the Service Gateway + route_rules { + destination = data.oci_core_services.all[0].services[0].cidr_block + destination_type = "SERVICE_CIDR_BLOCK" + network_entity_id = oci_core_service_gateway.sgw[0].id + } + + freeform_tags = var.freeform_tags +} + +# BDS needs broad intra-cluster connectivity. We open all traffic inside the +# VCN and the standard egress + ICMP rules. Production deployments should +# tighten these to the documented BDS port matrix. +resource "oci_core_security_list" "private" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-private-sl" + + egress_security_rules { + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + protocol = "all" + stateless = false + } + + # Intra-VCN — all protocols + ingress_security_rules { + source = var.vcn_cidr_block + protocol = "all" + stateless = false + } + + # ICMP from the world (path MTU + unreachables) + ingress_security_rules { + source = "0.0.0.0/0" + protocol = "1" + stateless = false + icmp_options { + type = 3 + code = 4 + } + } + + freeform_tags = var.freeform_tags +} + +resource "oci_core_security_list" "public" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + display_name = "${var.resource_prefix}-public-sl" + + egress_security_rules { + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + protocol = "all" + stateless = false + } + + ingress_security_rules { + source = "0.0.0.0/0" + protocol = "6" # TCP + stateless = false + tcp_options { + min = 22 + max = 22 + } + } + + ingress_security_rules { + source = var.vcn_cidr_block + protocol = "all" + stateless = false + } + + freeform_tags = var.freeform_tags +} + +resource "oci_core_subnet" "private" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + cidr_block = var.private_subnet_cidr + display_name = "${var.resource_prefix}-private-snet" + dns_label = "private" + prohibit_public_ip_on_vnic = true + route_table_id = oci_core_route_table.private[0].id + security_list_ids = [oci_core_security_list.private[0].id] + freeform_tags = var.freeform_tags +} + +resource "oci_core_subnet" "public" { + count = var.create_vcn ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this[0].id + cidr_block = var.public_subnet_cidr + display_name = "${var.resource_prefix}-public-snet" + dns_label = "public" + prohibit_public_ip_on_vnic = false + route_table_id = oci_core_route_table.public[0].id + security_list_ids = [oci_core_security_list.public[0].id] + freeform_tags = var.freeform_tags +} diff --git a/hadoop-spark-oci-sample/native/object_storage.tf b/hadoop-spark-oci-sample/native/object_storage.tf new file mode 100644 index 0000000..7c24abb --- /dev/null +++ b/hadoop-spark-oci-sample/native/object_storage.tf @@ -0,0 +1,72 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Object Storage buckets for Data Flow +############################################################################### + +resource "oci_objectstorage_bucket" "logs" { + count = var.deploy_dataflow && var.dataflow_create_logs_bucket ? 1 : 0 + + compartment_id = var.compartment_ocid + namespace = local.os_namespace + name = local.logs_bucket_name + access_type = "NoPublicAccess" + storage_tier = "Standard" + freeform_tags = var.freeform_tags +} + +resource "oci_objectstorage_bucket" "warehouse" { + count = var.deploy_dataflow && var.dataflow_create_warehouse_bucket ? 1 : 0 + + compartment_id = var.compartment_ocid + namespace = local.os_namespace + name = local.warehouse_bucket_name + access_type = "NoPublicAccess" + storage_tier = "Standard" + freeform_tags = var.freeform_tags +} + +resource "oci_objectstorage_bucket" "scripts" { + count = var.deploy_dataflow && var.dataflow_create_scripts_bucket ? 1 : 0 + + compartment_id = var.compartment_ocid + namespace = local.os_namespace + name = local.scripts_bucket_name + access_type = "NoPublicAccess" + storage_tier = "Standard" + freeform_tags = var.freeform_tags +} + +# Upload bundled sample scripts so the default Data Flow application has +# something to execute out of the box. +resource "oci_objectstorage_object" "sample_scripts" { + for_each = ( + var.deploy_dataflow + && var.dataflow_create_scripts_bucket + && var.dataflow_upload_sample_scripts + ) ? local.sample_scripts : {} + + namespace = local.os_namespace + bucket = oci_objectstorage_bucket.scripts[0].name + object = each.value.filename + source = each.value.source +} + +# Stage the use-case assets so the operator VM can pull them onto itself with +# instance-principal auth at boot. Only uploaded when an operator is deployed +# and a scripts bucket exists to hold them. +locals { + operator_assets = ( + var.deploy_operator && var.deploy_dataflow && var.dataflow_create_scripts_bucket + ) ? fileset("${path.module}/use-cases", "**") : toset([]) +} + +resource "oci_objectstorage_object" "operator_assets" { + for_each = local.operator_assets + + namespace = local.os_namespace + bucket = oci_objectstorage_bucket.scripts[0].name + object = "use-cases/${each.value}" + source = "${path.module}/use-cases/${each.value}" +} diff --git a/hadoop-spark-oci-sample/native/operator.tf b/hadoop-spark-oci-sample/native/operator.tf new file mode 100644 index 0000000..fc95351 --- /dev/null +++ b/hadoop-spark-oci-sample/native/operator.tf @@ -0,0 +1,225 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Operator VM + OCI Bastion +# +# An optional jump/control host in the private subnet, reachable ONLY through +# the OCI Bastion service (no public IP, no open SSH from the internet). The +# use-case scripts are staged on it, and it runs with instance-principal auth +# so a user who SSHes in can submit Data Flow runs and use Object Storage with +# no API keys on the box. +# +# Connect: read the `operator_bastion_session_hint` output, run the printed +# `oci bastion session create-managed-ssh ...`, then SSH via the bastion. +############################################################################### + +locals { + create_operator = var.deploy_operator + + # Bastion names allow only letters, digits and underscores — sanitize the + # (possibly hyphenated) resource_prefix. + bastion_name = replace("${var.resource_prefix}-bastion", "-", "_") + + # Warm pool OCID, surfaced to the operator so created apps can attach to it. + operator_pool_id = ( + var.deploy_dataflow && var.dataflow_create_pool + ? oci_dataflow_pool.this[0].id + : "" + ) + + # Bucket names only when the bucket is actually created (else empty → the + # use-case scripts treat the capability as absent). + operator_scripts_bucket = ( + var.deploy_dataflow && var.dataflow_create_scripts_bucket ? local.scripts_bucket_name : "" + ) + operator_logs_bucket = ( + var.deploy_dataflow && var.dataflow_create_logs_bucket ? local.logs_bucket_name : "" + ) + operator_warehouse_bucket = ( + var.deploy_dataflow && var.dataflow_create_warehouse_bucket ? local.warehouse_bucket_name : "" + ) + + # The operator self-pulls the use-case files from the scripts bucket at boot + # (instance principal), so this is only possible when a scripts bucket exists. + operator_assets_available = ( + var.deploy_operator && var.deploy_dataflow && var.dataflow_create_scripts_bucket + ) + + operator_user_data = templatefile("${path.module}/templates/operator_init.sh.tftpl", { + resource_prefix = var.resource_prefix + compartment_ocid = var.compartment_ocid + region = var.region + namespace = local.os_namespace + deploy_bds = var.deploy_bds + bds_high_availability = var.bds_is_high_availability + bds_secure = var.bds_is_secure + deploy_dataflow = var.deploy_dataflow + dataflow_create_pool = var.dataflow_create_pool + dataflow_pool_id = local.operator_pool_id + create_scripts_bucket = var.deploy_dataflow && var.dataflow_create_scripts_bucket + create_logs_bucket = var.deploy_dataflow && var.dataflow_create_logs_bucket + create_warehouse_bucket = var.deploy_dataflow && var.dataflow_create_warehouse_bucket + scripts_bucket = local.operator_scripts_bucket + logs_bucket = local.operator_logs_bucket + warehouse_bucket = local.operator_warehouse_bucket + assets_available = local.operator_assets_available + }) +} + +# Latest Oracle Linux 8 image for the chosen shape. +data "oci_core_images" "operator_ol8" { + count = local.create_operator ? 1 : 0 + + compartment_id = var.compartment_ocid + operating_system = "Oracle Linux" + operating_system_version = "8" + shape = var.operator_shape + sort_by = "TIMECREATED" + sort_order = "DESC" +} + +# Dedicated NSG so SSH (from the bastion, inside the VCN) reaches the operator +# even when an existing subnet is reused whose security list we don't control. +resource "oci_core_network_security_group" "operator" { + count = local.create_operator ? 1 : 0 + + compartment_id = var.compartment_ocid + vcn_id = local.vcn_id + display_name = "${var.resource_prefix}-operator-nsg" + freeform_tags = var.freeform_tags +} + +resource "oci_core_network_security_group_security_rule" "operator_ssh_in" { + count = local.create_operator ? 1 : 0 + + network_security_group_id = oci_core_network_security_group.operator[0].id + direction = "INGRESS" + protocol = "6" # TCP + source = var.vcn_cidr_block + source_type = "CIDR_BLOCK" + + tcp_options { + destination_port_range { + min = 22 + max = 22 + } + } +} + +resource "oci_core_instance" "operator" { + count = local.create_operator ? 1 : 0 + + compartment_id = var.compartment_ocid + availability_domain = local.ad_name + display_name = "${var.resource_prefix}-operator" + shape = var.operator_shape + + shape_config { + ocpus = var.operator_ocpus + memory_in_gbs = var.operator_memory_gbs + } + + create_vnic_details { + subnet_id = local.private_subnet_id + assign_public_ip = false + display_name = "${var.resource_prefix}-operator-vnic" + hostname_label = "operator" + nsg_ids = [oci_core_network_security_group.operator[0].id] + } + + source_details { + source_type = "image" + source_id = data.oci_core_images.operator_ol8[0].images[0].id + boot_volume_size_in_gbs = var.operator_boot_volume_gbs + } + + # Oracle Cloud Agent + Bastion plugin are required for Managed-SSH sessions. + agent_config { + is_management_disabled = false + is_monitoring_disabled = false + + plugins_config { + name = "Bastion" + desired_state = "ENABLED" + } + } + + metadata = { + ssh_authorized_keys = var.ssh_public_key + user_data = base64encode(local.operator_user_data) + } + + freeform_tags = merge(var.freeform_tags, { role = "operator" }) + + lifecycle { + precondition { + condition = length(var.ssh_public_key) > 0 + error_message = "deploy_operator = true requires ssh_public_key to be set (the key you'll use to open bastion sessions)." + } + + # Oracle republishes the OL8 image over time; don't recreate the VM for it. + ignore_changes = [source_details[0].source_id] + } +} + +resource "oci_bastion_bastion" "operator" { + count = local.create_operator && var.create_bastion ? 1 : 0 + + bastion_type = "STANDARD" + compartment_id = var.compartment_ocid + target_subnet_id = local.private_subnet_id + name = local.bastion_name + + client_cidr_block_allow_list = [for c in split(",", var.bastion_client_cidr_allow_list) : trimspace(c) if trimspace(c) != ""] + max_session_ttl_in_seconds = var.bastion_max_session_ttl_seconds + + freeform_tags = var.freeform_tags + + lifecycle { + precondition { + condition = length([for c in split(",", var.bastion_client_cidr_allow_list) : trimspace(c) if trimspace(c) != ""]) > 0 + error_message = "create_bastion = true requires bastion_client_cidr_allow_list to list at least one client /32 (e.g. your workstation's public IP as 203.0.113.4/32)." + } + } +} + +############################################################################### +# IAM — instance principal for the operator. Lets the VM submit Data Flow runs +# and use Object Storage without API keys. Uses the home-region provider like +# the other identity resources. +############################################################################### + +resource "oci_identity_dynamic_group" "operator" { + count = local.create_operator_iam ? 1 : 0 + provider = oci.home + + compartment_id = var.tenancy_ocid + name = "${var.resource_prefix}-operator-dg-${random_string.iam_suffix[0].result}" + description = "Dynamic group for the ${var.resource_prefix} operator VM (instance principal)." + matching_rule = "ALL {instance.id = '${oci_core_instance.operator[0].id}'}" + freeform_tags = var.freeform_tags +} + +resource "oci_identity_policy" "operator" { + count = local.create_operator_iam ? 1 : 0 + provider = oci.home + + # Lives in the tenancy root (like the Data Flow policy) because it grants + # "read objectstorage-namespaces in tenancy" — a policy can only grant on its + # own compartment subtree, and a child compartment's subtree excludes the + # tenancy root above it. The per-compartment statements below still resolve + # fine from here since the tenancy subtree includes every compartment. + compartment_id = var.tenancy_ocid + name = "${var.resource_prefix}-operator-policy-${random_string.iam_suffix[0].result}" + description = "Allow the operator VM to submit Data Flow runs and use the stack's Object Storage." + + statements = [ + "Allow dynamic-group ${oci_identity_dynamic_group.operator[0].name} to manage dataflow-family in compartment id ${var.compartment_ocid}", + "Allow dynamic-group ${oci_identity_dynamic_group.operator[0].name} to read buckets in compartment id ${var.compartment_ocid}", + "Allow dynamic-group ${oci_identity_dynamic_group.operator[0].name} to manage objects in compartment id ${var.compartment_ocid}", + "Allow dynamic-group ${oci_identity_dynamic_group.operator[0].name} to read objectstorage-namespaces in tenancy", + ] + + freeform_tags = var.freeform_tags +} diff --git a/hadoop-spark-oci-sample/native/outputs.tf b/hadoop-spark-oci-sample/native/outputs.tf new file mode 100644 index 0000000..38b1f06 --- /dev/null +++ b/hadoop-spark-oci-sample/native/outputs.tf @@ -0,0 +1,110 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +output "vcn_id" { + description = "OCID of the VCN used by the stack (either newly created or supplied)." + value = local.vcn_id +} + +output "private_subnet_id" { + description = "OCID of the private subnet hosting BDS / Data Flow." + value = local.private_subnet_id +} + +output "public_subnet_id" { + description = "OCID of the public subnet." + value = local.public_subnet_id +} + +output "bds_cluster_id" { + description = "OCID of the BDS (Hadoop) cluster, if deployed." + value = var.deploy_bds ? oci_bds_bds_instance.this[0].id : null +} + +output "bds_cluster_state" { + description = "Lifecycle state of the BDS cluster." + value = var.deploy_bds ? oci_bds_bds_instance.this[0].state : null +} + +output "bds_master_node_ips" { + description = "Private IPs of the BDS master nodes." + value = var.deploy_bds ? [ + for n in oci_bds_bds_instance.this[0].nodes : + n.ip_address if n.node_type == "MASTER" + ] : [] +} + +output "bds_utility_node_ips" { + description = "Private IPs of the BDS utility nodes (Ambari / Cloudera Manager / Hue)." + value = var.deploy_bds ? [ + for n in oci_bds_bds_instance.this[0].nodes : + n.ip_address if n.node_type == "UTILITY" + ] : [] +} + +output "dataflow_application_ids" { + description = "OCIDs of the Data Flow applications, keyed by name." + value = { + for k, v in oci_dataflow_application.this : k => v.id + } +} + +output "dataflow_pool_id" { + description = "OCID of the Data Flow warm pool, if created." + value = var.deploy_dataflow && var.dataflow_create_pool ? oci_dataflow_pool.this[0].id : null +} + +output "logs_bucket_name" { + description = "Name of the Data Flow logs bucket, if created." + value = var.deploy_dataflow && var.dataflow_create_logs_bucket ? oci_objectstorage_bucket.logs[0].name : null +} + +output "warehouse_bucket_name" { + description = "Name of the Data Flow warehouse bucket, if created." + value = var.deploy_dataflow && var.dataflow_create_warehouse_bucket ? oci_objectstorage_bucket.warehouse[0].name : null +} + +output "scripts_bucket_name" { + description = "Name of the Data Flow scripts bucket, if created." + value = var.deploy_dataflow && var.dataflow_create_scripts_bucket ? oci_objectstorage_bucket.scripts[0].name : null +} + +output "scripts_bucket_uri" { + description = "Object Storage URI prefix where Spark scripts live." + value = local.scripts_bucket_uri +} + +output "operator_instance_id" { + description = "OCID of the operator VM, if deployed." + value = var.deploy_operator ? oci_core_instance.operator[0].id : null +} + +output "operator_private_ip" { + description = "Private IP of the operator VM." + value = var.deploy_operator ? oci_core_instance.operator[0].private_ip : null +} + +output "bastion_id" { + description = "OCID of the OCI Bastion, if created." + value = var.deploy_operator && var.create_bastion ? oci_bastion_bastion.operator[0].id : null +} + +output "bastion_name" { + description = "Name of the OCI Bastion, if created." + value = var.deploy_operator && var.create_bastion ? oci_bastion_bastion.operator[0].name : null +} + +output "operator_bastion_session_hint" { + description = "Copy-paste command to open a Managed-SSH bastion session to the operator VM." + value = var.deploy_operator && var.create_bastion ? join(" ", [ + "oci bastion session create-managed-ssh", + "--bastion-id", oci_bastion_bastion.operator[0].id, + "--target-resource-id", oci_core_instance.operator[0].id, + "--target-os-username", "opc", + "--target-private-ip", oci_core_instance.operator[0].private_ip, + "--ssh-public-key-file", "~/.ssh/id_rsa.pub", + "--session-ttl", tostring(var.bastion_max_session_ttl_seconds), + "--display-name", "operator-session", + "--wait-for-state", "SUCCEEDED", + ]) : null +} diff --git a/hadoop-spark-oci-sample/native/provider.tf b/hadoop-spark-oci-sample/native/provider.tf new file mode 100644 index 0000000..6353e5d --- /dev/null +++ b/hadoop-spark-oci-sample/native/provider.tf @@ -0,0 +1,59 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +terraform { + required_version = ">= 1.3.0" + + required_providers { + oci = { + source = "oracle/oci" + version = ">= 5.30.0" + } + random = { + source = "hashicorp/random" + version = ">= 3.5.0" + } + null = { + source = "hashicorp/null" + version = ">= 3.2.0" + } + } +} + +# When the stack is launched from OCI Resource Manager, Resource Manager +# automatically injects tenancy_ocid, region, current_user_ocid, and the +# auth context. When running from the CLI the user supplies these via +# terraform.tfvars / environment variables / OCI config. +provider "oci" { + tenancy_ocid = var.tenancy_ocid + user_ocid = var.user_ocid + fingerprint = var.fingerprint + private_key_path = var.private_key_path + region = var.region +} + +# IAM (Identity) is a global service whose CREATE/UPDATE/DELETE operations are +# only accepted in the tenancy's home region. When the stack runs in any other +# region, identity writes against the local endpoint fail with: +# 403-NotAllowed, Please go to your home region to execute CREATE... +# We discover the home region (a read, allowed from any region) and pin an +# aliased provider to it; every oci_identity_* resource uses provider = oci.home. +data "oci_identity_region_subscriptions" "this" { + tenancy_id = var.tenancy_ocid +} + +locals { + home_region = one([ + for r in data.oci_identity_region_subscriptions.this.region_subscriptions : + r.region_name if r.is_home_region + ]) +} + +provider "oci" { + alias = "home" + tenancy_ocid = var.tenancy_ocid + user_ocid = var.user_ocid + fingerprint = var.fingerprint + private_key_path = var.private_key_path + region = local.home_region +} diff --git a/hadoop-spark-oci-sample/native/schema.yaml b/hadoop-spark-oci-sample/native/schema.yaml new file mode 100644 index 0000000..ede0e0b --- /dev/null +++ b/hadoop-spark-oci-sample/native/schema.yaml @@ -0,0 +1,537 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +title: "Spark + Hadoop on OCI (Data Flow + Big Data Service)" +description: | + Deploys a configurable Spark + Hadoop platform on OCI: + * Optional VCN with private + public subnets + * Optional OCI Big Data Service (managed Hadoop) cluster + * Optional OCI Data Flow (managed Spark) applications + warm pool + * Object Storage buckets and IAM wiring required for Data Flow +schemaVersion: 1.1.0 +version: "1.0.0" +locale: "en" + +variableGroups: + - title: "Placement" + variables: + - tenancy_ocid + - region + - compartment_ocid + - resource_prefix + + - title: "Hidden CLI-only auth" + visible: false + variables: + - user_ocid + - fingerprint + - private_key_path + + - title: "Networking" + variables: + - create_vcn + - vcn_cidr_block + - vcn_dns_label + - private_subnet_cidr + - public_subnet_cidr + - existing_vcn_id + - existing_private_subnet_id + - existing_public_subnet_id + + - title: "Big Data Service (Hadoop)" + variables: + - deploy_bds + - bds_display_name + - bds_cluster_version + - bds_cluster_profile + - bds_is_high_availability + - bds_is_secure + - bds_cluster_admin_password + - ssh_public_key + - bds_bootstrap_script_url + - bds_oracle_network_cidr + - bds_network_compartment_ocid + + - title: "BDS — Master nodes" + variables: + - bds_master_shape + - bds_master_ocpus + - bds_master_memory_gbs + - bds_master_block_volume_gbs + + - title: "BDS — Utility nodes" + variables: + - bds_utility_shape + - bds_utility_ocpus + - bds_utility_memory_gbs + - bds_utility_block_volume_gbs + + - title: "BDS — Worker nodes" + variables: + - bds_worker_shape + - bds_worker_ocpus + - bds_worker_memory_gbs + - bds_worker_count + - bds_worker_block_volume_gbs + + - title: "BDS — Compute-only worker nodes (optional, for elastic Spark)" + variables: + - bds_compute_only_worker_count + - bds_compute_only_worker_shape + - bds_compute_only_worker_ocpus + - bds_compute_only_worker_memory_gbs + + - title: "IAM / Policies" + variables: + - create_iam_resources + + - title: "Data Flow (Spark)" + variables: + - deploy_dataflow + - dataflow_create_logs_bucket + - dataflow_create_warehouse_bucket + - dataflow_create_scripts_bucket + - dataflow_upload_sample_scripts + + - title: "Data Flow — Warm pool" + variables: + - dataflow_create_pool + - dataflow_pool_shape + - dataflow_pool_ocpus + - dataflow_pool_memory_gbs + - dataflow_pool_min_executors + - dataflow_pool_max_executors + + - title: "Operator VM + Bastion" + variables: + - deploy_operator + - create_bastion + - operator_shape + - operator_ocpus + - operator_memory_gbs + - operator_boot_volume_gbs + - bastion_client_cidr_allow_list + - bastion_max_session_ttl_seconds + + - title: "Tagging" + variables: + - freeform_tags + +variables: + ########################################################################### + # Placement + ########################################################################### + tenancy_ocid: + type: oci:identity:tenancy:id + title: Tenancy + required: true + visible: false + + region: + type: oci:identity:region:name + title: Region + required: true + + compartment_ocid: + type: oci:identity:compartment:id + title: Compartment + required: true + + resource_prefix: + type: string + title: Resource name prefix + description: Prepended to all resources. Lower-case letters, digits, hyphens. + default: bigdata + required: true + pattern: "^[a-z][a-z0-9-]{1,18}$" + + user_ocid: { type: string, visible: false } + fingerprint: { type: string, visible: false } + private_key_path: { type: string, visible: false } + + ########################################################################### + # Networking + ########################################################################### + create_vcn: + type: boolean + title: Create a new VCN + description: If unchecked, you must supply existing VCN + subnet OCIDs. + default: true + required: true + + vcn_cidr_block: + type: string + title: VCN CIDR block + default: "10.0.0.0/16" + visible: create_vcn + + vcn_dns_label: + type: string + title: VCN DNS label + default: bdvcn + visible: create_vcn + + private_subnet_cidr: + type: string + title: Private subnet CIDR + default: "10.0.1.0/24" + visible: create_vcn + + public_subnet_cidr: + type: string + title: Public subnet CIDR + default: "10.0.0.0/24" + visible: create_vcn + + existing_vcn_id: + type: oci:core:vcn:id + title: Existing VCN + description: OCID of the VCN to reuse. + required: true + visible: + not: + - create_vcn + dependsOn: + compartmentId: compartment_ocid + + existing_private_subnet_id: + type: oci:core:subnet:id + title: Existing private subnet + required: true + visible: + not: + - create_vcn + dependsOn: + compartmentId: compartment_ocid + vcnId: existing_vcn_id + hidePublicSubnet: true + + existing_public_subnet_id: + type: oci:core:subnet:id + title: Existing public subnet (optional) + required: false + visible: + not: + - create_vcn + dependsOn: + compartmentId: compartment_ocid + vcnId: existing_vcn_id + hidePrivateSubnet: true + + ########################################################################### + # BDS + ########################################################################### + deploy_bds: + type: boolean + title: Deploy Big Data Service (Hadoop) + default: true + required: true + + bds_oracle_network_cidr: + type: string + title: BDS Oracle-managed network CIDR + description: | + CIDR for the Oracle-managed network BDS creates for the cluster. Must NOT + overlap your VCN/subnet. Default 172.16.0.0/16 is safe for the default + VCN (10.0.0.0/16); change only if your network already uses 172.16.0.0/16. + default: "172.16.0.0/16" + required: false + visible: deploy_bds + + bds_display_name: + type: string + title: Cluster display name + description: Leave empty to use -hadoop. + visible: deploy_bds + + bds_cluster_version: + type: enum + title: Cluster version + enum: + - ODH2_0 + - ODH1 + - ODH0_9 + - CDH6 + - CDH5 + default: ODH2_0 + visible: deploy_bds + + bds_cluster_profile: + type: enum + title: Cluster profile + description: | + HADOOP / HADOOP_EXTENDED ship full Hadoop. SPARK is Spark-only. + HIVE, HBASE, TRINO, KAFKA, DATAFLOW, DATA_SCIENCE, AIRFLOW are specialised. + enum: + - HADOOP + - HADOOP_EXTENDED + - SPARK + - HIVE + - HBASE + - TRINO + - KAFKA + - DATAFLOW + - DATA_SCIENCE + - AIRFLOW + default: HADOOP_EXTENDED + visible: deploy_bds + + bds_is_high_availability: + type: boolean + title: High availability (2 master + 2 utility nodes) + default: false + visible: deploy_bds + + bds_is_secure: + type: boolean + title: Secure cluster (Kerberos + Ranger) + default: false + visible: deploy_bds + + bds_cluster_admin_password: + type: password + title: Cluster admin password + description: Min 8 chars. Used for Ambari / Cloudera Manager. + required: true + visible: deploy_bds + confirmation: true + + ssh_public_key: + type: string + title: SSH public key + description: Used to log in to BDS nodes. + required: true + visible: deploy_bds + + bds_bootstrap_script_url: + type: string + title: Bootstrap script URL (Object Storage) + description: | + Optional. Object Storage URL of a shell script that BDS runs on every + node at cluster creation — the hook for customising core-site.xml, + hdfs-site.xml, yarn-site.xml, spark-defaults.conf, etc. + visible: deploy_bds + + # Master + bds_master_shape: + type: oci:core:instanceshape:name + title: Master shape + default: VM.Standard.E4.Flex + visible: deploy_bds + dependsOn: + compartmentId: compartment_ocid + + bds_master_ocpus: { type: number, title: Master OCPUs, default: 4, minimum: 1, maximum: 64, visible: deploy_bds } + bds_master_memory_gbs: { type: number, title: Master memory (GB), default: 64, minimum: 8, maximum: 1024, visible: deploy_bds } + bds_master_block_volume_gbs: { type: number, title: Master block volume (GB), default: 500, minimum: 150, maximum: 32768, visible: deploy_bds } + + # Utility + bds_utility_shape: + type: oci:core:instanceshape:name + title: Utility shape + default: VM.Standard.E4.Flex + visible: deploy_bds + dependsOn: + compartmentId: compartment_ocid + + bds_utility_ocpus: { type: number, title: Utility OCPUs, default: 4, minimum: 1, maximum: 64, visible: deploy_bds } + bds_utility_memory_gbs: { type: number, title: Utility memory (GB), default: 64, minimum: 8, maximum: 1024, visible: deploy_bds } + bds_utility_block_volume_gbs: { type: number, title: Utility block volume (GB), default: 500, minimum: 150, maximum: 32768, visible: deploy_bds } + + # Worker + bds_worker_shape: + type: oci:core:instanceshape:name + title: Worker shape + default: VM.Standard.E4.Flex + visible: deploy_bds + dependsOn: + compartmentId: compartment_ocid + + bds_worker_ocpus: { type: number, title: Worker OCPUs, default: 8, minimum: 1, maximum: 128, visible: deploy_bds } + bds_worker_memory_gbs: { type: number, title: Worker memory (GB), default: 128, minimum: 16, maximum: 2048, visible: deploy_bds } + bds_worker_count: { type: number, title: Worker count, default: 3, minimum: 3, maximum: 100, visible: deploy_bds } + bds_worker_block_volume_gbs: { type: number, title: Worker block volume (GB), default: 1000, minimum: 150, maximum: 32768, visible: deploy_bds } + + # Compute-only workers + bds_compute_only_worker_count: { type: number, title: Compute-only worker count, default: 0, minimum: 0, maximum: 100, visible: deploy_bds } + bds_compute_only_worker_shape: + type: oci:core:instanceshape:name + title: Compute-only worker shape + default: VM.Standard.E4.Flex + visible: + and: + - deploy_bds + - gt: + - bds_compute_only_worker_count + - 0 + dependsOn: + compartmentId: compartment_ocid + + bds_compute_only_worker_ocpus: { type: number, title: Compute-only worker OCPUs, default: 8, minimum: 1, maximum: 128, visible: deploy_bds } + bds_compute_only_worker_memory_gbs: { type: number, title: Compute-only worker memory (GB), default: 128, minimum: 16, maximum: 2048, visible: deploy_bds } + + ########################################################################### + # Data Flow + ########################################################################### + deploy_dataflow: + type: boolean + title: Deploy Data Flow (Spark) applications + default: true + required: true + + create_iam_resources: + type: boolean + title: Create tenancy-level IAM policies + description: | + Creates the dynamic group + policy for Data Flow and the bdsprod service + policy that lets BDS attach clusters to the subnet/VCN. Requires IAM admin + rights on the tenancy. Untick if you cannot create tenancy-level IAM + resources — you'll need to pre-create them out of band (see README). + default: true + visible: + or: + - deploy_dataflow + - deploy_bds + + bds_network_compartment_ocid: + type: oci:identity:compartment:id + title: BDS network compartment (optional) + description: | + Compartment that holds the VCN/subnet the cluster attaches to. Leave empty + to use the deployment compartment. Set only when reusing an existing + VCN/subnet that lives in a different compartment. + required: false + visible: + and: + - deploy_bds + - create_iam_resources + - not: + - create_vcn + + dataflow_create_logs_bucket: { type: boolean, title: Create logs bucket, default: true, visible: deploy_dataflow } + dataflow_create_warehouse_bucket: { type: boolean, title: Create warehouse bucket, default: true, visible: deploy_dataflow } + dataflow_create_scripts_bucket: { type: boolean, title: Create scripts bucket, default: true, visible: deploy_dataflow } + dataflow_upload_sample_scripts: { type: boolean, title: Upload bundled sample Spark scripts, default: true, visible: deploy_dataflow } + + dataflow_create_pool: + type: boolean + title: Create a Data Flow warm pool + description: Pre-warmed executors → near-zero application start-up latency. + default: false + visible: deploy_dataflow + + dataflow_pool_shape: + type: oci:core:instanceshape:name + title: Pool shape + default: VM.Standard.E4.Flex + visible: + and: + - deploy_dataflow + - dataflow_create_pool + dependsOn: + compartmentId: compartment_ocid + + dataflow_pool_ocpus: { type: number, title: Pool OCPUs, default: 4, minimum: 1, maximum: 64, visible: dataflow_create_pool } + dataflow_pool_memory_gbs: { type: number, title: Pool memory (GB), default: 32, minimum: 8, maximum: 1024, visible: dataflow_create_pool } + dataflow_pool_min_executors: { type: number, title: Pool min executors, default: 1, minimum: 1, maximum: 100, visible: dataflow_create_pool } + dataflow_pool_max_executors: { type: number, title: Pool max executors, default: 4, minimum: 1, maximum: 100, visible: dataflow_create_pool } + + ########################################################################### + # Operator VM + Bastion + ########################################################################### + deploy_operator: + type: boolean + title: Deploy operator VM behind OCI Bastion + description: | + A private jump host with the use-case scripts staged on it and + instance-principal auth, reachable only through the OCI Bastion service. + Open a session, SSH in, and run the use cases from there. + default: false + required: true + + create_bastion: + type: boolean + title: Create an OCI Bastion + description: Untick to reuse an existing bastion that targets the private subnet. + default: true + visible: deploy_operator + + operator_shape: + type: oci:core:instanceshape:name + title: Operator shape + default: VM.Standard.E4.Flex + visible: deploy_operator + dependsOn: + compartmentId: compartment_ocid + + operator_ocpus: { type: number, title: Operator OCPUs, default: 2, minimum: 1, maximum: 64, visible: deploy_operator } + operator_memory_gbs: { type: number, title: Operator memory (GB), default: 16, minimum: 8, maximum: 1024, visible: deploy_operator } + operator_boot_volume_gbs: { type: number, title: Operator boot volume (GB), default: 50, minimum: 50, maximum: 32768, visible: deploy_operator } + + bastion_client_cidr_allow_list: + type: string + title: Bastion client allow-list (/32 only) + description: | + Comma-separated single-host /32 CIDRs allowed to open bastion sessions + (e.g. "203.0.113.4/32" or "203.0.113.4/32,198.51.100.7/32"). No default — + open ranges like 0.0.0.0/0 or /24 are rejected; list your specific client + IP(s). Find yours with: curl ifconfig.me + required: true + pattern: "^(([0-9]{1,3}\\.){3}[0-9]{1,3}/32)(,\\s*([0-9]{1,3}\\.){3}[0-9]{1,3}/32)*$" + visible: + and: + - deploy_operator + - create_bastion + + bastion_max_session_ttl_seconds: + type: number + title: Bastion max session TTL (seconds) + default: 10800 + minimum: 1800 + maximum: 10800 + visible: + and: + - deploy_operator + - create_bastion + + ########################################################################### + # Tagging + ########################################################################### + freeform_tags: + type: oci:identity:tag:value + title: Free-form tags + required: false + +outputs: + bds_cluster_id: + title: BDS cluster OCID + type: link + bds_master_node_ips: + title: BDS master node private IPs + bds_utility_node_ips: + title: BDS utility node private IPs (Ambari / Cloudera Manager / Hue) + dataflow_application_ids: + title: Data Flow application OCIDs + dataflow_pool_id: + title: Data Flow pool OCID + type: link + scripts_bucket_uri: + title: Scripts bucket URI + logs_bucket_name: + title: Logs bucket name + warehouse_bucket_name: + title: Warehouse bucket name + operator_instance_id: + title: Operator VM OCID + type: link + operator_private_ip: + title: Operator VM private IP + bastion_id: + title: Bastion OCID + type: link + bastion_name: + title: Bastion name + operator_bastion_session_hint: + title: Open a bastion session (copy-paste command) + +primaryOutputButton: bds_cluster_id diff --git a/hadoop-spark-oci-sample/native/templates/operator_init.sh.tftpl b/hadoop-spark-oci-sample/native/templates/operator_init.sh.tftpl new file mode 100644 index 0000000..e83262c --- /dev/null +++ b/hadoop-spark-oci-sample/native/templates/operator_init.sh.tftpl @@ -0,0 +1,106 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Operator VM cloud-init. Installs tooling, writes a deployment descriptor the +# use-case scripts use to self-check, and stages the use-case assets from the +# Object Storage scripts bucket via instance-principal auth. +# +# Output: /var/log/operator-init.log +############################################################################### +set -uo pipefail +exec > /var/log/operator-init.log 2>&1 + +echo "operator-init starting at $(date -u)" + +USE_CASES_DIR=/home/opc/use-cases +mkdir -p "$USE_CASES_DIR" + +# --------------------------------------------------------------------------- +# 1. Tooling: jq + OCI CLI, installed SYSTEM-WIDE so both this script and the +# interactive opc login shell can run `oci`. The ol8_ksplice repo is flaky +# on these images and aborts whole dnf transactions, so disable it for our +# installs. A root `pip --user` install would hide under /root/.local and be +# invisible to opc and to the download below — so use a system pip install. +# --------------------------------------------------------------------------- +dnf install -y --disablerepo=ol8_ksplice jq python3 || dnf install -y jq python3 || true + +if ! command -v oci >/dev/null 2>&1; then + dnf install -y --disablerepo=ol8_ksplice python36-oci-cli 2>/dev/null \ + || pip3 install oci-cli || true +fi + +# Ensure the CLI is on PATH for this script (covers /usr/local/bin and, as a +# fallback, a root --user location) and export it for interactive logins too. +export PATH="/usr/local/bin:/root/.local/bin:$PATH" +echo 'export PATH="/usr/local/bin:/root/.local/bin:$PATH"' > /etc/profile.d/oci-cli-path.sh + +export OCI_CLI_AUTH=instance_principal + +if ! command -v oci >/dev/null 2>&1; then + echo "ERROR: oci CLI not found after install; asset download will be skipped." >&2 +fi + +# --------------------------------------------------------------------------- +# 2. Deployment descriptor — what the stack actually deployed. The use-case +# scripts source this to decide whether a use case can run. +# --------------------------------------------------------------------------- +cat > "$USE_CASES_DIR/deployment.env" <<'DEPLOYMENT_ENV' +# Generated by Terraform at operator provisioning. Describes the deployment so +# the use-case scripts can precheck capabilities. Do not edit by hand. +RESOURCE_PREFIX="${resource_prefix}" +COMPARTMENT_OCID="${compartment_ocid}" +REGION="${region}" +OS_NAMESPACE="${namespace}" + +DEPLOY_BDS="${deploy_bds}" +BDS_HIGH_AVAILABILITY="${bds_high_availability}" +BDS_SECURE="${bds_secure}" +DEPLOY_DATAFLOW="${deploy_dataflow}" +DATAFLOW_CREATE_POOL="${dataflow_create_pool}" +DATAFLOW_POOL_ID="${dataflow_pool_id}" + +CREATE_SCRIPTS_BUCKET="${create_scripts_bucket}" +CREATE_LOGS_BUCKET="${create_logs_bucket}" +CREATE_WAREHOUSE_BUCKET="${create_warehouse_bucket}" +SCRIPTS_BUCKET="${scripts_bucket}" +LOGS_BUCKET="${logs_bucket}" +WAREHOUSE_BUCKET="${warehouse_bucket}" +DEPLOYMENT_ENV + +# --------------------------------------------------------------------------- +# 3. Pull the use-case files from the scripts bucket with instance-principal +# auth. Retry to tolerate IAM (dynamic-group/policy) propagation lag right +# after first boot. +# --------------------------------------------------------------------------- +%{ if assets_available ~} +echo "downloading use-case assets from bucket ${scripts_bucket}" +for attempt in $(seq 1 30); do + if oci os object bulk-download \ + --namespace "${namespace}" \ + --bucket-name "${scripts_bucket}" \ + --prefix "use-cases/" \ + --download-dir "$USE_CASES_DIR" \ + --auth instance_principal \ + --overwrite; then + echo "asset download succeeded on attempt $attempt" + break + fi + echo "asset download attempt $attempt failed; sleeping 20s for IAM propagation" + sleep 20 +done +# bulk-download recreates the use-cases/ prefix as a subdir; flatten it. +if [ -d "$USE_CASES_DIR/use-cases" ]; then + cp -rf "$USE_CASES_DIR/use-cases/." "$USE_CASES_DIR/" && rm -rf "$USE_CASES_DIR/use-cases" +fi +%{ else ~} +echo "No scripts bucket was provisioned (Data Flow / scripts bucket disabled)." +echo "Use-case assets were not staged; capability checks still work via deployment.env." +%{ endif ~} + +# Make the run scripts executable and hand everything to opc. +find "$USE_CASES_DIR" -name '*.sh' -exec chmod +x {} \; 2>/dev/null || true +chown -R opc:opc "$USE_CASES_DIR" + +echo "operator-init complete at $(date -u)" diff --git a/hadoop-spark-oci-sample/native/terraform.tfvars.example b/hadoop-spark-oci-sample/native/terraform.tfvars.example new file mode 100644 index 0000000..1b6af88 --- /dev/null +++ b/hadoop-spark-oci-sample/native/terraform.tfvars.example @@ -0,0 +1,137 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Copy this file to terraform.tfvars and edit the values for your tenancy. +# Three showcase profiles are illustrated below — uncomment the one you want. +############################################################################### + +# ---- Auth (CLI only — Resource Manager fills these in automatically) -------- +tenancy_ocid = "ocid1.tenancy.oc1..xxxxxxxx" +user_ocid = "ocid1.user.oc1..xxxxxxxx" +fingerprint = "aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99" +private_key_path = "~/.oci/oci_api_key.pem" +region = "eu-frankfurt-1" +compartment_ocid = "ocid1.compartment.oc1..xxxxxxxx" + +resource_prefix = "bigdata" + +ssh_public_key = "ssh-rsa AAAA... your-key" +bds_cluster_admin_password = "ChangeMe-Strong-Pass-1!" + +############################################################################### +# Profile 1 — MINIMAL: Data Flow only, no Hadoop cluster +# Best for quickly trying Spark on OCI without paying for BDS nodes. +############################################################################### +# deploy_bds = false +# deploy_dataflow = true +# dataflow_create_pool = false +# +# dataflow_applications = [ +# { +# name = "pi-python" +# language = "PYTHON" +# spark_version = "3.5.0" +# num_executors = 2 +# driver_ocpus = 1 +# driver_memory_gbs = 16 +# executor_ocpus = 1 +# executor_memory_gbs = 16 +# configuration = { +# "spark.sql.shuffle.partitions" = "20" +# "spark.dynamicAllocation.enabled" = "true" +# } +# }, +# ] + +############################################################################### +# Profile 2 — STANDARD: small Hadoop cluster + a couple of Spark apps + warm pool +############################################################################### +deploy_bds = true +bds_cluster_version = "ODH2_0" +bds_cluster_profile = "HADOOP_EXTENDED" +bds_is_high_availability = false +bds_worker_count = 3 +bds_worker_ocpus = 8 +bds_worker_memory_gbs = 128 +bds_master_ocpus = 4 +bds_master_memory_gbs = 64 +bds_utility_ocpus = 4 +bds_utility_memory_gbs = 64 + +deploy_dataflow = true +dataflow_create_pool = true +dataflow_pool_min_executors = 1 +dataflow_pool_max_executors = 4 + +dataflow_applications = [ + { + name = "pi-python-small" + language = "PYTHON" + spark_version = "3.5.0" + num_executors = 2 + driver_ocpus = 1 + driver_memory_gbs = 16 + executor_ocpus = 1 + executor_memory_gbs = 16 + configuration = { + "spark.sql.shuffle.partitions" = "20" + } + }, + { + name = "pi-python-tuned" + language = "PYTHON" + spark_version = "3.5.0" + num_executors = 4 + driver_ocpus = 2 + driver_memory_gbs = 32 + executor_ocpus = 4 + executor_memory_gbs = 64 + configuration = { + "spark.sql.shuffle.partitions" = "200" + "spark.sql.adaptive.enabled" = "true" + "spark.sql.adaptive.coalescePartitions.enabled" = "true" + "spark.dynamicAllocation.enabled" = "true" + "spark.dynamicAllocation.minExecutors" = "2" + "spark.dynamicAllocation.maxExecutors" = "10" + "spark.serializer" = "org.apache.spark.serializer.KryoSerializer" + } + }, +] + +############################################################################### +# Profile 3 — PRODUCTION-ISH: HA Hadoop + compute-only workers + secure cluster +############################################################################### +# deploy_bds = true +# bds_cluster_profile = "HADOOP_EXTENDED" +# bds_cluster_version = "ODH2_0" +# bds_is_high_availability = true +# bds_is_secure = true +# bds_master_ocpus = 8 +# bds_master_memory_gbs = 128 +# bds_utility_ocpus = 8 +# bds_utility_memory_gbs = 128 +# bds_worker_count = 5 +# bds_worker_shape = "VM.Standard.E4.Flex" +# bds_worker_ocpus = 16 +# bds_worker_memory_gbs = 256 +# bds_worker_block_volume_gbs = 4000 +# bds_compute_only_worker_count = 3 +# bds_compute_only_worker_ocpus = 16 +# bds_compute_only_worker_memory_gbs = 256 +# bds_bootstrap_script_url = "https://objectstorage..oraclecloud.com/n//b/scripts/o/bootstrap.sh" +# +# deploy_dataflow = true +# dataflow_create_pool = true +# dataflow_pool_min_executors = 4 +# dataflow_pool_max_executors = 16 +# dataflow_pool_ocpus = 8 +# dataflow_pool_memory_gbs = 128 + +############################################################################### +# Reuse existing VCN instead of creating a new one +############################################################################### +# create_vcn = false +# existing_vcn_id = "ocid1.vcn.oc1..xxxxx" +# existing_private_subnet_id = "ocid1.subnet.oc1..xxxxx" +# existing_public_subnet_id = "ocid1.subnet.oc1..xxxxx" diff --git a/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/README.md b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/README.md new file mode 100644 index 0000000..bbccad8 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/README.md @@ -0,0 +1,72 @@ +# Use case 01 — Serverless ETL with Data Flow + +**Goal:** run a real ETL job — read raw CSV from Object Storage, clean and +enrich it, write partitioned Parquet to the warehouse bucket — **without +standing up a single server**. You pay only for the seconds the job runs. + +This is the cheapest, fastest way to get value out of the stack: no Hadoop +cluster, no warm pool. Just Object Storage + Data Flow. + +## Requires in the Resource Manager form + +| Field | Value | +|-------|-------| +| Deploy Data Flow (Spark) applications | **on** | +| Create scripts bucket / warehouse bucket | on (default) | +| Deploy operator VM behind OCI Bastion | on | + +Deploy BDS can be **off** — that's the point of this use case. + +## Run it (on the operator VM) + +Connect to the operator via Bastion (see [../README.md](../README.md)), then: + +```bash +cd use-cases/01-serverless-etl +./run.sh +``` + +`run.sh` self-checks that Data Flow is deployed, then: + +1. Uploads `customers_etl.py` and `sample_customers.csv` to the scripts bucket. +2. Ensures a Data Flow application `-customers-etl` exists (creates it on + first run, pointing at the uploaded script). +3. Submits a run, passing the input CSV and output path as arguments. + +It prints the run OCID and the commands to track it and list the output. + +### Inspect the output + +The job writes Parquet partitioned by `country`: + +```bash +oci os object list -bn "$WAREHOUSE_BUCKET" --prefix customers_clean/ +``` + +(`$WAREHOUSE_BUCKET` is exported from `deployment.env`, which `run.sh` sources.) +Driver logs land in the logs bucket and in **Data Flow → Runs → → Logs**. + +## What the job demonstrates + +`customers_etl.py` is a compact but realistic ETL: + +- Reads CSV with header + schema inference from Object Storage (`oci://` paths). +- Drops rows with null emails, trims/normalizes strings, lower-cases emails. +- Derives a `signup_year` and a `lifetime_value` aggregate per customer. +- Writes **Parquet partitioned by country** to the warehouse bucket — the layout + a downstream engine (Trino, Spark SQL, Autonomous DB external table) consumes. + +Swap in your own CSV (or a glob like `oci://bucket@ns/raw/*.csv`) and the same +application handles it — serverless Spark scales to the data, you don't manage a +cluster. + +## If it can't run + +If Data Flow isn't deployed, `run.sh` stops with: + +``` +This use case can't run on the current deployment. + Data Flow is not deployed. Set 'Deploy Data Flow (Spark) applications' = on. +``` + +Re-apply the stack with that enabled and retry. diff --git a/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/customers_etl.py b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/customers_etl.py new file mode 100644 index 0000000..fb84a12 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/customers_etl.py @@ -0,0 +1,80 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +"""Serverless ETL showcase for OCI Data Flow. + +Reads a raw customers CSV from Object Storage, cleans and enriches it, and writes +Parquet partitioned by country to the warehouse bucket. + +Usage (arguments are passed through `oci data-flow run create --arguments`): + + customers_etl.py + +Both URIs are oci:// paths, e.g. + oci://demo-dataflow-scripts@/sample_customers.csv + oci://demo-dataflow-warehouse@/customers_clean +""" + +import sys + +from pyspark.sql import SparkSession, functions as F + + +def main(): + if len(sys.argv) < 3: + raise SystemExit("usage: customers_etl.py ") + + input_uri, output_uri = sys.argv[1], sys.argv[2] + + spark = SparkSession.builder.appName("customers-etl").getOrCreate() + + raw = ( + spark.read.option("header", "true") + .option("inferSchema", "true") + .csv(input_uri) + ) + + print(f"Read {raw.count()} raw rows from {input_uri}") + + cleaned = ( + raw + # Drop rows we can't key on. + .filter(F.col("email").isNotNull() & (F.trim(F.col("email")) != "")) + # Normalize text fields. + .withColumn("email", F.lower(F.trim(F.col("email")))) + .withColumn("country", F.upper(F.trim(F.col("country")))) + .withColumn("name", F.initcap(F.trim(F.col("name")))) + # Derive a signup_year for downstream cohort analysis. + .withColumn("signup_year", F.year(F.to_date("signup_date"))) + # Coerce spend to a clean numeric. + .withColumn("amount", F.col("amount").cast("double")) + # De-duplicate on the natural key, keep highest spend. + .dropDuplicates(["email"]) + ) + + # Roll spend up to a lifetime_value per customer (here one row each, but the + # same shape works when the source has many orders per customer). + enriched = ( + cleaned.groupBy("email", "name", "country", "signup_year") + .agg( + F.round(F.sum("amount"), 2).alias("lifetime_value"), + F.count(F.lit(1)).alias("order_count"), + ) + ) + + written = enriched.count() + print(f"Writing {written} cleaned customer rows to {output_uri}") + + ( + enriched.repartition("country") + .write.mode("overwrite") + .partitionBy("country") + .parquet(output_uri) + ) + + print("ETL complete.") + spark.stop() + + +if __name__ == "__main__": + main() diff --git a/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/run.sh b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/run.sh new file mode 100755 index 0000000..b0d9c96 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/run.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Use case 01 — Serverless ETL. Run this ON the operator VM. +# +# Self-checks that Data Flow is deployed, uploads the job + sample data to the +# scripts bucket, ensures a Data Flow application exists, and submits a run. +# No arguments needed — everything comes from deployment.env. +############################################################################### +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib.sh +. "$HERE/../lib.sh" + +require_dataflow +require_scripts_bucket +require_warehouse_bucket + +# 1. Stage the job and a sample input in the scripts bucket. +put_script "$HERE/customers_etl.py" "customers_etl.py" +put_script "$HERE/sample_customers.csv" "sample_customers.csv" + +# 2. Ensure the Data Flow application exists (created on first run). +APP_NAME="${RESOURCE_PREFIX}-customers-etl" +FILE_URI="$(os_uri customers_etl.py "$SCRIPTS_BUCKET")" +APP_ID="$(ensure_dataflow_app "$APP_NAME" PYTHON 3.5.0 "$FILE_URI" 2 1 16 2 32)" +_grn "Application: $APP_NAME ($APP_ID)" + +# 3. Submit a run (matching the warm-pool shape if a pool is attached). +INPUT="$(os_uri sample_customers.csv "$SCRIPTS_BUCKET")" +OUTPUT="$(os_uri customers_clean "$WAREHOUSE_BUCKET")" +echo "Submitting run: input=$INPUT output=$OUTPUT" + +build_pool_run_args +RUN_ID="$(oci data-flow run create \ + --compartment-id "$COMPARTMENT_OCID" \ + --application-id "$APP_ID" \ + --display-name "customers-etl-$(date +%s)" \ + --arguments "$INPUT $OUTPUT" \ + ${POOL_RUN_ARGS[@]+"${POOL_RUN_ARGS[@]}"} \ + --query 'data.id' --raw-output)" + +_grn "Run submitted: $RUN_ID" +echo +echo "Track it: oci data-flow run get --run-id $RUN_ID --query 'data.\"lifecycle-state\"'" +echo "Output: oci os object list -bn $WAREHOUSE_BUCKET --prefix customers_clean/" diff --git a/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/sample_customers.csv b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/sample_customers.csv new file mode 100644 index 0000000..80fe753 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/01-serverless-etl/sample_customers.csv @@ -0,0 +1,11 @@ +id,name,email,country,signup_date,amount +1,ada lovelace,ADA@example.com,gb,2021-03-14,1240.50 +2,alan turing, alan@example.com ,GB,2020-06-23,980.00 +3,grace hopper,grace@example.com,us,2022-01-09,2310.75 +4,katherine johnson,KATHERINE@example.com,us,2021-11-30,1500.00 +5,,nobody@example.com,fr,2022-05-05,50.00 +6,linus torvalds,linus@example.com,fi,2023-02-17,640.20 +7,margaret hamilton,margaret@example.com,US,2020-09-01,3025.40 +8,dennis ritchie,,us,2019-12-12,800.00 +9,barbara liskov,barbara@example.com,us,2022-07-19,1875.65 +10,ada lovelace,ada@example.com,GB,2021-03-14,1240.50 diff --git a/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/README.md b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/README.md new file mode 100644 index 0000000..92d026e --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/README.md @@ -0,0 +1,113 @@ +# Use case 02 — Analytics on a managed Hadoop cluster + +**Goal:** run a Spark job with `spark-submit` against YARN on a real Hadoop +cluster — reading from and writing to HDFS. The classic "I want a cluster I can +log into" experience, fully managed by OCI Big Data Service (BDS). + +Use this when you need long-running services (Hive metastore, HBase, Trino), an +interactive cluster, or workloads that don't fit the serverless Data Flow model. + +## Requires in the Resource Manager form + +| Field | Value | +|-------|-------| +| Deploy Big Data Service (Hadoop) | **on** | +| Cluster profile | HADOOP_EXTENDED (default) | +| Deploy operator VM behind OCI Bastion | on | +| SSH public key | your key (for the operator **and** BDS nodes) | + +> BDS provisioning takes **~30 minutes** and the worker nodes bill continuously. +> Destroy the stack when you're done. + +## Run it (from the operator VM) + +`spark-submit` has to run **on a BDS node**, and SSH from the operator to BDS +uses **your** private key (we never put private keys on the operator). So you +must reach the operator with **agent forwarding**. In short: + +```bash +# on your LAPTOP, before connecting: +ssh-add ~/.ssh/id_rsa # load the key into your agent (not just -i !) +ssh-add -l # confirm it's listed + +# connect to the operator WITH -A (see ../README.md §2 for the full command) +ssh -A -o ProxyCommand="..." -i ~/.ssh/id_rsa opc@ + +# on the OPERATOR, confirm the key came across: +ssh-add -l # should list the same key +``` + +The full step-by-step (and why `-i` alone isn't enough) is in +[../README.md](../README.md) §2a–2c. Then: + +```bash +cd use-cases/02-hadoop-cluster-analytics +./submit.sh +``` + +`submit.sh` self-checks that BDS is deployed, resolves the cluster's utility/ +master node private IPs for you, and prints the exact `scp` / `ssh` / +`spark-submit` commands to copy the job over and run it on the cluster. It runs: + +```bash +spark-submit --master yarn --deploy-mode cluster \ + --num-executors 3 --executor-cores 4 --executor-memory 8g \ + /home/opc/sales_report.py \ + hdfs:///user/opc/sales/sales.csv hdfs:///user/opc/sales_report +``` + +Read the result back from HDFS: + +```bash +ssh opc@ 'hdfs dfs -cat /user/opc/sales_report/part-*.csv' +``` + +### Secure (Kerberos) clusters + +The commands above are for a **non-secure** cluster (this use case's intended +shape). If you deployed with **Secure cluster = on** (Kerberos + Ranger — the +[use case 04](../04-secure-ha-production/) shape), HDFS/YARN reject any command +without a Kerberos ticket: + +``` +org.apache.hadoop.security.AccessControlException: + Client cannot authenticate via:[TOKEN, KERBEROS] +``` + +`submit.sh` detects this (`BDS_SECURE=true` in `deployment.env`) and instead +prints a **Kerberos-aware** version of the steps: get a ticket first (`kinit` — +quickest as the `hdfs` superuser using its keytab, or create a principal for your +own user with `kadmin` on the master node), then run the job. Follow what the +script prints. To use the plain flow shown above, redeploy with Secure = off. + +Web UIs (Ambari, Hue, Spark History, YARN RM on port 8088) are served from the +utility node — tunnel to them over SSH from the operator. + +## What the job demonstrates + +`sales_report.py`: + +- Reads a sales CSV from **HDFS** (the on-cluster story). +- Computes revenue by region and product category, plus each category's share of + total revenue using a window function. +- Writes a single coalesced CSV report back to HDFS. + +It runs on YARN in cluster mode, so the driver and executors schedule across your +worker nodes — scale `bds_worker_count` (or add compute-only workers, see +[use case 04](../04-secure-ha-production/)) and the same job spreads wider. + +### Reading Object Storage from the cluster + +BDS reads `oci://` paths directly (HDFS connector + resource principal), so you +can keep data in Object Storage and treat the cluster as pure compute — swap the +`hdfs:///...` arguments for `oci://bucket@namespace/...`. `submit.sh` prints the +scripts-bucket URI to make that easy. + +## If it can't run + +If BDS isn't deployed, `submit.sh` stops with: + +``` +This use case can't run on the current deployment. + Big Data Service is not deployed. Set 'Deploy Big Data Service (Hadoop)' = on. +``` diff --git a/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/sales.csv b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/sales.csv new file mode 100644 index 0000000..62e471b --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/sales.csv @@ -0,0 +1,16 @@ +order_id,region,product_category,amount +1001,EMEA,Hardware,1200.00 +1002,EMEA,Software,450.00 +1003,AMER,Hardware,2300.50 +1004,AMER,Services,980.00 +1005,APAC,Software,610.25 +1006,APAC,Hardware,1750.00 +1007,EMEA,Services,330.00 +1008,AMER,Software,1280.75 +1009,APAC,Services,540.00 +1010,EMEA,Hardware,890.40 +1011,AMER,Hardware,3025.00 +1012,APAC,Software,220.10 +1013,EMEA,Software,775.60 +1014,AMER,Services,1490.00 +1015,APAC,Hardware,1340.80 diff --git a/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/sales_report.py b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/sales_report.py new file mode 100644 index 0000000..106cdec --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/sales_report.py @@ -0,0 +1,68 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +"""Sales report Spark job for an OCI Big Data Service (Hadoop) cluster. + +Runs on YARN via spark-submit. Reads a sales CSV, computes revenue by region and +product category with each category's share of total revenue, and writes a single +CSV report back out. + + spark-submit --master yarn --deploy-mode cluster \\ + sales_report.py + +Paths can be hdfs:///... or oci://bucket@namespace/... — BDS handles both. +""" + +import sys + +from pyspark.sql import SparkSession, Window, functions as F + + +def main(): + if len(sys.argv) < 3: + raise SystemExit("usage: sales_report.py ") + + input_path, output_path = sys.argv[1], sys.argv[2] + + spark = SparkSession.builder.appName("sales-report").getOrCreate() + + sales = ( + spark.read.option("header", "true") + .option("inferSchema", "true") + .csv(input_path) + ) + + by_segment = ( + sales.groupBy("region", "product_category") + .agg( + F.count(F.lit(1)).alias("order_count"), + F.round(F.sum("amount"), 2).alias("total_revenue"), + ) + ) + + # Each segment's share of overall revenue, via a window over everything. + overall = Window.partitionBy() + report = ( + by_segment.withColumn( + "revenue_share_pct", + F.round(100 * F.col("total_revenue") / F.sum("total_revenue").over(overall), 2), + ) + .orderBy(F.col("total_revenue").desc()) + ) + + report.show(truncate=False) + + # Coalesce to one file so the report is easy to read back from HDFS. + ( + report.coalesce(1) + .write.mode("overwrite") + .option("header", "true") + .csv(output_path) + ) + + print(f"Wrote sales report to {output_path}") + spark.stop() + + +if __name__ == "__main__": + main() diff --git a/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/submit.sh b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/submit.sh new file mode 100755 index 0000000..3021fcb --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/02-hadoop-cluster-analytics/submit.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Use case 02 — Hadoop cluster analytics. Run this ON the operator VM. +# +# spark-submit has to run ON a BDS node, and SSH from the operator to BDS needs +# YOUR private key (we never place private keys on the operator). So this script +# self-checks that BDS is deployed, resolves the cluster's node IPs for you, and +# prints the exact steps to copy the job over and run it on the cluster. +############################################################################### +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib.sh +. "$HERE/../lib.sh" + +require_bds + +echo "Resolving the BDS cluster in compartment $COMPARTMENT_OCID ..." +BDS_ID="$(bds_active_cluster_id)" + +if [ -z "$BDS_ID" ]; then + _yel "BDS is enabled but no ACTIVE cluster was found. Current clusters:" + bds_cluster_states + _yel "If a cluster is still CREATING, wait for it to reach ACTIVE and retry." + exit 1 +fi + +UTIL_IP="$(bds_node_ip "$BDS_ID" UTILITY)" +MASTER_IP="$(bds_node_ip "$BDS_ID" MASTER)" + +_grn "Cluster: $BDS_ID" +_grn "Utility node: ${UTIL_IP:-}" +_grn "Master node: ${MASTER_IP:-}" + +TARGET_IP="${UTIL_IP:-$MASTER_IP}" + +if [ "${BDS_SECURE:-false}" = "true" ]; then + cat </dev/null' | head -1) + PRINC=$(sudo klist -kt "$KT" | awk 'NR>3{print $4; exit}') + sudo -u hdfs kinit -kt "$KT" "$PRINC" && sudo -u hdfs klist # confirm the ticket + + sudo -u hdfs hdfs dfs -mkdir -p /user/hdfs/sales + sudo -u hdfs hdfs dfs -put -f /tmp/sales.csv /user/hdfs/sales/ + sudo -u hdfs spark-submit --master yarn --deploy-mode cluster \ + --num-executors 3 --executor-cores 4 --executor-memory 8g \ + /tmp/sales_report.py \ + hdfs:///user/hdfs/sales/sales.csv hdfs:///user/hdfs/sales_report + sudo -u hdfs hdfs dfs -cat /user/hdfs/sales_report/part-*.csv +EOF + cat < The warm pool bills continuously while it's up. Destroy the stack (or set the +> pool option off and re-apply) when you're done iterating. + +## Run it (on the operator VM) + +```bash +cd use-cases/03-warm-pool-low-latency +./run.sh +``` + +`run.sh` self-checks Data Flow (and warns if there's no warm pool), stages +`hourly_aggregate.py` + `events.csv`, ensures a Data Flow application +`-hourly-aggregate` **attached to the warm pool** (when present), and +submits a run. + +**Submit it a few times back to back** and compare the start latency: + +```bash +oci data-flow run list --compartment-id "$COMPARTMENT_OCID" \ + --query 'data[].{name:"display-name",state:"lifecycle-state",created:"time-created"}' +``` + +The first run may still acquire pool capacity; subsequent runs land on hot +executors and start almost immediately. Run [use case 01](../01-serverless-etl/) +(no pool) to feel the cold-start difference. + +## What this demonstrates + +- **Warm pool economics.** Trade a steady baseline cost for fast, predictable + starts. Tune the pool min/max executors to balance cost vs. burst headroom. +- **Same app, many runs.** Data Flow separates the *application* (definition) + from the *run* (execution). Schedule this app from OCI Functions, cron, or + Resource Scheduler and every invocation reuses the hot pool. +- The job (`hourly_aggregate.py`) does a windowed aggregation over an event-log + CSV — counts and unique users per hour per event type — the kind of rollup + you'd refresh frequently behind a dashboard. + +## If it can't run + +If Data Flow isn't deployed, `run.sh` stops and names the field to enable +(`Deploy Data Flow`). If only the warm pool is missing, it runs anyway with a +note to enable **Create a Data Flow warm pool** for fast starts. diff --git a/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/events.csv b/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/events.csv new file mode 100644 index 0000000..3dd086a --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/events.csv @@ -0,0 +1,16 @@ +event_time,user_id,event_type +2026-06-25 09:04:11,u1,view +2026-06-25 09:18:42,u2,view +2026-06-25 09:51:07,u1,click +2026-06-25 09:59:30,u3,view +2026-06-25 10:02:15,u2,click +2026-06-25 10:12:48,u4,view +2026-06-25 10:33:21,u1,purchase +2026-06-25 10:45:59,u3,click +2026-06-25 11:05:02,u5,view +2026-06-25 11:19:37,u2,view +2026-06-25 11:27:14,u4,purchase +2026-06-25 11:48:50,u5,click +2026-06-25 12:01:33,u1,view +2026-06-25 12:22:09,u6,view +2026-06-25 12:39:44,u3,purchase diff --git a/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/hourly_aggregate.py b/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/hourly_aggregate.py new file mode 100644 index 0000000..c2a42fa --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/hourly_aggregate.py @@ -0,0 +1,50 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +"""Hourly rollup job for OCI Data Flow, designed to be re-run frequently on a +warm pool. Reads an event log CSV, aggregates per hour and event type, and writes +a compact Parquet rollup the dashboard layer can read. + + hourly_aggregate.py +""" + +import sys + +from pyspark.sql import SparkSession, functions as F + + +def main(): + if len(sys.argv) < 3: + raise SystemExit("usage: hourly_aggregate.py ") + + input_uri, output_uri = sys.argv[1], sys.argv[2] + + spark = SparkSession.builder.appName("hourly-aggregate").getOrCreate() + + events = ( + spark.read.option("header", "true") + .option("inferSchema", "true") + .csv(input_uri) + ) + + rollup = ( + events + .withColumn("event_hour", F.date_trunc("hour", F.col("event_time"))) + .groupBy("event_hour", "event_type") + .agg( + F.count(F.lit(1)).alias("event_count"), + F.countDistinct("user_id").alias("unique_users"), + ) + .orderBy("event_hour", "event_type") + ) + + rollup.show(truncate=False) + + rollup.write.mode("overwrite").parquet(output_uri) + print(f"Wrote hourly rollup to {output_uri}") + + spark.stop() + + +if __name__ == "__main__": + main() diff --git a/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/run.sh b/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/run.sh new file mode 100755 index 0000000..2236452 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/03-warm-pool-low-latency/run.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Use case 03 — Low-latency repeated jobs. Run this ON the operator VM. +# +# Self-checks Data Flow (and warns if no warm pool), stages the job + seed +# data, ensures a Data Flow application (attached to the warm pool if present), +# and submits a run. Submit it a few times to feel the warm-pool fast starts. +############################################################################### +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib.sh +. "$HERE/../lib.sh" + +require_dataflow +require_scripts_bucket +require_warehouse_bucket +require_warm_pool # warn-only + +put_script "$HERE/hourly_aggregate.py" "hourly_aggregate.py" +put_script "$HERE/events.csv" "events.csv" + +APP_NAME="${RESOURCE_PREFIX}-hourly-aggregate" +FILE_URI="$(os_uri hourly_aggregate.py "$SCRIPTS_BUCKET")" +APP_ID="$(ensure_dataflow_app "$APP_NAME" PYTHON 3.5.0 "$FILE_URI" 2 1 16 2 16)" +_grn "Application: $APP_NAME ($APP_ID)" + +INPUT="$(os_uri events.csv "$SCRIPTS_BUCKET")" +OUTPUT="$(os_uri hourly_rollup "$WAREHOUSE_BUCKET")" + +build_pool_run_args +RUN_ID="$(oci data-flow run create \ + --compartment-id "$COMPARTMENT_OCID" \ + --application-id "$APP_ID" \ + --display-name "hourly-aggregate-$(date +%s)" \ + --arguments "$INPUT $OUTPUT" \ + ${POOL_RUN_ARGS[@]+"${POOL_RUN_ARGS[@]}"} \ + --query 'data.id' --raw-output)" + +_grn "Run submitted: $RUN_ID" +echo +echo "Re-run this script back-to-back and compare start times:" +echo " oci data-flow run list --compartment-id $COMPARTMENT_OCID \\" +echo " --query 'data[].{name:\"display-name\",state:\"lifecycle-state\",created:\"time-created\"}'" diff --git a/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/README.md b/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/README.md new file mode 100644 index 0000000..feb8835 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/README.md @@ -0,0 +1,88 @@ +# Use case 04 — Secure, highly-available production cluster + +**Goal:** the enterprise shape of the stack — a **Kerberized, Ranger-secured, +highly-available** Hadoop cluster with **elastic compute-only workers**, +cluster-wide tuning applied at provisioning via a **bootstrap script**, and a +Data Flow warm pool alongside for serverless jobs. + +The most expensive profile — a reference for the knobs. Size it down for a real +budget. + +## Requires in the Resource Manager form + +| Field | Value | +|-------|-------| +| Deploy Big Data Service (Hadoop) | **on** | +| High availability (2 master + 2 utility) | **on** | +| Secure cluster (Kerberos + Ranger) | **on** | +| Compute-only worker count | e.g. 3 | +| Bootstrap script URL (Object Storage) | URL of `bootstrap.sh` (see below) | +| Deploy operator VM behind OCI Bastion | on | + +> HA and Secure must match (the stack enforces it). BDS provisioning takes ~30 +> min and this is a large, always-billing footprint — destroy it when done. + +## Step 0 — upload the bootstrap script before deploying + +BDS reads the bootstrap script from Object Storage **at cluster-creation time**, +so it must exist before you apply. Upload `bootstrap.sh` to a bucket you control +and put its URL in the **Bootstrap script URL** field: + +```bash +NS=$(oci os ns get --query data --raw-output) +oci os bucket create --compartment-id --name bootstrap-scripts || true +oci os object put -bn bootstrap-scripts --file bootstrap.sh --force +echo "https://objectstorage..oraclecloud.com/n/$NS/b/bootstrap-scripts/o/bootstrap.sh" +``` + +## Verify and use it (from the operator VM) + +Connect to the operator with **agent forwarding** so your key reaches the BDS +nodes — `ssh-add ~/.ssh/id_rsa` on your laptop first, then connect with `-A` +(full steps and the common "Permission denied (publickey)" gotcha are in +[../README.md](../README.md) §2a–2c). Then: + +```bash +cd use-cases/04-secure-ha-production +./check.sh +``` + +`check.sh` confirms BDS is deployed and **warns if the cluster isn't secure/HA** +(so the Kerberos steps make sense), lists the cluster nodes, and prints how to +reach the Kerberized cluster: + +```bash +ssh opc@ +kinit # obtain a Kerberos ticket (secure cluster) +klist +spark-submit --master yarn --deploy-mode cluster \ + --num-executors 8 --executor-cores 4 --executor-memory 16g \ + your_job.py args... +``` + +Confirm the bootstrap tuning landed: + +```bash +ssh opc@ 'grep -A3 "stack bootstrap" /etc/spark3/conf/spark-defaults.conf' +``` + +## What this demonstrates + +| Capability | Form field | Why it matters | +|------------|-----------|----------------| +| **High availability** | High availability → 2+2 masters/utility | No single point of failure for NameNode/ResourceManager/services | +| **Security** | Secure cluster → Kerberos + Ranger | Authenticated users, fine-grained authorization, audit | +| **Elastic compute** | Compute-only worker count | Add Spark/YARN horsepower without growing HDFS | +| **Cluster-wide config** | Bootstrap script URL | Bake in Spark/YARN tuning, packages, keytabs at provisioning | +| **Hybrid serverless** | Data Flow warm pool | Serverless Spark next to the cluster for spiky/ad-hoc jobs | + +## Sizing it down + +Keep High availability + Secure on (they must stay paired), but shrink workers +(3 × 8 OCPU / 128 GB), set compute-only workers to 0, and lower the pool max. + +## If it can't run + +If BDS isn't deployed, `check.sh` stops and names **Deploy Big Data Service**. +If the cluster is deployed but not secure/HA, it runs with warnings naming the +**Secure cluster** / **High availability** fields to enable. diff --git a/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/bootstrap.sh b/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/bootstrap.sh new file mode 100755 index 0000000..63be136 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/bootstrap.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +# Production bootstrap script for use case 04. +# +# Upload this to Object Storage and point var.bds_bootstrap_script_url at it. +# BDS runs it on EVERY node at cluster creation. Use it to apply consistent +# Spark/YARN tuning, install OS packages, deploy keytabs, etc. +# +# Environment provided by BDS: +# $HOSTTYPE — MASTER | UTILITY | WORKER | COMPUTE_ONLY_WORKER +# $CLUSTER_VERSION +# $CLUSTER_PROFILE +# +# All output is captured in /var/logs/oracle/bds/bootstrap.log on the node. + +set -euo pipefail + +echo "Bootstrap starting on $(hostname) — type=${HOSTTYPE:-unknown} version=${CLUSTER_VERSION:-?}" + +# --------------------------------------------------------------------------- +# Spark tuning — applied wherever Spark executors run (workers + compute-only). +# These mirror sensible production defaults: adaptive query execution, Kryo +# serialization, and dynamic allocation so YARN can right-size jobs. +# --------------------------------------------------------------------------- +if [[ "${HOSTTYPE:-}" == "WORKER" || "${HOSTTYPE:-}" == "COMPUTE_ONLY_WORKER" || "${HOSTTYPE:-}" == "MASTER" ]]; then + SPARK_DEFAULTS=/etc/spark3/conf/spark-defaults.conf + if [[ -f "$SPARK_DEFAULTS" ]]; then + if ! grep -q "stack bootstrap" "$SPARK_DEFAULTS"; then + { + echo "" + echo "# --- Added by stack bootstrap (use case 04) ---" + echo "spark.sql.adaptive.enabled true" + echo "spark.sql.adaptive.coalescePartitions.enabled true" + echo "spark.sql.adaptive.skewJoin.enabled true" + echo "spark.serializer org.apache.spark.serializer.KryoSerializer" + echo "spark.shuffle.service.enabled true" + echo "spark.dynamicAllocation.enabled true" + echo "spark.dynamicAllocation.minExecutors 2" + echo "spark.dynamicAllocation.maxExecutors 50" + echo "spark.dynamicAllocation.executorIdleTimeout 120s" + } >> "$SPARK_DEFAULTS" + echo "Applied Spark tuning to $SPARK_DEFAULTS" + fi + fi +fi + +# --------------------------------------------------------------------------- +# Compute-only workers: label them so YARN scheduling/placement policies can +# prefer them for transient Spark executors (they carry no HDFS data). +# --------------------------------------------------------------------------- +if [[ "${HOSTTYPE:-}" == "COMPUTE_ONLY_WORKER" ]]; then + echo "This is an elastic compute-only worker — no DataNode role expected here." +fi + +# --------------------------------------------------------------------------- +# Example hooks you'd typically add in production (left as comments): +# - install monitoring agents / OS packages: yum install -y +# - deploy service keytabs for Kerberos: cp /path/keytab /etc/security/keytabs/ +# - mount additional storage, set ulimits, etc. +# Note: yarn-site.xml / core-site.xml are managed by Ambari/Cloudera Manager on +# secure clusters — change those through the management UI, not by hand here. +# --------------------------------------------------------------------------- + +echo "Bootstrap finished on $(hostname)" diff --git a/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/check.sh b/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/check.sh new file mode 100755 index 0000000..dced8e6 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/04-secure-ha-production/check.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Use case 04 — Secure HA production. Run this ON the operator VM. +# +# Verifies the deployment is the secure, highly-available shape this use case is +# about, resolves the cluster nodes, and prints how to reach the Kerberized +# cluster and run a job. (Like use case 02, spark-submit runs on the cluster and +# needs your SSH key — use 'ssh -A' agent forwarding into the operator.) +############################################################################### +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib.sh +. "$HERE/../lib.sh" + +require_bds_secure_ha + +BDS_ID="$(bds_active_cluster_id)" + +if [ -z "$BDS_ID" ]; then + _yel "BDS is enabled but no ACTIVE cluster found. Current clusters:" + bds_cluster_states + _yel "If a cluster is still CREATING, wait for ACTIVE and retry." + exit 1 +fi + +oci bds instance get --bds-instance-id "$BDS_ID" \ + --query 'data.nodes[].{type:"node-type",ip:"ip-address",state:"lifecycle-state"}' \ + --output table 2>/dev/null || true + +UTIL_IP="$(bds_node_ip "$BDS_ID" UTILITY)" + +cat <} +EOF + +# Quoted heredoc: the Kerberos commands print verbatim (no local expansion). +cat <<'EOF' + # --- on the node: this is a Kerberos cluster, so get a ticket before any job --- + # quickest for a demo — become the hdfs superuser via its keytab: + KT=$(sudo bash -c 'ls /etc/security/keytabs/*hdfs*.keytab 2>/dev/null' | head -1) + PRINC=$(sudo klist -kt "$KT" | awk 'NR>3{print $4; exit}') + sudo -u hdfs kinit -kt "$KT" "$PRINC" && sudo -u hdfs klist # confirm the ticket + + # then submit as hdfs (bring your own job; example): + sudo -u hdfs spark-submit --master yarn --deploy-mode cluster \ + --num-executors 8 --executor-cores 4 --executor-memory 16g \ + your_job.py args... +EOF + +cat < 'addprinc opc', needs the cluster/KDC admin creds), then +'kinit opc'. See the OCI Big Data Service docs on secure clusters. + +The bootstrap.sh in this folder is a DEPLOY-TIME artifact — you upload it to +Object Storage and point 'Bootstrap script URL' at it BEFORE the apply, and BDS +runs it on every node at cluster creation (you never run it yourself). Verify it +landed: + + ssh opc@${UTIL_IP:-} 'grep -A3 "stack bootstrap" /etc/spark3/conf/spark-defaults.conf' +EOF diff --git a/hadoop-spark-oci-sample/native/use-cases/README.md b/hadoop-spark-oci-sample/native/use-cases/README.md new file mode 100644 index 0000000..516c8f7 --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/README.md @@ -0,0 +1,253 @@ +# Use cases + +End-to-end walkthroughs that show what this stack can do. The workflow is: + +1. **Deploy the stack once via OCI Resource Manager**, enabling the + **operator VM** option (and whichever BDS / Data Flow / warm-pool options the + use cases you want need). +2. **Open an OCI Bastion session and SSH into the operator VM.** The use-case + scripts are already staged on it. +3. **Run the scripts.** Each one **self-checks what the stack actually + deployed** and, if a use case can't run on your configuration, tells you + exactly which Resource Manager form field to change — instead of failing with + an opaque error. + +There are no per-use-case `terraform.tfvars` files: with Resource Manager you +configure everything in the deploy form, and the scripts read what was deployed +from a descriptor (`deployment.env`) that Terraform writes onto the operator. + +| # | Use case | Showcases | Needs in the RM form | +|---|----------|-----------|----------------------| +| [01](01-serverless-etl/) | **Serverless ETL** | CSV → cleaned/partitioned Parquet via Data Flow, no cluster | Deploy Data Flow | +| [02](02-hadoop-cluster-analytics/) | **Hadoop cluster analytics** | `spark-submit` on YARN + HDFS on managed BDS | Deploy BDS | +| [03](03-warm-pool-low-latency/) | **Warm-pool low latency** | Repeated jobs that start in seconds | Deploy Data Flow + warm pool | +| [04](04-secure-ha-production/) | **Secure HA production** | Kerberos + Ranger, HA, elastic compute-only workers, bootstrap tuning | Deploy BDS (HA + secure) | + +## 1. Deploy via Resource Manager + +Zip the repo (exclude `.terraform/`, `*.tfstate*`, `.git/`) and create a Stack in +**Developer Services → Resource Manager → Stacks → Create Stack → upload**. The +form is rendered from `schema.yaml`. For the operator workflow, set: + +| Form field | Value | +|------------|-------| +| **Deploy operator VM behind OCI Bastion** | **on** | +| Create an OCI Bastion | on | +| SSH public key | your public key (used to open bastion sessions) | +| Deploy Data Flow / Deploy BDS / warm pool | per the use cases you want (see table) | + +Then **Plan → Apply**. Terraform uploads the `use-cases/` directory to the +scripts bucket; the operator boots, installs tooling, writes a `deployment.env` +descriptor of what was deployed, and **pulls the use-case files from the scripts +bucket** with instance-principal auth. This runs entirely within the apply — the +operator fetches its own files, so it works the same from Resource Manager as +from the CLI. + +> The operator stages its files from the Data Flow **scripts bucket**, so keep +> "Create scripts bucket" on (default). The first pull retries for a few minutes +> to absorb IAM propagation lag right after boot. + +## 2. Connect via OCI Bastion + +The operator has **no public IP**; you reach it only through the managed Bastion. +The connection is a two-hop SSH: your laptop → bastion → operator. + +### 2a. Load your key into the ssh-agent (do this first — it matters) + +Use cases **02 and 04** need to SSH from the operator onward to the BDS nodes, +and that only works via **agent forwarding**. Agent forwarding forwards your +**ssh-agent**, *not* the `-i` key on the command line — so the key must be loaded +into the agent on your laptop **before** you connect: + +```bash +ssh-add ~/.ssh/id_rsa # add your key to the agent +ssh-add -l # verify it's listed (you should see its fingerprint) +``` + +- No agent running? Start one: `eval "$(ssh-agent -s)"`, then `ssh-add`. +- macOS: `ssh-add --apple-use-keychain ~/.ssh/id_rsa`. +- This must be the private key whose public half you set as `ssh_public_key` when + you deployed — the same key the operator **and** the BDS nodes trust. + +> Skipping this is the #1 gotcha. If you connect with `-A` but the agent is empty, +> `-i` still logs you into the *operator* fine, but the operator has no key to +> offer the BDS nodes → `Permission denied (publickey)` at the `scp`/`ssh` step. + +### 2b. Open a bastion session and connect + +Read the ready-made session command from the stack outputs: + +```bash +terraform output -raw operator_bastion_session_hint +``` + +Run the printed `oci bastion session create-managed-ssh ...` (adjust +`--ssh-public-key-file` to your key), wait for `SUCCEEDED`, then connect **with +`-A`** (agent forwarding). Take the `` from the session you created: + +```bash +ssh -A \ + -o ProxyCommand="ssh -i ~/.ssh/id_rsa -W %h:%p -p 22 @host.bastion..oci.oraclecloud.com" \ + -i ~/.ssh/id_rsa opc@ +``` + +- `` = `terraform output -raw operator_private_ip`. +- `` e.g. `eu-frankfurt-1`. + +### 2c. Verify agent forwarding reached the operator + +Once on the operator, confirm the forwarded agent carries your key — this is what +makes the BDS use cases work: + +```bash +ssh-add -l # should list the SAME key as on your laptop +ssh -o BatchMode=yes opc@ hostname # should print the node hostname +``` + +If `ssh-add -l` says *"no identities"* / *"Could not open a connection to your +authentication agent"*, forwarding didn't carry a key — go back to **2a** on your +laptop (`ssh-add`), then reconnect with `-A`. Only Data Flow use cases (01, 03) +work without agent forwarding. + +## 3. Run the use cases + +On the operator, start by seeing what the stack deployed: + +```bash +cd use-cases +cat deployment.env # capability flags + bucket/compartment info +``` + +Each use case has one entry-point script. They all self-check the deployment +first, so a script that can't run on your configuration tells you exactly which +Resource Manager field to flip instead of failing obscurely. + +| # | Use case | Run on the operator | Needs | +|---|----------|---------------------|-------| +| 01 | Serverless ETL | `./01-serverless-etl/run.sh` | Data Flow | +| 02 | Hadoop cluster analytics | `./02-hadoop-cluster-analytics/submit.sh` | BDS | +| 03 | Warm-pool low latency | `./03-warm-pool-low-latency/run.sh` | Data Flow (+ warm pool) | +| 04 | Secure HA production | `./04-secure-ha-production/check.sh` | BDS (HA + secure) | + +```bash +# Data Flow use cases — submit a serverless Spark run end to end: +./01-serverless-etl/run.sh +./03-warm-pool-low-latency/run.sh + +# BDS use cases — resolve the cluster and print the on-node spark-submit steps +# (connect to the operator with `ssh -A` so your key reaches the BDS nodes): +./02-hadoop-cluster-analytics/submit.sh +./04-secure-ha-production/check.sh +``` + +> **01 and 03** drive everything themselves (upload the job, create/reuse the +> Data Flow application, submit the run). **02 and 04** can't run `spark-submit` +> for you — it has to execute on a BDS node — so they verify the cluster, fetch +> its node IPs, and print the exact `scp` / `ssh` / `spark-submit` commands to +> run from the operator. See each use case's own README for details. + +If a use case isn't supported by your deployment, the script says so and names +the form field to flip. For example, running a BDS use case on a Data-Flow-only +stack prints: + +``` +This use case can't run on the current deployment. + Big Data Service is not deployed. Set 'Deploy Big Data Service (Hadoop)' = on. +``` + +## What to expect + +### 01 — Serverless ETL +`run.sh` uploads `customers_etl.py` + `sample_customers.csv`, creates/reuses the +`-customers-etl` application, and submits a run (matched to the warm-pool +shape when a pool exists). It prints the run OCID. + +- The run reaches **`SUCCEEDED`** in ~1–2 min (seconds on a warm pool). Poll with + `oci data-flow run get --run-id --query 'data."lifecycle-state"'`. +- The 10-row sample is cleaned to **8 customer rows** (one null-email row dropped, + one duplicate email de-duped) and written as **Parquet partitioned by country** + (GB/US/FR/FI) to the warehouse bucket: + ``` + oci os object list -bn -dataflow-warehouse --prefix customers_clean/ + # customers_clean/country=GB/part-*.snappy.parquet, .../country=US/... etc. + ``` +- Driver output (`Read 10 raw rows`, `Writing 8 cleaned customer rows`) is in the + logs bucket and in the Data Flow console under the run's **Logs**. + +### 02 — Hadoop cluster analytics +`submit.sh` does **not** run the job (Spark has to run on a BDS node). It: + +- prints the resolved **cluster OCID** and the **utility + master node private + IPs**, then the exact `scp` / `ssh` / `spark-submit` commands to run from the + operator (use `ssh -A`). +- When you run those, `sales_report.py` writes a single **CSV report** to + `hdfs:///user/opc/sales_report` — revenue by region + product category, each + segment's **`revenue_share_pct`**, ordered by revenue. Read it with + `hdfs dfs -cat /user/opc/sales_report/part-*.csv`. + +If the cluster is still provisioning, it prints the cluster list with each +cluster's state instead (wait for `ACTIVE`). + +### 03 — Warm-pool low latency +`run.sh` behaves like 01 but the app runs on the **warm pool**. + +- Submits `-hourly-aggregate`; the run reaches **`SUCCEEDED`**, and on a + warm pool the **start latency is seconds** rather than ~1 min. +- Output is a **Parquet rollup** at `hourly_rollup/` — `event_count` + + `unique_users` per hour per `event_type`. +- Submit it **back-to-back** and compare the `time-created` → start gap across + runs to feel the warm-pool speedup: + ``` + oci data-flow run list --compartment-id \ + --query 'data[].{name:"display-name",state:"lifecycle-state",created:"time-created"}' + ``` + +### 04 — Secure HA production +`check.sh` is a readiness/how-to check for the production shape — it does **not** +submit a job. + +- It confirms the cluster is **secure + HA** (warns, naming the form field, if + not), prints a **table of every node** (type / IP / state), and the steps to + use the Kerberized cluster: `ssh` in, `kinit `, then `spark-submit`. +- It also shows how to confirm the **bootstrap tuning** landed + (`grep "stack bootstrap" /etc/spark3/conf/spark-defaults.conf` on a node). + +### When a run fails +For 01/03, if a run shows `FAILED`, the driver log in the **logs bucket** has the +Spark stack trace: +``` +oci os object list -bn -dataflow-logs --query 'data[].name' --output table +``` + +## Architecture + +``` + you ──ssh──► OCI Bastion ──► Operator VM (private subnet, instance principal) + │ cd use-cases; ./run.sh + ▼ + ┌──────────────────────────────────────────────┐ + │ Data Flow (serverless Spark) ◄── 01, 03 │ + │ Big Data Service (Hadoop) ◄── 02, 04 │ + │ Object Storage (scripts / logs / warehouse) │ + └──────────────────────────────────────────────┘ +``` + +> **Cost reminder.** The operator VM, BDS nodes, and a Data Flow warm pool all +> bill continuously while they exist. `terraform destroy` (or destroy the Stack) +> when you're done. Plain Data Flow runs bill only for the seconds they execute. + +## Tear down + +Running the use cases leaves state Terraform doesn't track — objects in the +buckets (uploaded scripts, run logs, output) and a running warm pool — and a +bucket can't be deleted while non-empty, nor a pool while running. The stack +tries to clean this up automatically on `destroy` (see `cleanup.tf`), but that +relies on the destroy host having OCI CLI auth (not guaranteed on a Resource +Manager runner). The reliable path is to empty things first **from the operator** +(it always has instance-principal auth): + +```bash +./use-cases/cleanup.sh # stops the pool, empties the buckets +``` + +Then run `terraform destroy` / destroy the Stack. diff --git a/hadoop-spark-oci-sample/native/use-cases/cleanup.sh b/hadoop-spark-oci-sample/native/use-cases/cleanup.sh new file mode 100755 index 0000000..e2067fa --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/cleanup.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Pre-destroy cleanup. Run this ON the operator (instance-principal auth) before +# `terraform destroy` to remove the runtime state that would otherwise block the +# teardown: +# * stops the Data Flow warm pool (a running pool can't be deleted) +# * empties the scripts / logs / warehouse buckets (a non-empty bucket can't +# be deleted) +# +# The Terraform stack also attempts this automatically via destroy-time hooks +# (cleanup.tf), but those depend on the destroy host having OCI CLI auth — this +# script is the reliable fallback because the operator always has instance +# principal. +############################################################################### +set -uo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=./lib.sh +. "$HERE/lib.sh" # loads deployment.env + OCI_CLI_AUTH=instance_principal + +require_deployment_env + +# 1. Stop the warm pool, if there is one. +if [ -n "${DATAFLOW_POOL_ID:-}" ]; then + echo "Stopping Data Flow pool $DATAFLOW_POOL_ID ..." + oci data-flow pool stop --pool-id "$DATAFLOW_POOL_ID" --wait-for-state SUCCEEDED || \ + _yel "Pool stop did not complete cleanly — check the console before destroying." +else + echo "No warm pool to stop." +fi + +# 2. Empty the buckets. +for b in "${SCRIPTS_BUCKET:-}" "${LOGS_BUCKET:-}" "${WAREHOUSE_BUCKET:-}"; do + [ -n "$b" ] || continue + echo "Emptying bucket $b ..." + oci os object bulk-delete -bn "$b" --namespace "$OS_NAMESPACE" --force || \ + _yel "Could not fully empty $b — check for in-progress multipart uploads." +done + +_grn "Cleanup complete. You can now run terraform destroy." diff --git a/hadoop-spark-oci-sample/native/use-cases/lib.sh b/hadoop-spark-oci-sample/native/use-cases/lib.sh new file mode 100755 index 0000000..55e48ae --- /dev/null +++ b/hadoop-spark-oci-sample/native/use-cases/lib.sh @@ -0,0 +1,219 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Shared helpers for the use-case scripts. Sourced (not executed) by each +# use case's run.sh / submit.sh. +# +# It loads deployment.env — a descriptor Terraform writes onto the operator VM +# that records what the stack actually deployed — so a use case can refuse to +# run (with a clear message) when its prerequisites aren't present, instead of +# failing with an opaque OCI error. +############################################################################### + +# Locate and source deployment.env. Written to /home/opc/use-cases by the +# operator's cloud-init; also lives next to these scripts when pulled from the +# bucket. Fall back to the directory this library sits in. +_lib_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +for _env in "$_lib_dir/deployment.env" "/home/opc/use-cases/deployment.env"; do + if [ -f "$_env" ]; then + # shellcheck disable=SC1090 + . "$_env" + DEPLOYMENT_ENV_FILE="$_env" + break + fi +done + +# All OCI CLI calls from the operator use instance-principal auth (no keys). +export OCI_CLI_AUTH="${OCI_CLI_AUTH:-instance_principal}" + +_red() { printf '\033[31m%s\033[0m\n' "$*"; } +_yel() { printf '\033[33m%s\033[0m\n' "$*"; } +_grn() { printf '\033[32m%s\033[0m\n' "$*"; } + +# Fail with a message that names the Resource Manager form field to change. +_cannot_run() { + echo + _red "This use case can't run on the current deployment." + echo " $1" + echo + echo "Re-apply the Resource Manager stack with the setting above, then retry." + exit 1 +} + +require_deployment_env() { + if [ -z "${DEPLOYMENT_ENV_FILE:-}" ]; then + _red "deployment.env not found." + echo "Expected it next to these scripts or at /home/opc/use-cases/deployment.env." + echo "It is written by the operator VM's cloud-init — are you running this on the operator?" + exit 1 + fi +} + +require_dataflow() { + require_deployment_env + [ "${DEPLOY_DATAFLOW:-false}" = "true" ] || \ + _cannot_run "Data Flow is not deployed. Set 'Deploy Data Flow (Spark) applications' = on." +} + +require_bds() { + require_deployment_env + [ "${DEPLOY_BDS:-false}" = "true" ] || \ + _cannot_run "Big Data Service is not deployed. Set 'Deploy Big Data Service (Hadoop)' = on." +} + +require_scripts_bucket() { + require_deployment_env + [ -n "${SCRIPTS_BUCKET:-}" ] || \ + _cannot_run "No scripts bucket. Set 'Create scripts bucket' = on (under Data Flow)." +} + +require_warehouse_bucket() { + require_deployment_env + [ -n "${WAREHOUSE_BUCKET:-}" ] || \ + _cannot_run "No warehouse bucket. Set 'Create warehouse bucket' = on (under Data Flow)." +} + +# Use case 04 expects a secure, highly-available cluster. Warn (don't fail) when +# the deployed cluster isn't secure/HA, so the Kerberos steps make sense. +require_bds_secure_ha() { + require_bds + if [ "${BDS_SECURE:-false}" != "true" ]; then + _yel "Note: this cluster is NOT secure (no Kerberos/Ranger). The kinit steps" + _yel "won't apply. Re-apply with 'Secure cluster (Kerberos + Ranger)' = on." + fi + if [ "${BDS_HIGH_AVAILABILITY:-false}" != "true" ]; then + _yel "Note: this cluster is NOT highly available (single master + utility)." + _yel "Re-apply with 'High availability' = on for the production shape." + fi +} + +# Warm pool is an optimization, not a hard requirement — warn, don't fail. +require_warm_pool() { + if [ "${DATAFLOW_CREATE_POOL:-false}" != "true" ] || [ -z "${DATAFLOW_POOL_ID:-}" ]; then + _yel "Note: no Data Flow warm pool is deployed — runs will cold-start (~1 min)." + _yel "To get fast starts, re-apply with 'Create a Data Flow warm pool' = on." + fi +} + +# Upload a local file to the scripts bucket. Usage: put_script [object_name] +put_script() { + local src="$1" name="${2:-$(basename "$1")}" + echo "Uploading $name to oci://$SCRIPTS_BUCKET@$OS_NAMESPACE/$name" + oci os object put \ + --namespace "$OS_NAMESPACE" \ + --bucket-name "$SCRIPTS_BUCKET" \ + --name "$name" \ + --file "$src" \ + --force >/dev/null +} + +# Idempotently ensure a Data Flow application exists, echo its OCID. +# Usage: ensure_dataflow_app \ +# \ +# +ensure_dataflow_app() { + local name="$1" lang="$2" sver="$3" file_uri="$4" + local num_exec="${5:-2}" d_ocpus="${6:-1}" d_mem="${7:-16}" e_ocpus="${8:-2}" e_mem="${9:-16}" + + # Reuse an existing application with this display name if one is already there + # (data-flow application list has no --lifecycle-state flag; --display-name is + # an exact server-side filter). + local existing + existing=$(oci data-flow application list \ + --compartment-id "$COMPARTMENT_OCID" \ + --display-name "$name" \ + --query 'data[0].id' --raw-output 2>/dev/null || true) + + if [ -n "$existing" ] && [ "$existing" != "null" ]; then + echo "$existing" + return 0 + fi + + # Optional extras: warm pool, and logs/warehouse buckets so runs have a place + # to write (mirrors how the Terraform-created applications are configured). + local extra_args=() + [ -n "${DATAFLOW_POOL_ID:-}" ] && extra_args+=(--pool-id "$DATAFLOW_POOL_ID") + [ -n "${LOGS_BUCKET:-}" ] && extra_args+=(--logs-bucket-uri "oci://$LOGS_BUCKET@$OS_NAMESPACE/") + [ -n "${WAREHOUSE_BUCKET:-}" ] && extra_args+=(--warehouse-bucket-uri "oci://$WAREHOUSE_BUCKET@$OS_NAMESPACE/") + + oci data-flow application create \ + --compartment-id "$COMPARTMENT_OCID" \ + --display-name "$name" \ + --language "$lang" \ + --spark-version "$sver" \ + --file-uri "$file_uri" \ + --num-executors "$num_exec" \ + --driver-shape "VM.Standard.E4.Flex" \ + --executor-shape "VM.Standard.E4.Flex" \ + --driver-shape-config "{\"ocpus\": $d_ocpus, \"memoryInGBs\": $d_mem}" \ + --executor-shape-config "{\"ocpus\": $e_ocpus, \"memoryInGBs\": $e_mem}" \ + --type BATCH \ + "${extra_args[@]}" \ + --query 'data.id' --raw-output +} + +# When a run is submitted against a warm pool, its driver/executor shape config +# must MATCH a shape configuration the pool provides — otherwise Data Flow +# rejects the run ("shape configuration not found in pool"). Populate a +# POOL_RUN_ARGS array with the pool's shape so the run create lands on it. Safe +# to call when there is no pool (leaves the array empty). +build_pool_run_args() { + POOL_RUN_ARGS=() + [ -n "${DATAFLOW_POOL_ID:-}" ] || return 0 + + local shape ocpus mem + shape=$(oci data-flow pool get --pool-id "$DATAFLOW_POOL_ID" \ + --query 'data.configurations[0].shape' --raw-output 2>/dev/null || true) + [ -n "$shape" ] && [ "$shape" != "null" ] || return 0 + + ocpus=$(oci data-flow pool get --pool-id "$DATAFLOW_POOL_ID" \ + --query 'data.configurations[0]."shape-config".ocpus' --raw-output 2>/dev/null || true) + mem=$(oci data-flow pool get --pool-id "$DATAFLOW_POOL_ID" \ + --query 'data.configurations[0]."shape-config"."memory-in-gbs"' --raw-output 2>/dev/null || true) + + POOL_RUN_ARGS=(--driver-shape "$shape" --executor-shape "$shape") + if [ -n "$ocpus" ] && [ "$ocpus" != "null" ]; then + POOL_RUN_ARGS+=( + --driver-shape-config "{\"ocpus\": $ocpus, \"memoryInGBs\": $mem}" + --executor-shape-config "{\"ocpus\": $ocpus, \"memoryInGBs\": $mem}" + ) + fi + echo "Matching run to warm-pool shape: $shape (${ocpus} OCPU / ${mem} GB)" +} + +# Echo the OCID of the first ACTIVE BDS cluster in the compartment (empty if +# none). Filters client-side and tolerates either CLI response shape +# (data.items[] or a bare data[] array). +bds_active_cluster_id() { + local id + id=$(oci bds instance list --compartment-id "$COMPARTMENT_OCID" --all \ + --query 'data.items[?"lifecycle-state"==`ACTIVE`].id | [0]' --raw-output 2>/dev/null || true) + if [ -z "$id" ] || [ "$id" = "null" ]; then + id=$(oci bds instance list --compartment-id "$COMPARTMENT_OCID" --all \ + --query 'data[?"lifecycle-state"==`ACTIVE`].id | [0]' --raw-output 2>/dev/null || true) + fi + [ "$id" = "null" ] && id="" + echo "$id" +} + +# Print a table of every BDS cluster and its state (for "not ready yet" hints). +bds_cluster_states() { + oci bds instance list --compartment-id "$COMPARTMENT_OCID" --all \ + --query 'data.items[].{name:"display-name",state:"lifecycle-state"}' --output table 2>/dev/null \ + || oci bds instance list --compartment-id "$COMPARTMENT_OCID" --all \ + --query 'data[].{name:"display-name",state:"lifecycle-state"}' --output table 2>/dev/null || true +} + +# Echo the private IP of the first node of a given type (UTILITY | MASTER). +bds_node_ip() { + local bds_id="$1" ntype="$2" ip + ip=$(oci bds instance get --bds-instance-id "$bds_id" \ + --query "data.nodes[?\"node-type\"==\`$ntype\`].\"ip-address\" | [0]" --raw-output 2>/dev/null || true) + [ "$ip" = "null" ] && ip="" + echo "$ip" +} + +# Convenience: oci:// URI for an object in a given bucket. +os_uri() { echo "oci://$2@$OS_NAMESPACE/$1"; } diff --git a/hadoop-spark-oci-sample/native/variables.tf b/hadoop-spark-oci-sample/native/variables.tf new file mode 100644 index 0000000..f6c5e0a --- /dev/null +++ b/hadoop-spark-oci-sample/native/variables.tf @@ -0,0 +1,548 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Authentication / placement +############################################################################### + +variable "tenancy_ocid" { + description = "OCID of the tenancy. Injected automatically by Resource Manager." + type = string +} + +variable "compartment_ocid" { + description = "OCID of the compartment in which all resources are created." + type = string +} + +variable "region" { + description = "OCI region identifier (e.g. eu-frankfurt-1). Injected automatically by Resource Manager." + type = string +} + +variable "user_ocid" { + description = "User OCID. Only required for the CLI auth flow; leave empty for Resource Manager / instance principal." + type = string + default = "" +} + +variable "fingerprint" { + description = "API key fingerprint. Only required for the CLI auth flow." + type = string + default = "" +} + +variable "private_key_path" { + description = "Path to the API private key file. Only required for the CLI auth flow." + type = string + default = "" +} + +############################################################################### +# Naming / tagging +############################################################################### + +variable "resource_prefix" { + description = "Prefix prepended to every resource display name." + type = string + default = "bigdata" + + validation { + condition = can(regex("^[a-z][a-z0-9-]{1,18}$", var.resource_prefix)) + error_message = "Prefix must start with a letter, be 2-19 chars, lower-case letters/digits/hyphens only." + } +} + +variable "freeform_tags" { + description = "Free-form tags applied to every resource." + type = map(string) + default = { + "stack" = "spark-hadoop-native" + } +} + +############################################################################### +# SSH access +############################################################################### + +variable "ssh_public_key" { + description = "SSH public key used for BDS node access. Required when deploy_bds = true." + type = string + default = "" +} + +############################################################################### +# Network — set create_vcn = false to reuse an existing one +############################################################################### + +variable "create_vcn" { + description = "When true, a new VCN + subnets + gateways are created. When false you must supply existing_*_id variables." + type = bool + default = true +} + +variable "vcn_cidr_block" { + description = "CIDR block used when create_vcn = true." + type = string + default = "10.0.0.0/16" +} + +variable "vcn_dns_label" { + description = "DNS label for the VCN (max 15 chars, alphanumeric)." + type = string + default = "bdvcn" +} + +variable "private_subnet_cidr" { + description = "CIDR for the private subnet that hosts BDS / Data Flow workloads." + type = string + default = "10.0.1.0/24" +} + +variable "public_subnet_cidr" { + description = "CIDR for the public subnet (jump host / bastion / NAT egress reporting)." + type = string + default = "10.0.0.0/24" +} + +variable "existing_vcn_id" { + description = "OCID of an existing VCN to reuse. Required when create_vcn = false." + type = string + default = "" +} + +variable "existing_private_subnet_id" { + description = "OCID of an existing private subnet (BDS + Data Flow). Required when create_vcn = false." + type = string + default = "" +} + +variable "existing_public_subnet_id" { + description = "OCID of an existing public subnet. Optional when create_vcn = false." + type = string + default = "" +} + +############################################################################### +# Big Data Service (Hadoop) cluster +############################################################################### + +variable "deploy_bds" { + description = "Deploy an OCI Big Data Service (Hadoop) cluster." + type = bool + default = true +} + +variable "bds_oracle_network_cidr" { + description = <<-EOT + CIDR block for the Oracle-managed network that BDS provisions for the + cluster. It must NOT overlap the VCN/subnet the cluster attaches to, + otherwise CreateBdsInstance fails with "Provided CIDR block ... overlaps + with subnet CIDR block ...". Defaults to 172.16.0.0/16, which does not + overlap the default VCN (10.0.0.0/16). Change it only if your VCN/subnet + already uses 172.16.0.0/16. + EOT + type = string + default = "172.16.0.0/16" +} + +variable "bds_display_name" { + description = "Display name for the BDS cluster. Defaults to -hadoop when empty." + type = string + default = "" +} + +variable "bds_cluster_version" { + description = "BDS cluster version. API values use underscores (ODH2_0), not dots." + type = string + default = "ODH2_0" + + validation { + condition = contains(["CDH5", "CDH6", "ODH0_9", "ODH1", "ODH2_0"], var.bds_cluster_version) + error_message = "Allowed values: CDH5, CDH6, ODH0_9, ODH1, ODH2_0." + } +} + +variable "bds_cluster_profile" { + description = "BDS cluster profile — controls which Hadoop ecosystem services are installed." + type = string + default = "HADOOP_EXTENDED" + + validation { + condition = contains([ + "HADOOP", "HADOOP_EXTENDED", "HIVE", "SPARK", + "HBASE", "TRINO", "KAFKA", "DATAFLOW", "DATA_SCIENCE", "AIRFLOW" + ], var.bds_cluster_profile) + error_message = "Invalid cluster profile." + } +} + +variable "bds_is_high_availability" { + description = "Whether to deploy 2 master + 2 utility nodes (HA) instead of 1+1." + type = bool + default = false +} + +variable "bds_is_secure" { + description = "Enable Kerberos + Sentry/Ranger on the cluster." + type = bool + default = false +} + +variable "bds_cluster_admin_password" { + description = "Cluster admin password in plaintext (Ambari/Cloudera Manager). Required when deploy_bds = true. Must meet OCI BDS complexity: 8+ chars with at least one uppercase, lowercase, digit, and special character. The module base64-encodes it before sending to the BDS API." + type = string + default = "" + sensitive = true +} + +# Master node +variable "bds_master_shape" { + description = "Shape of the master nodes." + type = string + default = "VM.Standard.E4.Flex" +} +variable "bds_master_ocpus" { + description = "OCPUs per master node (flex shapes only)." + type = number + default = 4 +} +variable "bds_master_memory_gbs" { + description = "Memory per master node in GB (flex shapes only)." + type = number + default = 64 +} +variable "bds_master_block_volume_gbs" { + description = "Block volume size per master node in GB." + type = number + default = 500 +} + +# Utility node +variable "bds_utility_shape" { + description = "Shape of the utility nodes." + type = string + default = "VM.Standard.E4.Flex" +} +variable "bds_utility_ocpus" { + description = "OCPUs per utility node." + type = number + default = 4 +} +variable "bds_utility_memory_gbs" { + description = "Memory per utility node in GB." + type = number + default = 64 +} +variable "bds_utility_block_volume_gbs" { + description = "Block volume size per utility node in GB." + type = number + default = 500 +} + +# Worker nodes +variable "bds_worker_shape" { + description = "Shape of the worker nodes." + type = string + default = "VM.Standard.E4.Flex" +} +variable "bds_worker_ocpus" { + description = "OCPUs per worker node." + type = number + default = 8 +} +variable "bds_worker_memory_gbs" { + description = "Memory per worker node in GB." + type = number + default = 128 +} +variable "bds_worker_count" { + description = "Number of worker nodes (minimum 3)." + type = number + default = 3 + + validation { + condition = var.bds_worker_count >= 3 + error_message = "BDS requires at least 3 worker nodes." + } +} +variable "bds_worker_block_volume_gbs" { + description = "Block volume size per worker node in GB." + type = number + default = 1000 +} + +# Compute-only worker nodes — for Spark workloads that need elastic compute +variable "bds_compute_only_worker_count" { + description = "Number of compute-only worker nodes (no HDFS storage)." + type = number + default = 0 +} +variable "bds_compute_only_worker_shape" { + description = "Shape of compute-only worker nodes." + type = string + default = "VM.Standard.E4.Flex" +} +variable "bds_compute_only_worker_ocpus" { + description = "OCPUs per compute-only worker node." + type = number + default = 8 +} +variable "bds_compute_only_worker_memory_gbs" { + description = "Memory per compute-only worker node in GB." + type = number + default = 128 +} + +variable "bds_bootstrap_script_url" { + description = "Optional Object Storage URL of a bootstrap script that customises Hadoop / Spark configs at cluster creation time." + type = string + default = "" +} + +############################################################################### +# Data Flow (managed Spark) — applications are defined declaratively +############################################################################### + +variable "deploy_dataflow" { + description = "Deploy OCI Data Flow applications + pool." + type = bool + default = true +} + +variable "create_iam_resources" { + description = <<-EOT + When true the stack creates the tenancy-level IAM resources it needs: + + * a dynamic group + policy that let Data Flow runs read/write the + Object Storage buckets in this compartment (when deploy_dataflow), and + * a policy that lets the Big Data Service (bdsprod) service principal + attach clusters to the VCN/subnet (when deploy_bds). Without this BDS + cluster creation fails with "not enough permissions to access subnet + or vcn details". + + Requires the caller to have IAM admin rights on the tenancy. Set to false + if those rights are unavailable and pre-create the dynamic group + policies + out of band (the bundled README has the matching rule and statements). + EOT + type = bool + default = true +} + +variable "bds_network_compartment_ocid" { + description = <<-EOT + Compartment that holds the VCN/subnet the BDS cluster attaches to. The + bdsprod service policy is scoped here. Leave empty to use compartment_ocid + (correct when create_vcn = true, or when an existing network lives in the + same compartment). Set this only when reusing an existing VCN/subnet that + resides in a different compartment. + EOT + type = string + default = "" +} + +variable "dataflow_create_logs_bucket" { + description = "Create an Object Storage bucket for Data Flow logs." + type = bool + default = true +} + +variable "dataflow_create_warehouse_bucket" { + description = "Create an Object Storage bucket for Data Flow warehouse / SQL output." + type = bool + default = true +} + +variable "dataflow_create_scripts_bucket" { + description = "Create an Object Storage bucket for the bundled sample Spark scripts." + type = bool + default = true +} + +variable "dataflow_upload_sample_scripts" { + description = "Upload the sample Spark scripts (examples/) into the scripts bucket." + type = bool + default = true +} + +variable "dataflow_create_pool" { + description = "Create a Data Flow warm pool so applications launch with near-zero start-up latency." + type = bool + default = false +} + +variable "dataflow_pool_min_executors" { + description = "Minimum executor count kept warm in the Data Flow pool." + type = number + default = 1 +} + +variable "dataflow_pool_max_executors" { + description = "Maximum executor count the Data Flow pool can scale to." + type = number + default = 4 +} + +variable "dataflow_pool_shape" { + description = "Shape used by the Data Flow warm pool nodes." + type = string + default = "VM.Standard.E4.Flex" +} + +variable "dataflow_pool_ocpus" { + description = "OCPUs per Data Flow pool node." + type = number + default = 4 +} + +variable "dataflow_pool_memory_gbs" { + description = "Memory per Data Flow pool node in GB." + type = number + default = 32 +} + +# Applications are defined as a list-of-objects so the user can deploy as many +# showcase Spark jobs as they want, each with its own configuration. +variable "dataflow_applications" { + description = <<-EOT + List of Data Flow applications to deploy. Each entry showcases a different + Spark configuration. Set to [] to skip application creation. + + Object schema: + name — display name suffix + language — PYTHON | JAVA | SCALA | SQL + spark_version — e.g. "3.5.0" | "3.2.1" + file_uri — Object Storage URI of the entry-point script/jar. + Leave empty to fall back to the bundled sample for the language. + class_name — required for JAVA/SCALA, otherwise empty + arguments — list of CLI arguments passed to the application + driver_shape — flex shape, e.g. VM.Standard.E4.Flex + driver_ocpus + driver_memory_gbs + executor_shape + executor_ocpus + executor_memory_gbs + num_executors + configuration — map of Spark properties (e.g. spark.sql.shuffle.partitions) + EOT + type = list(object({ + name = string + language = string + spark_version = string + file_uri = optional(string, "") + class_name = optional(string, "") + arguments = optional(list(string), []) + driver_shape = optional(string, "VM.Standard.E4.Flex") + driver_ocpus = optional(number, 1) + driver_memory_gbs = optional(number, 16) + executor_shape = optional(string, "VM.Standard.E4.Flex") + executor_ocpus = optional(number, 1) + executor_memory_gbs = optional(number, 16) + num_executors = optional(number, 2) + configuration = optional(map(string), {}) + })) + default = [ + { + name = "pi-python" + language = "PYTHON" + spark_version = "3.5.0" + num_executors = 2 + driver_ocpus = 1 + driver_memory_gbs = 16 + executor_ocpus = 1 + executor_memory_gbs = 16 + configuration = { + "spark.sql.shuffle.partitions" = "20" + "spark.dynamicAllocation.enabled" = "true" + "spark.dynamicAllocation.minExecutors" = "1" + "spark.dynamicAllocation.maxExecutors" = "4" + # Data Flow validates this as a plain integer in [60, 600] — no "s" suffix. + "spark.dynamicAllocation.executorIdleTimeout" = "60" + } + } + ] + + validation { + condition = alltrue([ + for a in var.dataflow_applications : + contains(["PYTHON", "JAVA", "SCALA", "SQL"], a.language) + ]) + error_message = "Each application's language must be PYTHON, JAVA, SCALA, or SQL." + } +} + +############################################################################### +# Operator VM + OCI Bastion +############################################################################### + +variable "deploy_operator" { + description = <<-EOT + Deploy an operator VM (jump/control host) in the private subnet, reachable + only through the OCI Bastion service. The use-case scripts are staged on it + and it carries instance-principal auth so you can submit Data Flow runs and + use Object Storage without API keys. + EOT + type = bool + default = false +} + +variable "create_bastion" { + description = "Create an OCI Bastion targeting the private subnet. Set false to reuse an existing bastion." + type = bool + default = true +} + +variable "operator_shape" { + description = "Compute shape for the operator VM." + type = string + default = "VM.Standard.E4.Flex" +} + +variable "operator_ocpus" { + description = "OCPUs for the operator VM (flex shapes only)." + type = number + default = 2 +} + +variable "operator_memory_gbs" { + description = "Memory (GB) for the operator VM (flex shapes only)." + type = number + default = 16 +} + +variable "operator_boot_volume_gbs" { + description = "Boot volume size (GB) for the operator VM." + type = number + default = 50 +} + +variable "bastion_client_cidr_allow_list" { + description = <<-EOT + Comma-separated list of single-host /32 CIDRs allowed to initiate bastion + sessions (e.g. "203.0.113.4/32" or "203.0.113.4/32,198.51.100.7/32"). There + is intentionally no default — open ranges like 0.0.0.0/0 are rejected; you + must list the specific client IP(s), each as /32. + EOT + type = string + default = "" + + validation { + condition = alltrue([ + for c in split(",", var.bastion_client_cidr_allow_list) : + can(regex("^([0-9]{1,3}\\.){3}[0-9]{1,3}/32$", trimspace(c))) + if trimspace(c) != "" + ]) + error_message = "Each bastion_client_cidr_allow_list entry must be a single host in /32 form, e.g. 203.0.113.4/32 (open ranges like /24 or 0.0.0.0/0 are not allowed)." + } +} + +variable "bastion_max_session_ttl_seconds" { + description = "Maximum bastion session TTL in seconds (1800–10800)." + type = number + default = 10800 + + validation { + condition = var.bastion_max_session_ttl_seconds >= 1800 && var.bastion_max_session_ttl_seconds <= 10800 + error_message = "Bastion max session TTL must be between 1800 and 10800 seconds." + } +} diff --git a/hadoop-spark-oci-sample/opensource/.gitignore b/hadoop-spark-oci-sample/opensource/.gitignore new file mode 100644 index 0000000..135c110 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/.gitignore @@ -0,0 +1,19 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +# Terraform +.terraform/ +.terraform.lock.hcl +*.tfstate +*.tfstate.* +*.tfvars +!example.tfvars.template +crash.log +override.tf +override.tf.json +*_override.tf +*_override.tf.json + +# Kubernetes / Resource Manager +kubeconfig +*.zip diff --git a/hadoop-spark-oci-sample/opensource/README.md b/hadoop-spark-oci-sample/opensource/README.md new file mode 100644 index 0000000..5a725e7 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/README.md @@ -0,0 +1,178 @@ +# Secure Hadoop & Spark on OKE + +A security-hardened, **one-click** Terraform stack that deploys Apache Hadoop +(HDFS) and Apache Spark on **Oracle Kubernetes Engine (OKE)**. Think of it as an +open-source, self-hosted alternative to Big Data Service / Data Flow: fill in a +form, apply **once**, and you get a locked-down cluster with the platform you +selected installed and **ready for your jobs**. + +Storage is configurable — deploy **HDFS**, **OCI Object Storage**, or **both**. +Spark runs natively on Kubernetes via the Spark Operator (no YARN). What you run +on the platform (your Spark/Hadoop jobs) is yours; this stack delivers the +turn-key, secured infrastructure underneath. + +The OKE cluster + node pool are provisioned through the official +[`terraform-oci-oke`](https://registry.terraform.io/modules/oracle-terraform-modules/oke/oci) +module; the in-cluster platform is provisioned with the `kubernetes` / `helm` +providers in the **same apply**. + +--- + +## One apply, one stack + +Everything — VCN, OKE cluster, node pool, OCI Bastion, IAM/Workload Identity, +the Object Storage bucket, **and** the in-cluster platform (Kerberos KDC, HDFS, +the Spark Operator, RBAC, NetworkPolicies) — lives in a single Terraform state +and deploys in a single `apply`. + +### How the platform gets installed (operator bootstrap) + +The Terraform runner (your laptop, or the Resource Manager job) has **no network +path to the cluster API** — that's by design: the API endpoint is private/locked. +So the platform is **not** installed with the `kubernetes`/`helm` Terraform +providers. Instead: + +1. The `terraform-oci-oke` module creates a small **private operator VM inside + the VCN**, with a private-endpoint kubeconfig and `kubectl`/`helm` installed. + Its instance principal is granted `manage clusters` (→ cluster-admin) by an + IAM policy the module creates. +2. The platform is rendered by Terraform as Kubernetes manifests + a helm + command (`platform.tf`) and handed to the operator as **cloud-init**. +3. On boot, the operator waits for a Ready node, then `kubectl apply`s the + manifests and `helm install`s the Spark Operator — **from inside the VCN**. + +Because the install runs from the operator, the apply never needs to reach the +API. This works in **Resource Manager** with a fully private cluster, no RM +private endpoint required. + +> **Asynchronous:** `apply` finishes once the operator VM exists; the platform +> comes up a few minutes later as cloud-init runs. Verify from the operator (see +> Deploy). Updating the platform later means re-running the bootstrap (changing +> the operator's cloud-init), not a plain `apply` of in-cluster resources. + +--- + +## Storage — configurable + +Two independent toggles (deploy either, or both): + +- **`deploy_hdfs`** — Kerberos-secured HDFS running on the cluster as + StatefulSets with block-volume PVCs, plus an in-cluster MIT KDC. Best for + HDFS-native workloads and data locality. +- **`deploy_object_storage`** — a private OCI Object Storage bucket as the data + lake, reached from Spark via `oci://…` with Workload Identity. Best for + decoupled, elastic storage. + +`deploy_spark` installs the Spark Operator. Users pick per deployment; Spark is +wired to whichever backends are present. + +--- + +## Security model + +| Control | How | +|---|---| +| **No public worker nodes** | Worker nodes run in a private subnet with no public IPs. | +| **Locked API endpoint** | The Kubernetes API endpoint is restricted by NSG to `admin_cidr`. | +| **Bastion-only host access** | Node SSH (and `kubectl` to a private endpoint) go through the managed OCI Bastion, restricted to `admin_cidr`. | +| **No static cloud credentials** | OKE **Workload Identity** (enhanced cluster) gives Spark pods short-lived, scoped OCI tokens for Object Storage — no keys. | +| **Kerberos** | When HDFS is deployed, an in-cluster KDC secures HDFS RPC (privacy), data transfer and web UIs; block-access tokens enabled. | +| **No exposed big-data services** | Every Service is `ClusterIP`/headless — **never** LoadBalancer/NodePort. Spark runs on Kubernetes, **not** YARN and **not** a Spark standalone master, so the classic unauth-RCE vectors (YARN ResourceManager REST; Spark master REST :6066) don't exist. | +| **Egress lockdown** | A default-deny-egress NetworkPolicy + allowlist (DNS, in-cluster/VCN, OCI Service Network :443) blocks the internet egress a compromised pod would use for C2 / exfiltration / crypto-mining. | +| **Pod Security / RBAC** | Namespace Pod Security Admission (`baseline` enforced); Spark uses a tightly-scoped Role, not cluster-admin. | +| **Encryption** | etcd encrypted at rest by OKE; Object Storage encrypted at rest; in-cluster TLS / Kerberos privacy. | +| **Hardened images** | `image_source = ocir` lets you run your own scanned/signed images instead of upstream public ones. | + +`admin_cidr = 0.0.0.0/0` is rejected by a validation rule. + +> ### NetworkPolicy enforcement +> The flannel CNI does **not** enforce NetworkPolicies on its own. The +> ingress/egress policies (rendered in `platform.tf`) are inert until a policy +> engine is installed. Install **Calico in policy-only mode** as a one-time +> follow-up so they take effect (it can be added to the operator's bootstrap +> script). Not bundled by default because the Calico operator + Installation CR +> is a CRD-then-CR pattern that is brittle on a brand-new cluster. + +--- + +## Deploy + +### Resource Manager (one-click) +```bash +zip -r hadoop-spark-oke.zip . -x '.git/*' '.terraform/*' '*.tfstate*' +``` +Console → **Resource Manager → Stacks → Create Stack → My configuration**, +upload the zip, fill in the form (**`admin_cidr` is required**), Plan, Apply. +The operator installs the platform from inside the VCN, so no RM private +endpoint is needed and the cluster can stay fully private. + +### Terraform CLI +```bash +cp example.tfvars.template my.tfvars # edit - admin_cidr is required +terraform init +terraform plan -var-file=my.tfvars +terraform apply -var-file=my.tfvars # one apply does everything +``` + +### After apply (verify the platform) +The platform comes up a few minutes after apply, installed by the operator. To +watch/verify, connect to the operator with the one-command helper (the +`operator_access` output prints it filled in) — it creates the Bastion session, +waits, and SSHes you in: +```bash +./scripts/connect-operator.sh -b -i +``` +Then, on the operator: +```bash +cloud-init status --wait # bootstrap finished +kubectl -n get pods # KDC, NameNode, DataNodes, Spark Operator + +# Smoke-test Spark-on-Kubernetes (spark_smoke_test output): +kubectl -n get configmap spark-examples \ + -o go-template='{{index .data "sparkpi.yaml"}}' | kubectl apply -f - +``` +The operator already has a working kubeconfig (instance principal). To run +kubectl from your own machine instead, open a Bastion port-forward to the +private API endpoint and fetch a kubeconfig with `--kube-endpoint PRIVATE_ENDPOINT`. + +**Prove it works:** ready-to-run demos (one per profile — Spark-only, +HDFS+Spark, Object-Storage+Spark) generate data and print evidence. See +[use-cases/](use-cases/). They are written to the operator at +`/home/opc/use-cases` on first boot: +```bash +cd ~/use-cases && NS= ./01-spark-only/run.sh +``` + +--- + +## Repository layout + +``` +provider.tf terraform / OCI providers (oci + home-region alias) +variables.tf all input variables +locals.tf derived values +data.tf ADs, services, region subscriptions (home region) +network.tf VCN, subnets (incl. private internal-LB subnet), gateways, NSGs +oke.tf OKE cluster + node pool + operator (terraform-oci-oke module) +bastion.tf OCI Bastion service +iam.tf Workload Identity dynamic group + policy (home region) +storage.tf Object Storage bucket +secrets.tf Kerberos passwords (random_password, when deploy_hdfs) +platform.tf in-cluster manifests (KDC/HDFS/RBAC/NetworkPolicies/Spark) + + the operator bootstrap that applies them and writes the demos +scripts/ KDC / keytab / HDFS entrypoint scripts (embedded as ConfigMaps) +use-cases/ runnable demos proving each profile (Spark / HDFS / Object Storage) +outputs.tf cluster OCID, operator access, platform outputs +schema.yaml Resource Manager form +``` + +## Honest status + +The stack is structurally validated (`terraform validate`). The in-cluster +platform — HDFS StatefulSets, the KDC, Kerberos handshakes, the Spark Operator, +and the operator cloud-init bootstrap — is genuinely complex and can only be +fully proven on a live cluster; expect to iterate during the first real +deployment. Before relying on it, validate (a) a clean first `apply` and +`destroy`, (b) that the operator's cloud-init applied the platform +(`cloud-init status` + `kubectl get pods` on the operator), and (c) install +Calico so the NetworkPolicies are actually enforced. diff --git a/hadoop-spark-oci-sample/opensource/bastion.tf b/hadoop-spark-oci-sample/opensource/bastion.tf new file mode 100644 index 0000000..0775f0e --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/bastion.tf @@ -0,0 +1,20 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# OCI Bastion service +# +# Used to reach the worker nodes (break-glass SSH) and, when the Kubernetes API +# endpoint is private, to tunnel kubectl to it. Sessions are ephemeral and may +# only be opened from admin_cidr. +############################################################################### + +resource "oci_bastion_bastion" "this" { + bastion_type = "standard" + compartment_id = var.compartment_ocid + target_subnet_id = oci_core_subnet.nodes.id + client_cidr_block_allow_list = [local.admin_cidr] + name = "${var.cluster_name}-bastion" + max_session_ttl_in_seconds = 10800 + freeform_tags = local.freeform_tags +} diff --git a/hadoop-spark-oci-sample/opensource/data.tf b/hadoop-spark-oci-sample/opensource/data.tf new file mode 100644 index 0000000..e77ff73 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/data.tf @@ -0,0 +1,26 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Data sources +############################################################################### + +data "oci_identity_availability_domains" "ads" { + compartment_id = var.tenancy_ocid +} + +# Region subscriptions - used to find the tenancy home region, where global IAM +# writes (dynamic groups / policies) must be executed. +data "oci_identity_region_subscriptions" "this" { + tenancy_id = var.tenancy_ocid +} + +# "All Services" object - used to route private-subnet egress to the +# Oracle Services Network through the Service Gateway. +data "oci_core_services" "all_services" { + filter { + name = "name" + values = ["All .* Services In Oracle Services Network"] + regex = true + } +} diff --git a/hadoop-spark-oci-sample/opensource/example.tfvars.template b/hadoop-spark-oci-sample/opensource/example.tfvars.template new file mode 100644 index 0000000..ec52dd1 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/example.tfvars.template @@ -0,0 +1,41 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Example variable file for local (CLI) runs. Copy it before editing: +# +# cp example.tfvars.template my.tfvars +# terraform apply -var-file=my.tfvars +# +# In OCI Resource Manager the console form (schema.yaml) collects every value. +############################################################################### + +# Context (Resource Manager fills these automatically). +tenancy_ocid = "ocid1.tenancy.oc1..xxxxxxxx" +compartment_ocid = "ocid1.compartment.oc1..xxxxxxxx" +region = "eu-frankfurt-1" + +# Required. +ssh_public_key = "ssh-rsa AAAA... you@example.com" + +# Required - the ONLY network allowed to reach the API endpoint / Bastion. +# Set this to your own IP. It must not be 0.0.0.0/0. +admin_cidr = "203.0.113.4/32" + +# Cluster. +cluster_name = "bigdata" +# Full version only (vMAJOR.MINOR.PATCH); 'v1.36' won't match a worker image. +# v1.35.2 is the latest GA; v1.36.0 is preview-only (not for production). +# List valid versions: oci ce cluster-options get --cluster-option-id all +kubernetes_version = "v1.35.2" +node_count = 3 + +# Storage backends - deploy either, or both. +deploy_hdfs = true +deploy_object_storage = true + +# Spark. +deploy_spark = true + +# Container images: "upstream" (public Apache images) or "ocir" (your own). +image_source = "upstream" diff --git a/hadoop-spark-oci-sample/opensource/iam.tf b/hadoop-spark-oci-sample/opensource/iam.tf new file mode 100644 index 0000000..b89b6a5 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/iam.tf @@ -0,0 +1,35 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# IAM - OKE Workload Identity for Object Storage access +# +# Created only when deploy_object_storage = true. OKE Workload Identity is NOT +# authorized through dynamic groups - per OCI docs, "You cannot currently use +# workload identities with dynamic groups". Instead, a policy grants `any-user` +# whose request.principal is a workload in THIS cluster + namespace direct access +# to the data bucket. Spark pods then reach Object Storage with short-lived, +# scoped tokens and no static keys. +# +# Ref: https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm +# +# NOTE: creating policies requires IAM administration permission in the tenancy. +# Global IAM writes must run in the home region (provider = oci.home). +############################################################################### + +resource "oci_identity_policy" "workload" { + count = var.deploy_object_storage ? 1 : 0 + provider = oci.home # global IAM writes must run in the home region + + compartment_id = var.compartment_ocid + name = "${var.cluster_name}-workload-policy" + description = "Allow ${var.cluster_name} OKE workloads to use the data bucket via Workload Identity" + + # Scope to workloads in this cluster + namespace (any service account, so both + # the Spark driver and executors are covered). The second statement adds the + # bucket-name condition for object operations (ListObjects/Get/Put/Delete). + statements = [ + "Allow any-user to read buckets in compartment id ${var.compartment_ocid} where all {request.principal.type = 'workload', request.principal.cluster_id = '${module.oke.cluster_id}', request.principal.namespace = '${local.namespace}'}", + "Allow any-user to manage objects in compartment id ${var.compartment_ocid} where all {request.principal.type = 'workload', request.principal.cluster_id = '${module.oke.cluster_id}', request.principal.namespace = '${local.namespace}', target.bucket.name = '${local.bucket_name}'}", + ] +} diff --git a/hadoop-spark-oci-sample/opensource/locals.tf b/hadoop-spark-oci-sample/opensource/locals.tf new file mode 100644 index 0000000..f4c5311 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/locals.tf @@ -0,0 +1,51 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Locals - derived values +############################################################################### + +locals { + ad_name = data.oci_identity_availability_domains.ads.availability_domains[0].name + + # Tolerate stray whitespace from RM forms / copy-paste. + admin_cidr = trimspace(var.admin_cidr) + + # Tenancy home region - global IAM writes must target it. + home_region = one([ + for r in data.oci_identity_region_subscriptions.this.region_subscriptions : + r.region_name if r.is_home_region + ]) + + # ---- Kerberos naming ----------------------------------------------------- + realm = upper(var.kerberos_realm) + domain = lower(var.kerberos_realm) + + # ---- HDFS ---------------------------------------------------------------- + effective_replication = min(var.hdfs_replication, var.hdfs_datanode_count) + + # ---- In-cluster naming / DNS --------------------------------------------- + namespace = var.cluster_name + kdc_host = "kdc.${var.cluster_name}.svc.cluster.local" + namenode_host = "namenode-0.hdfs-nn.${var.cluster_name}.svc.cluster.local" + hdfs_default = "hdfs://${local.namenode_host}:9000" + + # ---- Networking ---------------------------------------------------------- + service_cidr = data.oci_core_services.all_services.services[0]["cidr_block"] + service_id = data.oci_core_services.all_services.services[0]["id"] + + # ---- Object Storage ------------------------------------------------------ + bucket_name = "${var.cluster_name}-data" + os_namespace = data.oci_objectstorage_namespace.this.namespace + + freeform_tags = { + "project" = "hadoop-spark-oke" + "cluster" = var.cluster_name + "managed-by" = "terraform" + } + + common_labels = { + "app.kubernetes.io/part-of" = "hadoop-spark" + "app.kubernetes.io/managed-by" = "terraform" + } +} diff --git a/hadoop-spark-oci-sample/opensource/network.tf b/hadoop-spark-oci-sample/opensource/network.tf new file mode 100644 index 0000000..068f3ab --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/network.tf @@ -0,0 +1,307 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Networking +# +# VCN +# |- Internet Gateway -> Kubernetes API endpoint subnet (when public) +# |- NAT Gateway -> egress for private worker nodes +# |- Service Gateway -> OCI services (OKE control plane, OCIR, Object Storage) +# |- endpoint subnet -> the Kubernetes API endpoint +# |- nodes subnet -> worker nodes (always private, no public IPs) +# +# Worker nodes never have public IPs. The API endpoint is public-but-NSG-locked +# to admin_cidr by default (so Terraform can deploy the workload layer in one +# run); set cluster_endpoint_is_public = false for a fully private endpoint. +############################################################################### + +resource "oci_core_vcn" "this" { + compartment_id = var.compartment_ocid + cidr_blocks = [var.vcn_cidr] + display_name = "${var.cluster_name}-vcn" + dns_label = substr(replace(var.cluster_name, "-", ""), 0, 15) + freeform_tags = local.freeform_tags + + lifecycle { + precondition { + condition = var.deploy_hdfs || var.deploy_object_storage || var.deploy_spark + error_message = "Select at least one component to deploy: deploy_hdfs, deploy_object_storage, and/or deploy_spark." + } + } +} + +resource "oci_core_internet_gateway" "this" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-igw" + enabled = true + freeform_tags = local.freeform_tags +} + +resource "oci_core_nat_gateway" "this" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-natgw" + freeform_tags = local.freeform_tags +} + +resource "oci_core_service_gateway" "this" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-sgw" + freeform_tags = local.freeform_tags + + services { + service_id = local.service_id + } +} + +# -------------------------------------------------------------------------- +# Route tables +# -------------------------------------------------------------------------- +resource "oci_core_route_table" "endpoint" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-endpoint-rt" + freeform_tags = local.freeform_tags + + route_rules { + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + network_entity_id = var.cluster_endpoint_is_public ? oci_core_internet_gateway.this.id : oci_core_nat_gateway.this.id + } + route_rules { + destination = local.service_cidr + destination_type = "SERVICE_CIDR_BLOCK" + network_entity_id = oci_core_service_gateway.this.id + } +} + +resource "oci_core_route_table" "nodes" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-nodes-rt" + freeform_tags = local.freeform_tags + + route_rules { + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + network_entity_id = oci_core_nat_gateway.this.id + } + route_rules { + destination = local.service_cidr + destination_type = "SERVICE_CIDR_BLOCK" + network_entity_id = oci_core_service_gateway.this.id + } +} + +# -------------------------------------------------------------------------- +# Subnets +# -------------------------------------------------------------------------- +resource "oci_core_subnet" "endpoint" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + cidr_block = var.endpoint_subnet_cidr + display_name = "${var.cluster_name}-endpoint-subnet" + dns_label = "endpoint" + route_table_id = oci_core_route_table.endpoint.id + prohibit_public_ip_on_vnic = !var.cluster_endpoint_is_public + freeform_tags = local.freeform_tags +} + +resource "oci_core_subnet" "nodes" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + cidr_block = var.nodes_subnet_cidr + display_name = "${var.cluster_name}-nodes-subnet" + dns_label = "nodes" + route_table_id = oci_core_route_table.nodes.id + prohibit_public_ip_on_vnic = true + freeform_tags = local.freeform_tags +} + +# Dedicated, private subnet for service load balancers. OKE requires a service +# LB subnet on the cluster, and it must be distinct from the node-pool subnet. +# Kept private (no public IPs); this stack creates no LoadBalancer Services. +resource "oci_core_subnet" "int_lb" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + cidr_block = var.lb_subnet_cidr + display_name = "${var.cluster_name}-int-lb-subnet" + dns_label = "intlb" + route_table_id = oci_core_route_table.nodes.id + prohibit_public_ip_on_vnic = true + freeform_tags = local.freeform_tags +} + +# -------------------------------------------------------------------------- +# Network Security Groups +# -------------------------------------------------------------------------- +resource "oci_core_network_security_group" "endpoint" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-endpoint-nsg" + freeform_tags = local.freeform_tags +} + +resource "oci_core_network_security_group" "nodes" { + compartment_id = var.compartment_ocid + vcn_id = oci_core_vcn.this.id + display_name = "${var.cluster_name}-nodes-nsg" + freeform_tags = local.freeform_tags +} + +# ---- API endpoint NSG ---------------------------------------------------- +# kubectl in, from admin_cidr only. +resource "oci_core_network_security_group_security_rule" "endpoint_in_kubectl" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "INGRESS" + protocol = "6" + source = local.admin_cidr + source_type = "CIDR_BLOCK" + description = "kubectl / API access from admin_cidr" + tcp_options { + destination_port_range { + min = 6443 + max = 6443 + } + } +} + +# Worker nodes -> API endpoint. +resource "oci_core_network_security_group_security_rule" "endpoint_in_nodes_6443" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "INGRESS" + protocol = "6" + source = oci_core_network_security_group.nodes.id + source_type = "NETWORK_SECURITY_GROUP" + description = "Worker nodes to Kubernetes API" + tcp_options { + destination_port_range { + min = 6443 + max = 6443 + } + } +} + +resource "oci_core_network_security_group_security_rule" "endpoint_in_nodes_12250" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "INGRESS" + protocol = "6" + source = oci_core_network_security_group.nodes.id + source_type = "NETWORK_SECURITY_GROUP" + description = "Worker nodes control-plane channel" + tcp_options { + destination_port_range { + min = 12250 + max = 12250 + } + } +} + +resource "oci_core_network_security_group_security_rule" "endpoint_in_icmp" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "INGRESS" + protocol = "1" + source = oci_core_network_security_group.nodes.id + source_type = "NETWORK_SECURITY_GROUP" + description = "Path-MTU discovery from worker nodes" + icmp_options { + type = 3 + code = 4 + } +} + +resource "oci_core_network_security_group_security_rule" "endpoint_out_nodes" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "EGRESS" + protocol = "6" + destination = oci_core_network_security_group.nodes.id + destination_type = "NETWORK_SECURITY_GROUP" + description = "API endpoint to worker nodes (kubelet, etc.)" +} + +resource "oci_core_network_security_group_security_rule" "endpoint_out_icmp" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "EGRESS" + protocol = "1" + destination = oci_core_network_security_group.nodes.id + destination_type = "NETWORK_SECURITY_GROUP" + description = "Path-MTU discovery to worker nodes" + icmp_options { + type = 3 + code = 4 + } +} + +resource "oci_core_network_security_group_security_rule" "endpoint_out_osn" { + network_security_group_id = oci_core_network_security_group.endpoint.id + direction = "EGRESS" + protocol = "6" + destination = local.service_cidr + destination_type = "SERVICE_CIDR_BLOCK" + description = "API endpoint to OCI services" + tcp_options { + destination_port_range { + min = 443 + max = 443 + } + } +} + +# ---- Worker nodes NSG ---------------------------------------------------- +resource "oci_core_network_security_group_security_rule" "nodes_in_self" { + network_security_group_id = oci_core_network_security_group.nodes.id + direction = "INGRESS" + protocol = "all" + source = oci_core_network_security_group.nodes.id + source_type = "NETWORK_SECURITY_GROUP" + description = "Node-to-node and pod-to-pod traffic" +} + +resource "oci_core_network_security_group_security_rule" "nodes_in_endpoint" { + network_security_group_id = oci_core_network_security_group.nodes.id + direction = "INGRESS" + protocol = "6" + source = oci_core_network_security_group.endpoint.id + source_type = "NETWORK_SECURITY_GROUP" + description = "API endpoint to worker nodes (kubelet 10250, etc.)" +} + +resource "oci_core_network_security_group_security_rule" "nodes_in_endpoint_icmp" { + network_security_group_id = oci_core_network_security_group.nodes.id + direction = "INGRESS" + protocol = "1" + source = oci_core_network_security_group.endpoint.id + source_type = "NETWORK_SECURITY_GROUP" + description = "Path-MTU discovery from API endpoint" + icmp_options { + type = 3 + code = 4 + } +} + +resource "oci_core_network_security_group_security_rule" "nodes_in_ssh" { + network_security_group_id = oci_core_network_security_group.nodes.id + direction = "INGRESS" + protocol = "6" + source = var.vcn_cidr + source_type = "CIDR_BLOCK" + description = "SSH from inside the VCN (OCI Bastion)" + tcp_options { + destination_port_range { + min = 22 + max = 22 + } + } +} + +resource "oci_core_network_security_group_security_rule" "nodes_out_all" { + network_security_group_id = oci_core_network_security_group.nodes.id + direction = "EGRESS" + protocol = "all" + destination = "0.0.0.0/0" + destination_type = "CIDR_BLOCK" + description = "Worker node egress (NAT for internet, Service Gateway for OCI services)" +} diff --git a/hadoop-spark-oci-sample/opensource/oke.tf b/hadoop-spark-oci-sample/opensource/oke.tf new file mode 100644 index 0000000..9df7433 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/oke.tf @@ -0,0 +1,156 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# OKE - cluster and worker node pool (terraform-oci-oke module) +# +# Provisioned through the official Oracle module instead of raw resources: +# https://registry.terraform.io/modules/oracle-terraform-modules/oke/oci +# +# Scope is deliberately narrow: the module manages ONLY the OKE cluster and the +# worker node pool. It brings NOTHING of its own network or access layer - +# * create_vcn = false -> reuses the VCN/subnets/gateways in network.tf +# * nsgs ... never -> attaches the hand-tuned NSGs from network.tf +# * create_bastion = false -> the OCI Bastion *service* in bastion.tf stays +# * create_operator = false -> no operator host +# * create_iam_* = false -> Workload Identity IAM stays in iam.tf +# so the strict security posture documented in README.md is preserved exactly. +# +# An ENHANCED cluster keeps OKE Workload Identity available; pod networking uses +# the flannel overlay. +############################################################################### + +module "oke" { + source = "oracle-terraform-modules/oke/oci" + version = "5.4.3" + + providers = { + oci = oci + oci.home = oci.home + } + + # ---- Identity / tenancy ------------------------------------------------ + tenancy_id = var.tenancy_ocid + compartment_id = var.compartment_ocid + region = var.region + ssh_public_key = trimspace(var.ssh_public_key) + + # ---- Bring your own network (network.tf) ------------------------------- + create_vcn = false + vcn_id = oci_core_vcn.this.id + + # Map the module's roles onto the existing subnets. Everything we do not use is + # pinned to create = "never" so the module provisions nothing here. + # + # The cluster module REQUIRES a service load balancer subnet (a hard + # precondition), even though this stack never creates LoadBalancer Services. + # int_lb points at a dedicated PRIVATE subnet (network.tf) - it must be + # distinct from the workers subnet, or OKE rejects node placement ("service + # subnets cannot be used by node pools"). preferred_load_balancer = + # "internal" keeps any future service LB private, never public. + # The operator shares the private nodes subnet (a plain VM there is fine, and + # the nodes NSG already lets it reach the API endpoint on 6443). + subnets = { + cp = { create = "never", id = oci_core_subnet.endpoint.id } + workers = { create = "never", id = oci_core_subnet.nodes.id } + int_lb = { create = "never", id = oci_core_subnet.int_lb.id } + operator = { create = "never", id = oci_core_subnet.nodes.id } + pods = { create = "never" } + pub_lb = { create = "never" } + bastion = { create = "never" } + } + + # The module must NOT create or even reference our NSGs through the `nsgs` + # map: its per-NSG `count` is derived from the entry's `id`, and feeding it an + # id that only exists after apply (our network.tf NSGs, created in this same + # run) makes the count unknown at plan time ("Invalid count argument"). So we + # pin every map entry to create = "never" with NO id, and instead attach our + # existing NSGs via the additive control_plane_nsg_ids / worker_nsg_ids lists + # below - lists tolerate apply-time-unknown values, counts do not. + nsgs = { + cp = { create = "never" } + workers = { create = "never" } + pods = { create = "never" } + int_lb = { create = "never" } + pub_lb = { create = "never" } + bastion = { create = "never" } + operator = { create = "never" } + } + + # Attach the hand-tuned NSGs from network.tf to the control-plane endpoint and + # the worker nodes. + control_plane_nsg_ids = [oci_core_network_security_group.endpoint.id] + worker_nsg_ids = [oci_core_network_security_group.nodes.id] + + # Leave the VCN's default security list untouched - access control lives in + # the NSGs (network.tf), matching the original hand-written stack. + lockdown_default_seclist = false + + # ---- Cluster ----------------------------------------------------------- + create_cluster = true + cluster_name = var.cluster_name + cluster_type = "enhanced" + cni_type = "flannel" + kubernetes_version = var.kubernetes_version + control_plane_is_public = var.cluster_endpoint_is_public + assign_public_ip_to_control_plane = var.cluster_endpoint_is_public + pods_cidr = "10.244.0.0/16" + services_cidr = "10.96.0.0/16" + + # Keep any service load balancer private (internal). This stack does not + # create LoadBalancer Services, but the module requires the setting. + preferred_load_balancer = "internal" + + # ---- Access layer ------------------------------------------------------ + # Keep the OCI Bastion *service* (bastion.tf); do not create the module's + # bastion VM. DO create the operator: a private VM inside the VCN that + # installs the in-cluster platform from its cloud-init (platform.tf), so the + # single apply never needs runner-to-API connectivity (works in Resource + # Manager with a fully private cluster). + create_bastion = false + create_operator = true + + operator_nsg_ids = [oci_core_network_security_group.nodes.id] + operator_install_kubectl_from_repo = true + operator_install_helm = true + operator_await_cloudinit = false # module await needs its own bastion; we use the OCI Bastion service + operator_cloud_init = local.operator_cloud_init + + # ---- IAM --------------------------------------------------------------- + # Workload Identity (Spark -> Object Storage) is wired in iam.tf. The module + # creates ONLY the operator's dynamic group + "manage clusters" policy, which + # OKE maps to cluster-admin so the operator can install the platform. All + # other module IAM stays off. Writes go to the home region via oci.home. + # Note: the *_policy toggles are strings ("never"/"auto"/"always"), not bools. + create_iam_resources = true + create_iam_operator_policy = "always" + create_iam_worker_policy = "never" + create_iam_autoscaler_policy = "never" + create_iam_kms_policy = "never" + create_iam_tag_namespace = false + create_iam_defined_tags = false + + # ---- Worker node pool -------------------------------------------------- + worker_pools = { + "${var.cluster_name}-pool" = { + description = "hadoop-spark worker pool" + mode = "node-pool" + size = var.node_count + shape = var.node_shape + ocpus = var.node_ocpus + memory = var.node_memory_gbs + boot_volume_size = var.node_boot_volume_gbs + kubernetes_version = var.kubernetes_version + image_type = "oke" + os = "Oracle Linux" + os_version = "8" + node_labels = { workload = "hadoop-spark" } + } + } + + # ---- Tags -------------------------------------------------------------- + freeform_tags = { + cluster = local.freeform_tags + workers = local.freeform_tags + } +} diff --git a/hadoop-spark-oci-sample/opensource/outputs.tf b/hadoop-spark-oci-sample/opensource/outputs.tf new file mode 100644 index 0000000..220d29f --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/outputs.tf @@ -0,0 +1,98 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Outputs +############################################################################### + +output "cluster_id" { + description = "OCID of the OKE cluster." + value = module.oke.cluster_id +} + +output "cluster_name" { + description = "Name of the OKE cluster." + value = var.cluster_name +} + +output "kubernetes_version" { + description = "Kubernetes version of the cluster." + value = var.kubernetes_version +} + +output "bastion_id" { + description = "OCID of the OCI Bastion (for SSH to nodes / tunnelling kubectl to a private endpoint)." + value = oci_bastion_bastion.this.id +} + +output "object_storage_bucket" { + description = "Object Storage bucket used as the data lake." + value = var.deploy_object_storage ? local.bucket_name : "not deployed" +} + +output "operator_private_ip" { + description = "Private IP of the operator host (reach it via the OCI Bastion to run kubectl)." + value = try(module.oke.operator_private_ip, null) +} + +output "operator_access" { + description = "One command to connect to the operator: creates the Bastion session, waits, and SSHes in (host-key checks disabled for the ephemeral tunnel). Needs the oci CLI and the SSH private key matching ssh_public_key (default ~/.ssh/id_rsa). Run from the stack directory." + value = "./scripts/connect-operator.sh -b ${oci_bastion_bastion.this.id} -i ${try(module.oke.operator_private_ip, "")}" +} + +output "cluster_summary" { + description = "Human-readable summary of what was deployed." + value = join("\n", [ + "Cluster : ${var.cluster_name} (OKE, Kubernetes ${var.kubernetes_version})", + "API endpoint : ${var.cluster_endpoint_is_public ? "public, NSG-locked to ${local.admin_cidr}" : "private (reach via Bastion/operator)"}", + "Worker nodes : ${var.node_count} x ${var.node_shape} (private subnet, no public IPs)", + "Operator : private VM in-VCN; installs the platform via cloud-init", + "Namespace : ${local.namespace}", + "HDFS : ${var.deploy_hdfs ? "enabled (Kerberos realm ${local.realm}, ${var.hdfs_datanode_count} DataNodes)" : "disabled"}", + "Object Storage : ${var.deploy_object_storage ? "enabled (bucket ${local.bucket_name}, Workload Identity)" : "disabled"}", + "Spark : ${var.deploy_spark ? "enabled (Spark Operator, runs on Kubernetes)" : "disabled"}", + "Images : ${var.image_source}", + "Security : private nodes, NSG-locked API, Kerberos (HDFS), Pod Security (baseline), default-deny NetworkPolicies", + "", + "The operator installs the platform asynchronously after apply. Reach the", + "operator via the OCI Bastion, then: kubectl -n ${local.namespace} get pods", + ]) +} + +############################################################################### +# In-cluster platform +############################################################################### + +output "namespace" { + description = "Kubernetes namespace the platform runs in." + value = local.namespace +} + +output "kdc_host" { + description = "In-cluster DNS name of the Kerberos KDC." + value = var.deploy_hdfs ? local.kdc_host : "HDFS not deployed" +} + +output "hdfs_url" { + description = "HDFS default filesystem URL (fs.defaultFS)." + value = var.deploy_hdfs ? local.hdfs_default : "HDFS not deployed" +} + +output "hadoop_user_password" { + description = "Password for the 'hadoop@' Kerberos principal (kinit). Sensitive." + value = try(random_password.hadoop_user[0].result, "HDFS not deployed") + sensitive = true +} + +output "object_storage_path" { + description = "Object Storage path Spark should use." + value = var.deploy_object_storage ? "oci://${local.bucket_name}@${local.os_namespace}/" : "Object Storage not deployed" +} + +output "spark_smoke_test" { + description = "Run the SparkPi example (from the operator host) to confirm Spark-on-Kubernetes works." + value = var.deploy_spark ? join(" ", [ + "kubectl -n ${local.namespace} get configmap spark-examples", + "-o go-template='{{index .data \"sparkpi.yaml\"}}' | kubectl apply -f -", + ]) : "Spark not deployed" +} diff --git a/hadoop-spark-oci-sample/opensource/platform.tf b/hadoop-spark-oci-sample/opensource/platform.tf new file mode 100644 index 0000000..65dee86 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/platform.tf @@ -0,0 +1,582 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# In-cluster platform manifests + operator bootstrap +# +# Single-apply one-click in Resource Manager requires the platform to be +# installed from INSIDE the VCN (the RM/CLI runner cannot reach a private API +# endpoint). So instead of the kubernetes/helm Terraform providers, the platform +# is rendered here as Kubernetes manifests (yamlencode of HCL objects) and a +# helm command, then handed to the module operator's cloud-init (see oke.tf). +# The operator applies them with kubectl/helm using its instance-principal +# kubeconfig - no runner-to-API connectivity needed. +# +# Toggle-driven: HDFS+KDC when deploy_hdfs, Spark Operator when deploy_spark, +# Object Storage wiring when deploy_object_storage. +############################################################################### + +locals { + # ---- Shared Kerberos config --------------------------------------------- + krb5_conf = <<-EOT + [libdefaults] + default_realm = ${local.realm} + dns_lookup_realm = false + dns_lookup_kdc = false + rdns = false + udp_preference_limit = 1 + ticket_lifetime = 24h + renew_lifetime = 7d + forwardable = true + default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 + default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 + permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 + + [realms] + ${local.realm} = { + kdc = ${local.kdc_host} + admin_server = ${local.kdc_host} + } + + [domain_realm] + .svc.cluster.local = ${local.realm} + svc.cluster.local = ${local.realm} + EOT + + kdc_conf = <<-EOT + [kdcdefaults] + kdc_ports = 88 + kdc_tcp_ports = 88 + + [realms] + ${local.realm} = { + acl_file = /var/kerberos/krb5kdc/kadm5.acl + supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal + max_life = 24h 0m 0s + max_renewable_life = 7d 0h 0m 0s + } + EOT + + core_site_xml = <<-EOT + + + fs.defaultFS${local.hdfs_default} + hadoop.security.authenticationkerberos + hadoop.security.authorizationtrue + hadoop.rpc.protectionprivacy + hadoop.security.auth_to_localDEFAULT + + EOT + + hdfs_site_xml = <<-EOT + + + dfs.replication${local.effective_replication} + dfs.namenode.name.dirfile:///hadoop/dfs/name + dfs.datanode.data.dirfile:///hadoop/dfs/data + dfs.namenode.rpc-bind-host0.0.0.0 + dfs.namenode.servicerpc-bind-host0.0.0.0 + dfs.namenode.http-bind-host0.0.0.0 + dfs.namenode.datanode.registration.ip-hostname-checkfalse + dfs.permissions.enabledtrue + dfs.block.access.token.enabletrue + dfs.data.transfer.protectionprivacy + dfs.http.policyHTTP_ONLY + + ignore.secure.ports.for.testingtrue + dfs.datanode.address0.0.0.0:9866 + dfs.datanode.http.address0.0.0.0:9864 + dfs.datanode.ipc.address0.0.0.0:9867 + dfs.namenode.kerberos.principalhdfs/_HOST@${local.realm} + dfs.namenode.keytab.file/keytabs/hdfs.keytab + dfs.namenode.kerberos.internal.spnego.principalHTTP/_HOST@${local.realm} + dfs.datanode.kerberos.principalhdfs/_HOST@${local.realm} + dfs.datanode.keytab.file/keytabs/hdfs.keytab + dfs.web.authentication.kerberos.principalHTTP/_HOST@${local.realm} + dfs.web.authentication.kerberos.keytab/keytabs/hdfs.keytab + + EOT + + # ---- Namespace ---------------------------------------------------------- + m_namespace = { + apiVersion = "v1" + kind = "Namespace" + metadata = { + name = local.namespace + labels = merge(local.common_labels, { + "pod-security.kubernetes.io/enforce" = "baseline" + "pod-security.kubernetes.io/audit" = "restricted" + "pod-security.kubernetes.io/warn" = "restricted" + }) + } + } + + # ---- Spark RBAC --------------------------------------------------------- + m_spark_sa = { + apiVersion = "v1" + kind = "ServiceAccount" + metadata = { name = "spark", namespace = local.namespace, labels = local.common_labels } + } + m_spark_role = { + apiVersion = "rbac.authorization.k8s.io/v1" + kind = "Role" + metadata = { name = "spark-driver", namespace = local.namespace, labels = local.common_labels } + rules = [{ + apiGroups = [""] + resources = ["pods", "services", "configmaps", "persistentvolumeclaims"] + verbs = ["create", "get", "list", "watch", "delete", "deletecollection", "update", "patch"] + }] + } + m_spark_rb = { + apiVersion = "rbac.authorization.k8s.io/v1" + kind = "RoleBinding" + metadata = { name = "spark-driver", namespace = local.namespace, labels = local.common_labels } + roleRef = { apiGroup = "rbac.authorization.k8s.io", kind = "Role", name = "spark-driver" } + subjects = [{ kind = "ServiceAccount", name = "spark", namespace = local.namespace }] + } + + # ---- NetworkPolicies (need a policy engine - see README) ---------------- + m_np_deny_ingress = { + apiVersion = "networking.k8s.io/v1" + kind = "NetworkPolicy" + metadata = { name = "default-deny-ingress", namespace = local.namespace, labels = local.common_labels } + spec = { podSelector = {}, policyTypes = ["Ingress"] } + } + m_np_allow_intra = { + apiVersion = "networking.k8s.io/v1" + kind = "NetworkPolicy" + metadata = { name = "allow-intra-namespace", namespace = local.namespace, labels = local.common_labels } + spec = { + podSelector = {} + policyTypes = ["Ingress"] + ingress = [{ from = [{ podSelector = {} }] }] + } + } + m_np_deny_egress = { + apiVersion = "networking.k8s.io/v1" + kind = "NetworkPolicy" + metadata = { name = "default-deny-egress", namespace = local.namespace, labels = local.common_labels } + spec = { podSelector = {}, policyTypes = ["Egress"] } + } + m_np_allow_egress = { + apiVersion = "networking.k8s.io/v1" + kind = "NetworkPolicy" + metadata = { name = "allow-egress-platform", namespace = local.namespace, labels = local.common_labels } + spec = { + podSelector = {} + policyTypes = ["Egress"] + egress = [ + # Cluster DNS + { ports = [{ port = 53, protocol = "UDP" }, { port = 53, protocol = "TCP" }] }, + # In-cluster / VCN (pods, services, nodes + API endpoint) + { to = [ + { ipBlock = { cidr = "10.244.0.0/16" } }, + { ipBlock = { cidr = "10.96.0.0/16" } }, + { ipBlock = { cidr = var.vcn_cidr } }, + ] }, + # OCI Service Network (Object Storage + Workload Identity) over HTTPS + { to = [{ ipBlock = { cidr = local.service_cidr } }], ports = [{ port = 443, protocol = "TCP" }] }, + ] + } + } + + # ---- KDC ---------------------------------------------------------------- + m_kdc_secret = { + apiVersion = "v1" + kind = "Secret" + type = "Opaque" + metadata = { name = "kerberos-creds", namespace = local.namespace, labels = local.common_labels } + stringData = { + KDC_DB_PASSWORD = try(one(random_password.kdc_db[*].result), "") + KADMIN_PASSWORD = try(one(random_password.kadmin[*].result), "") + HADOOP_USER_PASSWORD = try(one(random_password.hadoop_user[*].result), "") + } + } + m_kdc_config = { + apiVersion = "v1" + kind = "ConfigMap" + metadata = { name = "kdc-config", namespace = local.namespace, labels = local.common_labels } + data = { + "kdc-entrypoint.sh" = file("${path.module}/scripts/kdc-entrypoint.sh") + "keytab-init.sh" = file("${path.module}/scripts/keytab-init.sh") + "kadm5.acl" = "admin/admin@${local.realm} *\n" + "krb5.conf" = local.krb5_conf + "kdc.conf" = local.kdc_conf + } + } + m_kdc_svc = { + apiVersion = "v1" + kind = "Service" + metadata = { name = "kdc", namespace = local.namespace, labels = local.common_labels } + spec = { + clusterIP = "None" + selector = { app = "kdc" } + ports = [ + { name = "kerberos-tcp", port = 88, protocol = "TCP", targetPort = 88 }, + { name = "kerberos-udp", port = 88, protocol = "UDP", targetPort = 88 }, + { name = "kadmin", port = 749, protocol = "TCP", targetPort = 749 }, + ] + } + } + m_kdc_sts = { + apiVersion = "apps/v1" + kind = "StatefulSet" + metadata = { name = "kdc", namespace = local.namespace, labels = merge(local.common_labels, { app = "kdc" }) } + spec = { + replicas = 1 + serviceName = "kdc" + selector = { matchLabels = { app = "kdc" } } + template = { + metadata = { labels = merge(local.common_labels, { app = "kdc" }) } + spec = { + containers = [{ + name = "kdc" + image = var.kdc_image + command = ["/bin/bash", "/kdc-config/kdc-entrypoint.sh"] + env = [ + { name = "REALM", value = local.realm }, + { name = "KDC_DB_PASSWORD", valueFrom = { secretKeyRef = { name = "kerberos-creds", key = "KDC_DB_PASSWORD" } } }, + { name = "KADMIN_PASSWORD", valueFrom = { secretKeyRef = { name = "kerberos-creds", key = "KADMIN_PASSWORD" } } }, + { name = "HADOOP_USER_PASSWORD", valueFrom = { secretKeyRef = { name = "kerberos-creds", key = "HADOOP_USER_PASSWORD" } } }, + ] + ports = [ + { containerPort = 88, protocol = "TCP" }, + { containerPort = 88, protocol = "UDP" }, + { containerPort = 749, protocol = "TCP" }, + ] + volumeMounts = [ + { name = "kdc-config", mountPath = "/kdc-config" }, + { name = "realm-db", mountPath = "/var/kerberos/krb5kdc" }, + ] + securityContext = { runAsUser = 0 } + readinessProbe = { tcpSocket = { port = 88 }, initialDelaySeconds = 20, periodSeconds = 10 } + }] + volumes = [{ name = "kdc-config", configMap = { name = "kdc-config" } }] + } + } + volumeClaimTemplates = [{ + metadata = { name = "realm-db" } + spec = { + accessModes = ["ReadWriteOnce"] + storageClassName = var.storage_class + resources = { requests = { storage = "5Gi" } } + } + }] + } + } + + # ---- Hadoop config + NameNode/DataNode ---------------------------------- + m_hadoop_config = { + apiVersion = "v1" + kind = "ConfigMap" + metadata = { name = "hadoop-config", namespace = local.namespace, labels = local.common_labels } + data = { + "namenode-entrypoint.sh" = file("${path.module}/scripts/namenode-entrypoint.sh") + "datanode-entrypoint.sh" = file("${path.module}/scripts/datanode-entrypoint.sh") + "krb5.conf" = local.krb5_conf + "core-site.xml" = local.core_site_xml + "hdfs-site.xml" = local.hdfs_site_xml + } + } + m_nn_svc = { + apiVersion = "v1" + kind = "Service" + metadata = { name = "hdfs-nn", namespace = local.namespace, labels = local.common_labels } + spec = { + clusterIP = "None" + selector = { app = "namenode" } + ports = [{ name = "rpc", port = 9000 }, { name = "http", port = 9870 }] + } + } + m_nn_sts = { + apiVersion = "apps/v1" + kind = "StatefulSet" + metadata = { name = "namenode", namespace = local.namespace, labels = merge(local.common_labels, { app = "namenode" }) } + spec = { + replicas = 1 + serviceName = "hdfs-nn" + selector = { matchLabels = { app = "namenode" } } + template = { + metadata = { labels = merge(local.common_labels, { app = "namenode" }) } + spec = { + initContainers = [local.keytab_init_container["hdfs-nn"]] + containers = [{ + name = "namenode" + image = var.hadoop_image + command = ["/bin/bash", "/hadoop-config/namenode-entrypoint.sh"] + securityContext = { runAsUser = 0 } + ports = [{ containerPort = 9000 }, { containerPort = 9870 }] + volumeMounts = [ + { name = "hadoop-config", mountPath = "/hadoop-config" }, + { name = "keytabs", mountPath = "/keytabs" }, + { name = "name", mountPath = "/hadoop/dfs/name" }, + ] + readinessProbe = { tcpSocket = { port = 9000 }, initialDelaySeconds = 30, periodSeconds = 15 } + }] + volumes = local.hdfs_pod_volumes + } + } + volumeClaimTemplates = [{ + metadata = { name = "name" } + spec = { + accessModes = ["ReadWriteOnce"] + storageClassName = var.storage_class + resources = { requests = { storage = "${var.hdfs_namenode_storage_gbs}Gi" } } + } + }] + } + } + m_dn_svc = { + apiVersion = "v1" + kind = "Service" + metadata = { name = "hdfs-dn", namespace = local.namespace, labels = local.common_labels } + spec = { + clusterIP = "None" + selector = { app = "datanode" } + ports = [{ name = "data", port = 9866 }, { name = "http", port = 9864 }, { name = "ipc", port = 9867 }] + } + } + m_dn_sts = { + apiVersion = "apps/v1" + kind = "StatefulSet" + metadata = { name = "datanode", namespace = local.namespace, labels = merge(local.common_labels, { app = "datanode" }) } + spec = { + replicas = var.hdfs_datanode_count + serviceName = "hdfs-dn" + selector = { matchLabels = { app = "datanode" } } + template = { + metadata = { labels = merge(local.common_labels, { app = "datanode" }) } + spec = { + initContainers = [local.keytab_init_container["hdfs-dn"]] + containers = [{ + name = "datanode" + image = var.hadoop_image + command = ["/bin/bash", "/hadoop-config/datanode-entrypoint.sh"] + securityContext = { runAsUser = 0 } + ports = [{ containerPort = 9866 }, { containerPort = 9864 }, { containerPort = 9867 }] + volumeMounts = [ + { name = "hadoop-config", mountPath = "/hadoop-config" }, + { name = "keytabs", mountPath = "/keytabs" }, + { name = "data", mountPath = "/hadoop/dfs/data" }, + ] + }] + volumes = local.hdfs_pod_volumes + } + } + volumeClaimTemplates = [{ + metadata = { name = "data" } + spec = { + accessModes = ["ReadWriteOnce"] + storageClassName = var.storage_class + resources = { requests = { storage = "${var.hdfs_datanode_storage_gbs}Gi" } } + } + }] + } + } + + # Keytab init-container per governing service (hdfs-nn for the NameNode, + # hdfs-dn for the DataNodes). POD_SERVICE + POD_NAMESPACE let keytab-init.sh + # build the pod FQDN deterministically, matching HDFS's _HOST resolution. + keytab_init_container = { + for svc in ["hdfs-nn", "hdfs-dn"] : svc => { + name = "keytab-init" + image = var.kdc_image + command = ["/bin/bash", "/kdc-config/keytab-init.sh"] + env = [ + { name = "REALM", value = local.realm }, + { name = "KADMIN_PASSWORD", valueFrom = { secretKeyRef = { name = "kerberos-creds", key = "KADMIN_PASSWORD" } } }, + { name = "POD_NAME", valueFrom = { fieldRef = { fieldPath = "metadata.name" } } }, + { name = "POD_SERVICE", value = svc }, + { name = "POD_NAMESPACE", value = local.namespace }, + ] + securityContext = { runAsUser = 0 } + volumeMounts = [ + { name = "kdc-config", mountPath = "/kdc-config" }, + { name = "keytabs", mountPath = "/keytabs" }, + ] + } + } + hdfs_pod_volumes = [ + { name = "kdc-config", configMap = { name = "kdc-config" } }, + { name = "hadoop-config", configMap = { name = "hadoop-config" } }, + { name = "keytabs", emptyDir = {} }, + ] + + # ---- Spark examples + Object Storage hints ------------------------------ + m_spark_examples = { + apiVersion = "v1" + kind = "ConfigMap" + metadata = { name = "spark-examples", namespace = local.namespace, labels = local.common_labels } + data = { + "sparkpi.yaml" = <<-EOT + apiVersion: sparkoperator.k8s.io/v1beta2 + kind: SparkApplication + metadata: + name: spark-pi + namespace: ${local.namespace} + spec: + type: Scala + mode: cluster + image: ${var.spark_image} + imagePullPolicy: IfNotPresent + mainClass: org.apache.spark.examples.SparkPi + mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-${var.spark_version}.jar" + sparkVersion: "${var.spark_version}" + restartPolicy: + type: Never + driver: + cores: 1 + memory: "1g" + serviceAccount: spark + executor: + cores: 1 + instances: 2 + memory: "1g" + EOT + } + } + m_object_storage = { + apiVersion = "v1" + kind = "ConfigMap" + metadata = { name = "object-storage-config", namespace = local.namespace, labels = local.common_labels } + data = { + "README" = <<-EOT + OCI Object Storage access for Spark + =================================== + Bucket : ${local.bucket_name} + Namespace : ${local.os_namespace} + Path : oci://${local.bucket_name}@${local.os_namespace}/ + + Authentication uses OKE Workload Identity (iam.tf) - the 'spark' service + account is matched by that policy, so no API keys are needed. The Spark + image must include the OCI HDFS connector (oci-hdfs-connector). + + SECURITY: Spark runs on Kubernetes (not YARN, not a standalone master), + so the YARN ResourceManager and Spark master REST RCE vectors do not + exist. Keep the driver UI ClusterIP; do not enable spark.acls + doAs + (CVE-2022-33891). + EOT + } + } + + # ---- Assemble manifests (toggle-gated) ---------------------------------- + # Each object is yamlencode'd to a string here so every list element has a + # uniform type (string) - the manifest objects have different shapes and could + # not be unified inside a conditional/concat otherwise. + manifests_yaml = concat( + [yamlencode(local.m_namespace)], + var.deploy_spark ? [yamlencode(local.m_spark_sa), yamlencode(local.m_spark_role), yamlencode(local.m_spark_rb)] : [], + [yamlencode(local.m_np_deny_ingress), yamlencode(local.m_np_allow_intra), yamlencode(local.m_np_deny_egress), yamlencode(local.m_np_allow_egress)], + var.deploy_hdfs ? [ + yamlencode(local.m_kdc_secret), yamlencode(local.m_kdc_config), yamlencode(local.m_kdc_svc), yamlencode(local.m_kdc_sts), + yamlencode(local.m_hadoop_config), yamlencode(local.m_nn_svc), yamlencode(local.m_nn_sts), yamlencode(local.m_dn_svc), yamlencode(local.m_dn_sts), + ] : [], + var.deploy_spark ? [yamlencode(local.m_spark_examples)] : [], + var.deploy_object_storage ? [yamlencode(local.m_object_storage)] : [], + ) + + platform_yaml = join("\n---\n", local.manifests_yaml) + + # ---- Spark Operator via helm (run on the operator) ---------------------- + spark_helm = var.deploy_spark ? join(" ", [ + "helm repo add spark-operator https://kubeflow.github.io/spark-operator;", + "helm repo update;", + "helm upgrade --install spark-operator spark-operator/spark-operator", + "--version ${var.spark_operator_chart_version}", + "--namespace ${local.namespace}", + "--set sparkJobNamespace=${local.namespace}", + "--set webhook.enable=true", + "--set service.type=ClusterIP", + "--set webhook.serviceType=ClusterIP", + "--set uiService.enable=false", + ]) : "echo 'Spark not enabled; skipping spark-operator'" + + # ---- Use-case demos written to /home/opc/use-cases ---------------------- + # Each repo file -> a base64'd write command, run by the bootstrap script (as + # root, in the late scripts-user stage AFTER the opc user exists), so the demos + # land in opc's home with no SSH key and no inbound reachability needed. + use_case_manifest = { + "/home/opc/use-cases/README.md" = "use-cases/README.md" + "/home/opc/use-cases/lib/common.sh" = "use-cases/lib/common.sh" + "/home/opc/use-cases/01-spark-only/run.sh" = "use-cases/01-spark-only/run.sh" + "/home/opc/use-cases/01-spark-only/job.py" = "use-cases/01-spark-only/job.py" + "/home/opc/use-cases/02-hdfs-spark/run.sh" = "use-cases/02-hdfs-spark/run.sh" + "/home/opc/use-cases/02-hdfs-spark/job.py" = "use-cases/02-hdfs-spark/job.py" + "/home/opc/use-cases/03-objstore-spark/run.sh" = "use-cases/03-objstore-spark/run.sh" + "/home/opc/use-cases/03-objstore-spark/job.py" = "use-cases/03-objstore-spark/job.py" + } + use_case_writes = join("\n", [ + for dest, src in local.use_case_manifest : + "mkdir -p '${dirname(dest)}'; echo '${base64encode(file("${path.module}/${src}"))}' | base64 -d > '${dest}'" + ]) + + # ---- Operator bootstrap cloud-init -------------------------------------- + operator_bootstrap = <<-EOT + #!/bin/bash + set -uo pipefail + export OCI_CLI_AUTH=instance_principal + export KUBECONFIG=/home/opc/.kube/config + log() { echo "[platform-bootstrap] $*"; } + + log "installing use-case demos to /home/opc/use-cases" + ${local.use_case_writes} + chmod +x /home/opc/use-cases/lib/common.sh /home/opc/use-cases/*/run.sh + chown -R opc:opc /home/opc/use-cases + + # Make instance-principal auth the default in every interactive shell, so the + # kubeconfig exec plugin (oci ce cluster generate-token) never falls back to + # looking for ~/.oci/config and hanging in a fresh session. + grep -q 'OCI_CLI_AUTH' /home/opc/.bashrc 2>/dev/null || \ + echo 'export OCI_CLI_AUTH=instance_principal' >> /home/opc/.bashrc + + # Generate our own kubeconfig with the operator's instance principal, rather + # than depend on the module's runcmd ordering (which can place opc's + # kubeconfig AFTER this script runs). Discover the cluster by name to avoid a + # dependency cycle on module.oke outputs. + log "discovering cluster and generating kubeconfig" + mkdir -p /home/opc/.kube + for i in $(seq 1 60); do + CID=$(oci ce cluster list --compartment-id '${var.compartment_ocid}' --name '${var.cluster_name}' --lifecycle-state ACTIVE --query 'data[0].id' --raw-output 2>/dev/null) + if [ -n "$CID" ] && oci ce cluster create-kubeconfig --cluster-id "$CID" --file "$KUBECONFIG" --region '${var.region}' --token-version 2.0.0 --kube-endpoint PRIVATE_ENDPOINT --overwrite 2>/dev/null; then + log "kubeconfig ready (cluster $CID)"; break + fi + log "waiting for cluster/kubeconfig ($i)"; sleep 10 + done + # Bake the auth mode into the token exec so kubectl works regardless of + # whether OCI_CLI_AUTH is set in the caller's shell (the generated exec block + # otherwise relies on that env var). + sed -i 's|^\( *\)- generate-token|\1- generate-token\n\1- --auth\n\1- instance_principal|' "$KUBECONFIG" 2>/dev/null || true + chown -R opc:opc /home/opc/.kube + + log "waiting for at least one Ready node" + for i in $(seq 1 120); do + if kubectl get nodes --no-headers 2>/dev/null | grep -q ' Ready '; then + log "node Ready"; break + fi + sleep 10 + done + + log "applying platform manifests" + cat > /tmp/platform.yaml <<'PLATFORM_EOF' + ${local.platform_yaml} + PLATFORM_EOF + kubectl apply -f /tmp/platform.yaml + + log "installing Spark Operator (if enabled)" + ${local.spark_helm} + + log "done" + EOT + + operator_cloud_init = [ + { + content_type = "text/x-shellscript" + filename = "50-platform-bootstrap.sh" + content = local.operator_bootstrap + }, + ] +} diff --git a/hadoop-spark-oci-sample/opensource/provider.tf b/hadoop-spark-oci-sample/opensource/provider.tf new file mode 100644 index 0000000..95aaf7f --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/provider.tf @@ -0,0 +1,49 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Terraform / provider configuration +# +# A single stack provisions everything in ONE apply: the OKE cluster and all OCI +# infrastructure (oci provider), plus a private operator VM whose cloud-init +# installs the in-cluster platform (Kerberos KDC, HDFS, Spark Operator). The +# platform is delivered as manifests + helm run from the operator INSIDE the VCN +# (see platform.tf / oke.tf), so there are no kubernetes/helm providers and the +# apply never needs to reach the cluster API - it works in Resource Manager with +# a fully private cluster. +############################################################################### + +terraform { + required_version = ">= 1.3.0" + + required_providers { + oci = { + source = "oracle/oci" + version = ">= 5.30.0" + } + tls = { + source = "hashicorp/tls" + version = ">= 4.0.0" + } + random = { + source = "hashicorp/random" + version = ">= 3.5.0" + } + null = { + source = "hashicorp/null" + version = ">= 3.2.0" + } + } +} + +provider "oci" { + region = var.region +} + +# Home-region alias. Global IAM writes (the Workload Identity dynamic group / +# policy in iam.tf, and any module IAM) must be executed in the tenancy home +# region, not the working region. +provider "oci" { + alias = "home" + region = local.home_region +} diff --git a/hadoop-spark-oci-sample/opensource/schema.yaml b/hadoop-spark-oci-sample/opensource/schema.yaml new file mode 100644 index 0000000..9242bda --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/schema.yaml @@ -0,0 +1,325 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +title: "Secure Hadoop & Spark on OKE" +description: "Deploys a security-hardened Oracle Kubernetes Engine (OKE) cluster for Apache Hadoop (HDFS) and Apache Spark. Storage is configurable - HDFS, OCI Object Storage, or both. This is the Stage 1 infrastructure layer." +schemaVersion: 1.1.0 +version: "20240101" +locale: "en" + +variableGroups: + - title: "Hidden / auto-populated" + visible: false + variables: + - tenancy_ocid + - compartment_ocid + - region + + - title: "General" + variables: + - cluster_name + - admin_cidr + - ssh_public_key + + - title: "OKE cluster" + variables: + - kubernetes_version + - cluster_endpoint_is_public + + - title: "Worker node pool" + variables: + - node_count + - node_shape + - node_ocpus + - node_memory_gbs + - node_boot_volume_gbs + + - title: "Storage backends" + variables: + - deploy_hdfs + - deploy_object_storage + + - title: "HDFS sizing" + variables: + - hdfs_replication + - hdfs_datanode_count + - hdfs_namenode_storage_gbs + - hdfs_datanode_storage_gbs + - kerberos_realm + - storage_class + + - title: "Spark" + variables: + - deploy_spark + - spark_operator_chart_version + + - title: "Container images" + variables: + - image_source + - hadoop_image + - spark_image + - kdc_image + + - title: "Networking" + variables: + - vcn_cidr + - endpoint_subnet_cidr + - nodes_subnet_cidr + - lb_subnet_cidr + + - title: "Software versions" + variables: + - hadoop_version + - spark_version + +variables: + + tenancy_ocid: + type: oci:identity:tenancy:id + title: "Tenancy" + compartment_ocid: + type: oci:identity:compartment:id + title: "Compartment" + required: true + region: + type: oci:identity:region:name + title: "Region" + required: true + + cluster_name: + type: string + title: "Cluster name" + description: "Prefix for every resource." + required: true + pattern: "^[a-z][a-z0-9-]{1,24}$" + default: "bigdata" + admin_cidr: + type: string + title: "Admin CIDR" + description: "The only network allowed to reach the Kubernetes API and open Bastion sessions. Use your own IP, e.g. 203.0.113.4/32. Must not be 0.0.0.0/0." + required: true + ssh_public_key: + type: oci:core:ssh:publickey + title: "SSH public key" + description: "Installed on worker nodes for break-glass access through the Bastion." + required: true + + kubernetes_version: + type: string + title: "Kubernetes version" + description: "Full version only (vMAJOR.MINOR.PATCH), e.g. v1.35.2. v1.35.2 is the latest GA; v1.36.0 is preview-only. A partial version like v1.36 has no matching worker image." + default: "v1.35.2" + required: true + pattern: "^v[0-9]+\\.[0-9]+\\.[0-9]+$" + cluster_endpoint_is_public: + type: boolean + title: "Public (NSG-locked) API endpoint" + description: "On: API endpoint is public but restricted to the Admin CIDR (lets the Stage 2 workload layer deploy in one run). Off: fully private endpoint, reached via Bastion." + default: true + + node_count: + type: integer + title: "Worker node count" + minimum: 2 + maximum: 100 + default: 3 + required: true + node_shape: + type: oci:core:instanceshape:name + title: "Worker node shape" + required: true + default: "VM.Standard.E5.Flex" + dependsOn: + compartmentId: ${compartment_ocid} + node_ocpus: + type: integer + title: "OCPUs per node" + minimum: 2 + maximum: 128 + default: 4 + node_memory_gbs: + type: integer + title: "Memory (GB) per node" + minimum: 16 + maximum: 2048 + default: 64 + node_boot_volume_gbs: + type: integer + title: "Boot volume (GB) per node" + minimum: 50 + maximum: 32768 + default: 100 + + deploy_hdfs: + type: boolean + title: "Deploy HDFS" + description: "Kerberos-secured HDFS (NameNode + DataNodes) running on the cluster." + default: true + deploy_object_storage: + type: boolean + title: "Deploy OCI Object Storage" + description: "Create a private bucket as a data lake, wired to Spark via Workload Identity." + default: true + + hdfs_replication: + type: integer + title: "HDFS replication factor" + minimum: 1 + maximum: 10 + default: 3 + visible: deploy_hdfs + hdfs_datanode_count: + type: integer + title: "HDFS DataNode count" + minimum: 1 + maximum: 100 + default: 3 + visible: deploy_hdfs + hdfs_namenode_storage_gbs: + type: integer + title: "NameNode volume (GB)" + minimum: 50 + maximum: 32768 + default: 100 + visible: deploy_hdfs + hdfs_datanode_storage_gbs: + type: integer + title: "DataNode volume (GB)" + minimum: 50 + maximum: 32768 + default: 200 + visible: deploy_hdfs + kerberos_realm: + type: string + title: "Kerberos realm" + pattern: "^[A-Z][A-Z0-9.-]{2,48}$" + default: "HADOOP.INTERNAL" + visible: deploy_hdfs + storage_class: + type: string + title: "HDFS StorageClass" + description: "Kubernetes StorageClass for HDFS / KDC volumes (oci-bv = OCI Block Volume CSI)." + default: "oci-bv" + visible: deploy_hdfs + + deploy_spark: + type: boolean + title: "Deploy Apache Spark (Spark Operator)" + default: true + spark_operator_chart_version: + type: string + title: "Spark Operator Helm chart version" + default: "1.4.6" + visible: deploy_spark + + image_source: + type: enum + title: "Container image source" + description: "'upstream' = public Apache images (one-click). 'ocir' = your own hardened images in OCI Registry." + enum: + - "upstream" + - "ocir" + default: "upstream" + required: true + hadoop_image: + type: string + title: "Hadoop image" + description: "Fully-qualified (OKE CRI-O enforces short-name mode)." + default: "docker.io/apache/hadoop:3.3.6" + spark_image: + type: string + title: "Spark image" + description: "Fully-qualified (OKE CRI-O enforces short-name mode)." + default: "docker.io/apache/spark:3.5.3" + kdc_image: + type: string + title: "KDC base image" + default: "docker.io/library/oraclelinux:8" + visible: deploy_hdfs + + vcn_cidr: + type: string + title: "VCN CIDR" + default: "10.20.0.0/16" + required: true + endpoint_subnet_cidr: + type: string + title: "API endpoint subnet CIDR" + default: "10.20.0.0/28" + required: true + nodes_subnet_cidr: + type: string + title: "Worker node subnet CIDR" + default: "10.20.1.0/24" + required: true + lb_subnet_cidr: + type: string + title: "Internal LB subnet CIDR" + description: "Private subnet for service load balancers (required by OKE; kept private)." + default: "10.20.2.0/24" + required: true + + hadoop_version: + type: string + title: "Hadoop version" + default: "3.3.6" + required: true + spark_version: + type: string + title: "Spark version" + default: "3.5.3" + required: true + +outputs: + cluster_id: + type: string + title: "OKE cluster OCID" + bastion_id: + type: string + title: "Bastion OCID" + object_storage_bucket: + type: string + title: "Object Storage bucket" + ssh_to_operator: + type: copyableString + title: "SSH to operator host" + operator_private_ip: + type: string + title: "Operator private IP" + cluster_summary: + type: string + title: "Summary" + namespace: + type: string + title: "Platform namespace" + hdfs_url: + type: string + title: "HDFS URL (fs.defaultFS)" + kdc_host: + type: string + title: "Kerberos KDC host" + object_storage_path: + type: string + title: "Object Storage path for Spark" + spark_smoke_test: + type: copyableString + title: "Run SparkPi smoke test" + +outputGroups: + - title: "Cluster" + outputs: + - cluster_id + - bastion_id + - operator_private_ip + - ssh_to_operator + - cluster_summary + - title: "Platform" + outputs: + - namespace + - hdfs_url + - kdc_host + - object_storage_bucket + - object_storage_path + - spark_smoke_test + +primaryOutputButton: cluster_summary diff --git a/hadoop-spark-oci-sample/opensource/scripts/connect-operator.sh b/hadoop-spark-oci-sample/opensource/scripts/connect-operator.sh new file mode 100755 index 0000000..e849abf --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/scripts/connect-operator.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# One-command connect to the OKE operator through the OCI Bastion. +# +# Creates a port-forwarding Bastion session, waits for it to become ACTIVE, and +# drops you into an SSH shell on the operator (via a ProxyCommand jump - no +# local port to manage). The session is deleted automatically on exit. +# +# Usage: +# scripts/connect-operator.sh # reads `terraform output` +# scripts/connect-operator.sh -b -i +# scripts/connect-operator.sh ... -k ~/.ssh/id_rsa # custom key +# scripts/connect-operator.sh ... -- kubectl -n bigdata get pods # run a command +# +# Requires: oci CLI, ssh. Your SSH public key (.pub) must be the one you +# passed to the stack as ssh_public_key. +############################################################################### +set -euo pipefail + +BASTION_ID="${BASTION_ID:-}" +OPERATOR_IP="${OPERATOR_IP:-}" +KEY="${KEY:-$HOME/.ssh/id_rsa}" +REMOTE_CMD=() + +while [ $# -gt 0 ]; do + case "$1" in + -b|--bastion-id) BASTION_ID="$2"; shift 2 ;; + -i|--operator-ip) OPERATOR_IP="$2"; shift 2 ;; + -k|--key) KEY="$2"; shift 2 ;; + --) shift; REMOTE_CMD=("$@"); break ;; + -h|--help) sed -n '2,20p' "$0"; exit 0 ;; + *) echo "unknown argument: $1 (see --help)" >&2; exit 1 ;; + esac +done + +# Fall back to Terraform outputs when run from the stack directory. +if command -v terraform >/dev/null 2>&1; then + [ -n "$BASTION_ID" ] || BASTION_ID="$(terraform output -raw bastion_id 2>/dev/null || true)" + [ -n "$OPERATOR_IP" ] || OPERATOR_IP="$(terraform output -raw operator_private_ip 2>/dev/null || true)" +fi + +[ -n "$BASTION_ID" ] || { echo "error: bastion OCID not set (-b, \$BASTION_ID, or terraform output bastion_id)" >&2; exit 1; } +[ -n "$OPERATOR_IP" ] || { echo "error: operator IP not set (-i, \$OPERATOR_IP, or terraform output operator_private_ip)" >&2; exit 1; } +[ -f "$KEY" ] || { echo "error: SSH private key not found: $KEY" >&2; exit 1; } +[ -f "$KEY.pub" ] || { echo "error: SSH public key not found: $KEY.pub" >&2; exit 1; } +command -v oci >/dev/null 2>&1 || { echo "error: oci CLI not found" >&2; exit 1; } + +REGION="$(printf '%s' "$BASTION_ID" | cut -d. -f4)" # region is embedded in the OCID +BASTION_HOST="host.bastion.${REGION}.oci.oraclecloud.com" + +echo ">> creating port-forwarding session to ${OPERATOR_IP}:22 (region ${REGION})" +SID="$(oci bastion session create-port-forwarding \ + --bastion-id "$BASTION_ID" \ + --target-private-ip "$OPERATOR_IP" --target-port 22 \ + --ssh-public-key-file "$KEY.pub" \ + --session-ttl 10800 --display-name "operator-connect" \ + --query 'data.id' --raw-output)" + +cleanup() { oci bastion session delete --session-id "$SID" --force >/dev/null 2>&1 || true; } +trap cleanup EXIT + +echo ">> waiting for session to become ACTIVE ($SID)" +for _ in $(seq 1 60); do + st="$(oci bastion session get --session-id "$SID" --query 'data."lifecycle-state"' --raw-output 2>/dev/null || true)" + [ "$st" = "ACTIVE" ] && break + [ "$st" = "FAILED" ] && { echo "error: session entered FAILED state" >&2; exit 1; } + sleep 5 +done +[ "${st:-}" = "ACTIVE" ] || { echo "error: session did not become ACTIVE in time" >&2; exit 1; } + +# IdentitiesOnly=yes: use ONLY the -i key, so ssh-agent keys aren't offered +# first and exhaust the bastion's MaxAuthTries (a common publickey failure). +# ServerAlive*: detect a half-open bastion tunnel (idle/TTL-expired) and drop the +# connection instead of letting every tunnelled kubectl hang indefinitely. +SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes \ + -o ServerAliveInterval=30 -o ServerAliveCountMax=3) +PROXY="ssh ${SSH_OPTS[*]} -i '$KEY' -W %h:%p -p 22 ${SID}@${BASTION_HOST}" + +# A freshly-ACTIVE Bastion session takes a few seconds before its SSH gateway +# accepts the key. Probe (non-interactively) until it does, so the interactive +# connection below doesn't fail on the not-yet-ready window. +echo ">> waiting for the session to accept connections" +for _ in $(seq 1 18); do + # -n: read stdin from /dev/null so the probe never consumes stdin meant for the + # final command (lets you pipe a file: connect-operator.sh ... -- 'cat > f' < f). + if ssh -n "${SSH_OPTS[@]}" -i "$KEY" -o BatchMode=yes -o ConnectTimeout=10 \ + -o ProxyCommand="$PROXY" "opc@${OPERATOR_IP}" true 2>/dev/null; then + break + fi + sleep 5 +done + +echo ">> connecting to opc@${OPERATOR_IP}" +# ${REMOTE_CMD[@]+...} expands to nothing when empty - safe under `set -u` on +# bash 3.2 (macOS), where a bare "${REMOTE_CMD[@]}" on an empty array errors. +ssh "${SSH_OPTS[@]}" -i "$KEY" -o ProxyCommand="$PROXY" \ + "opc@${OPERATOR_IP}" ${REMOTE_CMD[@]+"${REMOTE_CMD[@]}"} diff --git a/hadoop-spark-oci-sample/opensource/scripts/datanode-entrypoint.sh b/hadoop-spark-oci-sample/opensource/scripts/datanode-entrypoint.sh new file mode 100644 index 0000000..bb54e68 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/scripts/datanode-entrypoint.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# HDFS DataNode entrypoint. +############################################################################### +set -uo pipefail +export HADOOP_CONF_DIR=/hadoop-config +cp /hadoop-config/krb5.conf /etc/krb5.conf 2>/dev/null || true + +HDFS_BIN="$(command -v hdfs || echo "$HADOOP_HOME/bin/hdfs")" + +echo "[datanode] starting" +exec "$HDFS_BIN" --config /hadoop-config datanode diff --git a/hadoop-spark-oci-sample/opensource/scripts/kdc-entrypoint.sh b/hadoop-spark-oci-sample/opensource/scripts/kdc-entrypoint.sh new file mode 100644 index 0000000..503f422 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/scripts/kdc-entrypoint.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Kerberos KDC entrypoint (runs in the kdc StatefulSet pod). +# Environment: REALM, KDC_DB_PASSWORD, KADMIN_PASSWORD, HADOOP_USER_PASSWORD +############################################################################### +set -uo pipefail +echo "[kdc] starting; realm=$REALM" + +# Install the krb5 server on the upstream base image (a prebuilt OCIR image +# already has it - this is then a no-op). +if ! command -v kdb5_util >/dev/null 2>&1; then + echo "[kdc] installing krb5-server" + dnf install -y krb5-server krb5-workstation || { echo "[kdc] package install failed"; exit 1; } +fi + +# /etc/krb5.conf and the realm-dir config come from the mounted ConfigMap. +cp /kdc-config/krb5.conf /etc/krb5.conf +mkdir -p /var/kerberos/krb5kdc +cp /kdc-config/kdc.conf /var/kerberos/krb5kdc/kdc.conf +cp /kdc-config/kadm5.acl /var/kerberos/krb5kdc/kadm5.acl + +# Initialise the realm database once - it lives on a PersistentVolume. +if [ ! -f /var/kerberos/krb5kdc/principal ]; then + echo "[kdc] creating realm database" + kdb5_util create -s -r "$REALM" -P "$KDC_DB_PASSWORD" || { echo "[kdc] kdb5_util failed"; exit 1; } + kadmin.local -q "addprinc -pw $KADMIN_PASSWORD admin/admin@$REALM" + kadmin.local -q "addprinc -pw $HADOOP_USER_PASSWORD hadoop@$REALM" + echo "[kdc] realm initialised" +fi + +# Start the KDC (forks), then run the admin server in the foreground so the +# container's lifecycle tracks kadmind. +echo "[kdc] starting krb5kdc + kadmind" +krb5kdc +exec kadmind -nofork diff --git a/hadoop-spark-oci-sample/opensource/scripts/keytab-init.sh b/hadoop-spark-oci-sample/opensource/scripts/keytab-init.sh new file mode 100644 index 0000000..dbec28b --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/scripts/keytab-init.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# HDFS keytab init-container. +# +# Creates this pod's Kerberos service principals against the KDC and extracts a +# keytab into the shared /keytabs volume that the main container consumes. +# Environment: REALM, KADMIN_PASSWORD, POD_SERVICE, POD_NAMESPACE +############################################################################### +set -uo pipefail + +# Build the FQDN deterministically from the pod's stable network identity, using +# the Downward API (POD_NAME=metadata.name). Do NOT use `hostname`/`hostname -f`: +# the base image has no reliable hostname resolution (returns empty), and the +# pod's resolv.conf search domains don't include the governing service subdomain +# - either way the wrong principal gets baked into the keytab. This matches the +# DNS name HDFS resolves _HOST to (reverse DNS of the pod IP). +: "${POD_NAME:?POD_NAME is required}" +: "${POD_SERVICE:?POD_SERVICE is required}" +: "${POD_NAMESPACE:?POD_NAMESPACE is required}" +FQDN="${POD_NAME}.${POD_SERVICE}.${POD_NAMESPACE}.svc.cluster.local" +echo "[keytab-init] host=$FQDN" + +if ! command -v kadmin >/dev/null 2>&1; then + echo "[keytab-init] installing krb5-workstation" + dnf install -y krb5-workstation || { echo "[keytab-init] package install failed"; exit 1; } +fi +cp /kdc-config/krb5.conf /etc/krb5.conf + +KADM="kadmin -p admin/admin@$REALM -w $KADMIN_PASSWORD" + +# Wait for the KDC / kadmind to answer. +ready="" +for i in $(seq 1 60); do + if $KADM -q "get_principal admin/admin@$REALM" >/dev/null 2>&1; then + ready="yes"; break + fi + echo "[keytab-init] waiting for KDC ($i/60)"; sleep 5 +done +[ -z "$ready" ] && { echo "[keytab-init] KDC never became reachable"; exit 1; } + +# Create this pod's principals (idempotent) and extract a single keytab. +for p in "hdfs/$FQDN" "HTTP/$FQDN"; do + $KADM -q "addprinc -randkey $p" >/dev/null 2>&1 || true +done +rm -f /keytabs/hdfs.keytab +if ! $KADM -q "ktadd -k /keytabs/hdfs.keytab hdfs/$FQDN HTTP/$FQDN"; then + echo "[keytab-init] ktadd failed"; exit 1 +fi +chmod 600 /keytabs/hdfs.keytab +echo "[keytab-init] keytab written for $FQDN" diff --git a/hadoop-spark-oci-sample/opensource/scripts/namenode-entrypoint.sh b/hadoop-spark-oci-sample/opensource/scripts/namenode-entrypoint.sh new file mode 100644 index 0000000..34223fe --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/scripts/namenode-entrypoint.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# HDFS NameNode entrypoint. +############################################################################### +set -uo pipefail +export HADOOP_CONF_DIR=/hadoop-config +cp /hadoop-config/krb5.conf /etc/krb5.conf 2>/dev/null || true + +HDFS_BIN="$(command -v hdfs || echo "$HADOOP_HOME/bin/hdfs")" + +# Format the NameNode once; its metadata lives on a PersistentVolume. +if [ ! -d /hadoop/dfs/name/current ]; then + echo "[namenode] formatting filesystem" + "$HDFS_BIN" --config /hadoop-config namenode -format -force -nonInteractive \ + || { echo "[namenode] format failed"; exit 1; } +fi + +echo "[namenode] starting" +exec "$HDFS_BIN" --config /hadoop-config namenode diff --git a/hadoop-spark-oci-sample/opensource/secrets.tf b/hadoop-spark-oci-sample/opensource/secrets.tf new file mode 100644 index 0000000..cfeff8a --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/secrets.tf @@ -0,0 +1,28 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Kerberos credentials (when deploy_hdfs = true) +# +# Generated by Terraform and delivered to the cluster inside the operator's +# bootstrap manifests (platform.tf -> the kerberos-creds Secret). Consumed by +# the KDC and the HDFS keytab init-containers. +############################################################################### + +resource "random_password" "kdc_db" { + count = var.deploy_hdfs ? 1 : 0 + length = 32 + special = false +} + +resource "random_password" "kadmin" { + count = var.deploy_hdfs ? 1 : 0 + length = 32 + special = false +} + +resource "random_password" "hadoop_user" { + count = var.deploy_hdfs ? 1 : 0 + length = 24 + special = false +} diff --git a/hadoop-spark-oci-sample/opensource/storage.tf b/hadoop-spark-oci-sample/opensource/storage.tf new file mode 100644 index 0000000..a79f911 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/storage.tf @@ -0,0 +1,66 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# OCI Object Storage (created when deploy_object_storage = true) +# +# A private, versioned bucket used as a data lake. Spark pods reach it with no +# static credentials via OKE Workload Identity (see iam.tf). OCI Object Storage +# is encrypted at rest by default. +############################################################################### + +data "oci_objectstorage_namespace" "this" { + compartment_id = var.compartment_ocid +} + +resource "oci_objectstorage_bucket" "data" { + count = var.deploy_object_storage ? 1 : 0 + + compartment_id = var.compartment_ocid + namespace = data.oci_objectstorage_namespace.this.namespace + name = local.bucket_name + access_type = "NoPublicAccess" + versioning = "Enabled" + freeform_tags = local.freeform_tags +} + +# The OCI provider cannot delete a non-empty bucket (there is no force_destroy), +# and versioning keeps old object versions around - so `terraform destroy` fails +# with 409-BucketNotEmpty once any data has been written. When force_destroy_bucket +# is true, empty the bucket (all objects AND versions) at destroy time, BEFORE the +# bucket resource is removed (depends_on drives that ordering). Requires the `oci` +# CLI (which bundles python3) on the apply/destroy host - present in OCI Resource +# Manager. Set force_destroy_bucket = false to protect real data from teardown. +resource "null_resource" "empty_bucket_on_destroy" { + count = var.deploy_object_storage && var.force_destroy_bucket ? 1 : 0 + + triggers = { + bucket = local.bucket_name + namespace = data.oci_objectstorage_namespace.this.namespace + region = var.region + } + + depends_on = [oci_objectstorage_bucket.data] + + provisioner "local-exec" { + when = destroy + interpreter = ["/bin/bash", "-c"] + environment = { + B = self.triggers.bucket + N = self.triggers.namespace + R = self.triggers.region + } + command = <<-EOT + set -uo pipefail + oci os object bulk-delete -bn "$B" -ns "$N" --region "$R" --force 2>/dev/null || true + oci os object list-object-versions -bn "$B" -ns "$N" --region "$R" --all --output json 2>/dev/null \ + | python3 -c 'import sys, json, subprocess, os +B, N, R = os.environ["B"], os.environ["N"], os.environ["R"] +items = (json.load(sys.stdin).get("data", {}) or {}).get("items", []) or [] +for it in items: + subprocess.run(["oci", "os", "object", "delete", "-bn", B, "-ns", N, "--region", R, + "--name", it["name"], "--version-id", it["version-id"], "--force"], check=False) +print("emptied %d object versions from %s" % (len(items), B))' + EOT + } +} diff --git a/hadoop-spark-oci-sample/opensource/use-cases/01-spark-only/job.py b/hadoop-spark-oci-sample/opensource/use-cases/01-spark-only/job.py new file mode 100644 index 0000000..b3a2c41 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/01-spark-only/job.py @@ -0,0 +1,62 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +""" +Spark-only demo: generate a synthetic dataset entirely in-cluster and run a +distributed aggregation. No external storage is touched - this proves the +Spark-on-Kubernetes engine (driver + executors via the Spark Operator) works. + +It models a stream of retail transactions and answers "revenue by region". +The job narrates the transformation as STEP/SCHEMA/SAMPLE lines (input -> logic +-> output, with real data tables) plus PROOF:/RESULT: lines the runner greps. +""" +from pyspark.sql import SparkSession, functions as F + + +def show(df, n=5): + # Print an actual Spark table so viewers SEE the data, not just row counts. + for line in df._jdf.showString(n, 20, False).rstrip("\n").split("\n"): + print(f"SAMPLE: {line}") + + +spark = SparkSession.builder.appName("spark-only-demo").getOrCreate() +sc = spark.sparkContext +print(f"PROOF: spark.version={spark.version} executors~={sc.defaultParallelism}") + +# ---- STEP 1: INPUT - generate ~5M synthetic transactions on the executors --- +n = 5_000_000 +txns = ( + spark.range(n) + .withColumn("region", (F.col("id") % F.lit(5)).cast("int")) + .withColumn("category", (F.rand(seed=1) * 8).cast("int")) + .withColumn("amount", F.round(F.rand(seed=2) * 100, 2)) + .drop("id") +) +generated = txns.count() +print("STEP: 1/3 INPUT - synthetic retail transactions (region, category, amount)") +print(f"SCHEMA: input {txns.schema.simpleString()}") +print(f"PROOF: input_rows={generated}") +show(txns, 5) + +# ---- STEP 2: TRANSFORM ------------------------------------------------------ +print("STEP: 2/3 TRANSFORM - GROUP BY region -> count(*) AS txns, round(sum(amount),2) AS revenue") + +# ---- STEP 3: OUTPUT - distributed aggregation ------------------------------- +by_region = ( + txns.groupBy("region") + .agg(F.count("*").alias("txns"), F.round(F.sum("amount"), 2).alias("revenue")) + .orderBy("region") +) +rows = by_region.collect() +print("STEP: 3/3 OUTPUT - revenue by region") +print(f"SCHEMA: output {by_region.schema.simpleString()}") +print(f"PROOF: output_rows={len(rows)} (aggregated down from {generated})") +show(by_region, 20) +for r in rows: + print(f"RESULT: region={r['region']} txns={r['txns']} revenue={r['revenue']}") + +top = max(rows, key=lambda r: r["revenue"]) +print(f"RESULT: top_region={top['region']} revenue={top['revenue']}") +print("PROOF: spark-only demo OK") + +spark.stop() diff --git a/hadoop-spark-oci-sample/opensource/use-cases/01-spark-only/run.sh b/hadoop-spark-oci-sample/opensource/use-cases/01-spark-only/run.sh new file mode 100755 index 0000000..f15a65e --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/01-spark-only/run.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Demo 1 - Spark only (deploy_spark = true) +# +# Proves Spark-on-Kubernetes works: the Spark Operator schedules a driver + +# executors that generate data and run a distributed aggregation, with no +# external storage. Use this with the "spark only" deployment profile. +# +# Run from the operator host: ./run.sh +############################################################################### +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib/common.sh +source "$HERE/../lib/common.sh" + +require_namespace +kubectl -n "$NS" get deploy -l app.kubernetes.io/name=spark-operator >/dev/null 2>&1 \ + || kubectl -n "$NS" get pods | grep -q spark-operator \ + || die "spark-operator not found in namespace $NS (is deploy_spark = true?)" + +APP="spark-only-demo" +submit_pyspark "$APP" "$HERE/job.py" +show_proof "$APP" + +ok "Demo 1 complete: Spark generated data and aggregated it on Kubernetes." +info "Cleanup: kubectl -n $NS delete sparkapplication $APP configmap ${APP}-code" diff --git a/hadoop-spark-oci-sample/opensource/use-cases/02-hdfs-spark/job.py b/hadoop-spark-oci-sample/opensource/use-cases/02-hdfs-spark/job.py new file mode 100644 index 0000000..96c991a --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/02-hdfs-spark/job.py @@ -0,0 +1,59 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +""" +HDFS + Spark demo: read a CSV that was landed in Kerberos-secured HDFS, run a +distributed aggregation, and write the results back to HDFS. Proves Spark can +authenticate to secured HDFS (via a keytab) and read/write it. + +The job narrates the transformation as STEP/SCHEMA/SAMPLE lines (input -> logic +-> output, with real data tables) plus PROOF:/RESULT: lines the runner greps. + +Args: +""" +import sys +from pyspark.sql import SparkSession, functions as F + + +def show(df, n=5): + # Print an actual Spark table so viewers SEE the data, not just row counts. + for line in df._jdf.showString(n, 20, False).rstrip("\n").split("\n"): + print(f"SAMPLE: {line}") + + +src, dst = sys.argv[1], sys.argv[2] +spark = SparkSession.builder.appName("hdfs-spark-demo").getOrCreate() +print(f"PROOF: spark.version={spark.version}") +print(f"PROOF: defaultFS={spark.sparkContext._jsc.hadoopConfiguration().get('fs.defaultFS')}") + +# ---- STEP 1: INPUT - read the CSV from Kerberos-secured HDFS ---------------- +df = spark.read.option("header", True).csv(src) +rows = df.count() +print("STEP: 1/3 INPUT - CSV read from Kerberos-secured HDFS") +print(f"SCHEMA: input {df.schema.simpleString()}") +print(f"PROOF: read_rows_from_hdfs={rows} src={src}") +show(df, 5) + +# ---- STEP 2: TRANSFORM ------------------------------------------------------ +print("STEP: 2/3 TRANSFORM - GROUP BY category -> count(*) AS txns, round(sum(amount),2) AS revenue") + +# ---- STEP 3: OUTPUT - aggregate, print, and write back to HDFS -------------- +agg = ( + df.groupBy("category") + .agg( + F.count("*").alias("txns"), + F.round(F.sum(F.col("amount").cast("double")), 2).alias("revenue"), + ) + .orderBy("category") +) +result = agg.collect() +print("STEP: 3/3 OUTPUT - revenue by category (written back to HDFS)") +print(f"SCHEMA: output {agg.schema.simpleString()}") +print(f"PROOF: output_rows={len(result)} (aggregated down from {rows})") +show(agg, 20) +for r in result: + print(f"RESULT: category={r['category']} txns={r['txns']} revenue={r['revenue']}") + +agg.coalesce(1).write.mode("overwrite").option("header", True).csv(dst) +print(f"PROOF: wrote_results_to_hdfs={dst}") +spark.stop() diff --git a/hadoop-spark-oci-sample/opensource/use-cases/02-hdfs-spark/run.sh b/hadoop-spark-oci-sample/opensource/use-cases/02-hdfs-spark/run.sh new file mode 100755 index 0000000..ee11d09 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/02-hdfs-spark/run.sh @@ -0,0 +1,147 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Demo 2 - HDFS + Spark (deploy_hdfs = true, deploy_spark = true) +# +# Part A (core proof): authenticate to the KDC and write/read data in +# Kerberos-secured HDFS directly from the NameNode pod. +# Part B (integration): run a Spark job that reads that data from HDFS using a +# keytab, aggregates it, and writes results back to HDFS. +# +# Run from the operator host: ./run.sh +############################################################################### +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib/common.sh +source "$HERE/../lib/common.sh" + +require_namespace +NN_POD="${NN_POD:-namenode-0}" +KDC_POD="${KDC_POD:-kdc-0}" +HDFS_FS="hdfs://${NN_POD}.hdfs-nn.${NS}.svc.cluster.local:9000" +IN="${HDFS_FS}/demo/input" +OUT="${HDFS_FS}/demo/output" + +kubectl -n "$NS" get pod "$NN_POD" >/dev/null 2>&1 || die "$NN_POD not found (is deploy_hdfs = true and the pod Running?)" + +# Realm + the hadoop principal's password (from the Kerberos secret). +REALM="${REALM:-$(kubectl -n "$NS" get cm kdc-config -o go-template='{{index .data "krb5.conf"}}' 2>/dev/null | sed -n 's/.*default_realm = //p' | head -1)}" +REALM="${REALM:-HADOOP.INTERNAL}" +KPW="$(kubectl -n "$NS" get secret kerberos-creds -o jsonpath='{.data.HADOOP_USER_PASSWORD}' | base64 -d)" +info "realm=$REALM namenode=$NN_POD" + +############################### Part A: HDFS ################################## +log "Part A - generate data and land it in secured HDFS" + +# Generate a synthetic CSV locally, then stream it into HDFS via the pod. +TMP="$(mktemp)"; trap 'rm -f "$TMP"' EXIT +awk 'BEGIN{srand(7);print "category,amount";for(i=0;i<100000;i++)printf "%d,%.2f\n",int(rand()*8),rand()*1000}' > "$TMP" +info "generated $(wc -l <"$TMP") lines locally" + +# HADOOP_CONF_DIR must point at the mounted config, or the hdfs CLI defaults to +# fs.defaultFS=file:/// and writes to the pod's LOCAL disk instead of HDFS (an +# interactive `kubectl exec` shell doesn't inherit the entrypoint's env). +HC="export HADOOP_CONF_DIR=/hadoop-config;" + +# HDFS root "/" is owned by the hdfs superuser, so the hadoop end-user cannot +# create /demo. Provision it as the superuser (using the NameNode's own keytab) +# and hand ownership to hadoop. The exact hdfs principal - its _HOST resolves to +# the pod's reverse-DNS name - is read straight from the keytab, so we never guess. +kubectl -n "$NS" exec "$NN_POD" -- bash -lc "$HC"' P=$(klist -kt /keytabs/hdfs.keytab | awk "/hdfs\//{print \$NF; exit}"); kinit -kt /keytabs/hdfs.keytab "$P" && hdfs dfs -mkdir -p /demo && hdfs dfs -chown hadoop:hadoop /demo' || die "failed to provision /demo as the hdfs superuser" +ok "provisioned /demo in HDFS (owner hadoop) via the hdfs superuser" + +# kinit as the hadoop end-user (password via stdin, never on the command line). +# Proves user auth works and leaves hadoop's ticket in the pod cache for the write. +printf '%s' "$KPW" | kubectl -n "$NS" exec -i "$NN_POD" -- \ + bash -lc "kinit hadoop@$REALM && klist" || die "kinit failed" +ok "kinit hadoop@$REALM succeeded (Kerberos auth works)" + +kubectl -n "$NS" exec -i "$NN_POD" -- bash -lc "$HC hdfs dfs -mkdir -p /demo/input && hdfs dfs -put -f - /demo/input/data.csv" < "$TMP" +kubectl -n "$NS" exec "$NN_POD" -- bash -lc "$HC"' echo "HDFS listing:"; hdfs dfs -ls /demo/input; echo "HDFS head:"; hdfs dfs -cat /demo/input/data.csv | head -3' +ok "Part A complete: data written to and read from Kerberos-secured HDFS." + +############################### Part B: Spark ################################# +log "Part B - Spark reads HDFS with a keytab, aggregates, writes back" + +# Export a keytab for hadoop@REALM from the KDC (norandkey keeps the password key). +if kubectl -n "$NS" get pod "$KDC_POD" >/dev/null 2>&1; then + kubectl -n "$NS" exec "$KDC_POD" -- bash -lc \ + "kadmin.local -q 'ktadd -norandkey -k /tmp/hadoop.keytab hadoop@$REALM'" >/dev/null + kubectl -n "$NS" exec "$KDC_POD" -- cat /tmp/hadoop.keytab > "$TMP.keytab" + # spark.kerberos.keytab is read by the SUBMITTER - the spark-operator pod that + # runs spark-submit - which then distributes the keytab to the driver. So the + # keytab must exist ON THE OPERATOR POD, not just mounted in the driver. + OP="$(kubectl -n "$NS" get pod -l app.kubernetes.io/name=spark-operator -o jsonpath='{.items[0].metadata.name}')" + [ -n "$OP" ] || die "spark-operator pod not found" + base64 "$TMP.keytab" | kubectl -n "$NS" exec -i "$OP" -- sh -c 'base64 -d > /tmp/hadoop.keytab' + rm -f "$TMP.keytab" + ok "keytab for hadoop@$REALM placed on the spark-operator (submitter) pod: $OP" +else + die "$KDC_POD not found - cannot export a keytab for the Spark job" +fi + +APP="hdfs-spark-demo" +kubectl -n "$NS" delete sparkapplication "$APP" >/dev/null 2>&1 || true +kubectl -n "$NS" create configmap "$APP-code" --from-file=job.py="$HERE/job.py" \ + --dry-run=client -o yaml | kubectl apply -f - >/dev/null + +kubectl apply -f - </dev/null +apiVersion: sparkoperator.k8s.io/v1beta2 +kind: SparkApplication +metadata: + name: $APP + namespace: $NS +spec: + type: Python + pythonVersion: "3" + mode: cluster + image: "$SPARK_IMAGE" + imagePullPolicy: IfNotPresent + mainApplicationFile: "local:///opt/spark/jobs/job.py" + arguments: ["$IN", "$OUT"] + sparkVersion: "$SPARK_VERSION" + restartPolicy: + type: Never + sparkConf: + # keytab path is on the submitter (operator) pod; Spark distributes it to + # the driver, which logs in and fetches HDFS delegation tokens. + spark.kerberos.principal: "hadoop@$REALM" + spark.kerberos.keytab: "/tmp/hadoop.keytab" + spark.kerberos.access.hadoopFileSystems: "$HDFS_FS" + spark.hadoop.hadoop.security.authentication: "kerberos" + volumes: + - name: job-code + configMap: { name: $APP-code } + - name: hadoop-conf + configMap: { name: hadoop-config } + - name: krb5 + configMap: { name: kdc-config } + driver: + cores: 1 + memory: "1g" + serviceAccount: $SPARK_SA + env: + - { name: HADOOP_CONF_DIR, value: /opt/hadoop/conf } + volumeMounts: &mounts + - { name: job-code, mountPath: /opt/spark/jobs } + - { name: hadoop-conf, mountPath: /opt/hadoop/conf } + - { name: krb5, mountPath: /etc/krb5.conf, subPath: krb5.conf } + executor: + cores: 1 + instances: 2 + memory: "1g" + env: + - { name: HADOOP_CONF_DIR, value: /opt/hadoop/conf } + volumeMounts: *mounts +YAML + +wait_sparkapp "$APP" +show_proof "$APP" + +log "verifying the Spark output landed in HDFS" +kubectl -n "$NS" exec "$NN_POD" -- bash -lc "$HC"' echo "OUTPUT:"; hdfs dfs -ls /demo/output; hdfs dfs -cat /demo/output/part-*.csv | head -10' + +ok "Demo 2 complete: secured HDFS + Spark read/write proven end to end." +info "Cleanup: kubectl -n $NS delete sparkapplication $APP configmap ${APP}-code" diff --git a/hadoop-spark-oci-sample/opensource/use-cases/03-objstore-spark/job.py b/hadoop-spark-oci-sample/opensource/use-cases/03-objstore-spark/job.py new file mode 100644 index 0000000..b7a04dd --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/03-objstore-spark/job.py @@ -0,0 +1,73 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +""" +Object Storage + Spark demo: generate data, write it to an OCI Object Storage +bucket over oci://, read it back, aggregate, and write the results back to the +bucket. Authentication is OKE Workload Identity (no keys) via the OCI HDFS +connector. Proves the bucket round-trip + compute. + +The job narrates the transformation as STEP/SCHEMA/SAMPLE lines (input -> round +-trip -> logic -> output, with real data tables) plus PROOF:/RESULT: lines. + +Arg: e.g. oci://bigdata-data@/demo +""" +import sys +from pyspark.sql import SparkSession, functions as F + + +def show(df, n=5): + # Print an actual Spark table so viewers SEE the data, not just row counts. + for line in df._jdf.showString(n, 20, False).rstrip("\n").split("\n"): + print(f"SAMPLE: {line}") + + +base = sys.argv[1].rstrip("/") +inp, outp = base + "/input", base + "/output" + +spark = SparkSession.builder.appName("objstore-spark-demo").getOrCreate() +print(f"PROOF: spark.version={spark.version}") + +# ---- STEP 1: INPUT - generate transactions and write them to Object Storage - +df = ( + spark.range(100000) + .withColumn("category", (F.rand(seed=1) * 8).cast("int")) + .withColumn("amount", F.round(F.rand(seed=2) * 1000, 2)) + .drop("id") +) +print("STEP: 1/4 INPUT - synthetic transactions generated in-cluster") +print(f"SCHEMA: input {df.schema.simpleString()}") +show(df, 5) +df.write.mode("overwrite").option("header", True).csv(inp) +print(f"PROOF: wrote_input_to_object_storage={inp}") + +# ---- STEP 2: ROUND-TRIP - read the data back from Object Storage ------------ +back = spark.read.option("header", True).csv(inp) +rows = back.count() +print("STEP: 2/4 ROUND-TRIP - read the data back from Object Storage over oci://") +print(f"PROOF: read_back_rows={rows}") +show(back, 5) + +# ---- STEP 3: TRANSFORM ------------------------------------------------------ +print("STEP: 3/4 TRANSFORM - GROUP BY category -> count(*) AS txns, round(sum(amount),2) AS revenue") + +# ---- STEP 4: OUTPUT - aggregate, print, and write back to Object Storage ---- +agg = ( + back.groupBy("category") + .agg( + F.count("*").alias("txns"), + F.round(F.sum(F.col("amount").cast("double")), 2).alias("revenue"), + ) + .orderBy("category") +) +result = agg.collect() +print("STEP: 4/4 OUTPUT - revenue by category (written back to Object Storage)") +print(f"SCHEMA: output {agg.schema.simpleString()}") +print(f"PROOF: output_rows={len(result)} (aggregated down from {rows})") +show(agg, 20) +for r in result: + print(f"RESULT: category={r['category']} txns={r['txns']} revenue={r['revenue']}") + +agg.coalesce(1).write.mode("overwrite").option("header", True).csv(outp) +print(f"PROOF: wrote_results_to_object_storage={outp}") +spark.stop() diff --git a/hadoop-spark-oci-sample/opensource/use-cases/03-objstore-spark/run.sh b/hadoop-spark-oci-sample/opensource/use-cases/03-objstore-spark/run.sh new file mode 100755 index 0000000..7f044c0 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/03-objstore-spark/run.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Demo 3 - Object Storage + Spark (deploy_object_storage = true, deploy_spark = true) +# +# A Spark job generates data, writes it to the OCI Object Storage bucket over +# oci://, reads it back, aggregates, and writes results back - authenticating +# with OKE Workload Identity (no API keys). Proves the bucket round-trip. +# +# Run from the operator host: ./run.sh +# +# Env you may need to set: +# OS_NAMESPACE Object Storage namespace (from the object_storage_path +# output: oci://@/). Required if it +# cannot be auto-detected. +# REGION OCI region id (e.g. eu-frankfurt-1) - recommended. +# CONNECTOR_VERSION oci-hdfs-connector version (default below). +# AUTHENTICATOR connector auth class for OKE Workload Identity (default +# below) - adjust if your connector version differs. +############################################################################### +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=../lib/common.sh +source "$HERE/../lib/common.sh" + +require_namespace +BUCKET="${BUCKET:-${NS}-data}" +CONNECTOR_VERSION="${CONNECTOR_VERSION:-3.3.4.1.4.1}" # 3.3.4.1.4.1 added the OKE Workload Identity authenticator +AUTHENTICATOR="${AUTHENTICATOR:-com.oracle.bmc.hdfs.auth.OkeWorkloadIdentityCustomAuthenticator}" + +# Best-effort auto-detect of the Object Storage namespace from the operator. +if [ -z "${OS_NAMESPACE:-}" ]; then + OS_NAMESPACE="$(OCI_CLI_AUTH=instance_principal oci os ns get --query data --raw-output 2>/dev/null || true)" +fi +[ -n "${OS_NAMESPACE:-}" ] || die "set OS_NAMESPACE (see the object_storage_path output: oci://${BUCKET}@/)" + +BASE="oci://${BUCKET}@${OS_NAMESPACE}/demo" +info "bucket=$BUCKET os_namespace=$OS_NAMESPACE base=$BASE" +info "connector=oci-hdfs-connector:${CONNECTOR_VERSION} auth=${AUTHENTICATOR}" + +REGION_LINE="" +[ -n "${REGION:-}" ] && REGION_LINE=" spark.hadoop.fs.oci.client.regionCodeOrId: \"${REGION}\"" + +APP="objstore-spark-demo" +kubectl -n "$NS" delete sparkapplication "$APP" >/dev/null 2>&1 || true +kubectl -n "$NS" create configmap "$APP-code" --from-file=job.py="$HERE/job.py" \ + --dry-run=client -o yaml | kubectl apply -f - >/dev/null + +kubectl apply -f - </dev/null +apiVersion: sparkoperator.k8s.io/v1beta2 +kind: SparkApplication +metadata: + name: $APP + namespace: $NS +spec: + type: Python + pythonVersion: "3" + mode: cluster + image: "$SPARK_IMAGE" + imagePullPolicy: IfNotPresent + mainApplicationFile: "local:///opt/spark/jobs/job.py" + arguments: ["$BASE"] + sparkVersion: "$SPARK_VERSION" + restartPolicy: + type: Never + sparkConf: + # The oci-hdfs-connector needs Guava >= 27, but Spark 3.5 bundles Guava + # 14.0.1 in \$SPARK_HOME/jars and it wins on the classpath. userClassPathFirst + # "fixes" Guava but splits Hadoop AND Jersey across two classloaders + # (LinkageError / ClassCastException), because it flips class ordering for + # EVERY shared library, not just Guava. Instead we keep the normal classloader + # order and simply PREPEND a modern Guava (+ its required failureaccess) via + # extraClassPath. extraClassPath is fixed at JVM launch, so --packages can't + # satisfy it; an init container stages the jars into a shared volume first. + spark.jars.packages: "com.oracle.oci.sdk:oci-hdfs-connector:$CONNECTOR_VERSION" + spark.jars.ivy: "/tmp/.ivy2" + spark.driver.extraClassPath: "/opt/extra-jars/guava-32.1.3-jre.jar:/opt/extra-jars/failureaccess-1.0.1.jar" + spark.executor.extraClassPath: "/opt/extra-jars/guava-32.1.3-jre.jar:/opt/extra-jars/failureaccess-1.0.1.jar" + spark.hadoop.fs.oci.impl: "com.oracle.bmc.hdfs.BmcFilesystem" + spark.hadoop.fs.AbstractFileSystem.oci.impl: "com.oracle.bmc.hdfs.Bmc" + spark.hadoop.fs.oci.client.custom.authenticator: "$AUTHENTICATOR" +$REGION_LINE + volumes: + - name: job-code + configMap: { name: $APP-code } + - name: extra-jars + emptyDir: {} + driver: + cores: 1 + memory: "1g" + serviceAccount: $SPARK_SA + # Stage a modern Guava (+ failureaccess, required by Guava >= 27) so + # extraClassPath can put it ahead of Spark's Guava 14 on the same classpath. + # Reuses the Spark image (has python3) - no extra image pull, no CRI-O + # short-name concern. Egress to Maven Central goes via the NAT gateway. + initContainers: &fetchjars + - name: fetch-guava + image: "$SPARK_IMAGE" + command: + - python3 + - -c + - | + import urllib.request as u + b = "https://repo1.maven.org/maven2/com/google/guava" + u.urlretrieve(b + "/guava/32.1.3-jre/guava-32.1.3-jre.jar", "/opt/extra-jars/guava-32.1.3-jre.jar") + u.urlretrieve(b + "/failureaccess/1.0.1/failureaccess-1.0.1.jar", "/opt/extra-jars/failureaccess-1.0.1.jar") + volumeMounts: + - { name: extra-jars, mountPath: /opt/extra-jars } + volumeMounts: + - { name: job-code, mountPath: /opt/spark/jobs } + - { name: extra-jars, mountPath: /opt/extra-jars } + executor: + cores: 1 + instances: 2 + memory: "1g" + initContainers: *fetchjars + volumeMounts: + - { name: job-code, mountPath: /opt/spark/jobs } + - { name: extra-jars, mountPath: /opt/extra-jars } +YAML + +wait_sparkapp "$APP" +show_proof "$APP" +ok "Demo 3 complete: Spark wrote and read data in Object Storage via Workload Identity." +info "Cleanup: kubectl -n $NS delete sparkapplication $APP configmap ${APP}-code" diff --git a/hadoop-spark-oci-sample/opensource/use-cases/README.md b/hadoop-spark-oci-sample/opensource/use-cases/README.md new file mode 100644 index 0000000..d7343ae --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/README.md @@ -0,0 +1,69 @@ +# Examples — proving the three deployment profiles + +Production-shaped demos that **generate data and exercise the deployed Hadoop / +Spark platform**, printing clear `PROOF:` / `RESULT:` evidence that each +configuration works: + +| Demo | Profile | What it proves | +|------|---------|----------------| +| [`01-spark-only`](01-spark-only/) | `deploy_spark` | Spark-on-Kubernetes: driver + executors generate data and run a distributed aggregation — no external storage. | +| [`02-hdfs-spark`](02-hdfs-spark/) | `deploy_hdfs` + `deploy_spark` | Kerberos auth → write/read data in secured HDFS, then Spark reads HDFS with a keytab, aggregates, writes back. | +| [`03-objstore-spark`](03-objstore-spark/) | `deploy_object_storage` + `deploy_spark` | Spark round-trips data through an OCI Object Storage bucket over `oci://` using OKE Workload Identity (no keys). | + +## Where to run them + +Run from the **operator host** — it sits inside the VCN with `kubectl`, `helm` +and a working kubeconfig. The stack **writes these demos to the operator at +`/home/opc/use-cases`** on first boot (no keys, no copying). Reach the operator +through the OCI Bastion with the one-command helper (the `operator_access` +output prints it filled in): + +```bash +./scripts/connect-operator.sh -b -i +``` + +Then, on the operator: + +```bash +cd ~/use-cases +NS=bigdata ./01-spark-only/run.sh # NS = your cluster_name +``` + +(To iterate locally instead, the same files live in this repo's `use-cases/`.) + +Each script is self-contained: it submits a `SparkApplication` (via the Spark +Operator), waits for it to finish, and prints the proof from the driver log. A +non-zero exit means a step failed (the script prints `[FAIL] …` and the driver +log). + +## Configuration (environment variables) + +| Var | Default | Meaning | +|-----|---------|---------| +| `NS` | `bigdata` | Namespace = your `cluster_name`. | +| `SPARK_IMAGE` | `docker.io/apache/spark:3.5.3` | Image for the Spark jobs (fully-qualified; must include PySpark). | +| `SPARK_VERSION` | `3.5.3` | Spark version label. | +| `TIMEOUT` | `900` | Seconds to wait for a job. | +| `OS_NAMESPACE`, `REGION`, `CONNECTOR_VERSION`, `AUTHENTICATOR` | — | Demo 3 only (Object Storage). | + +## Honest notes + +- **PySpark image:** the demos run Python jobs, so `SPARK_IMAGE` must include + PySpark. `apache/spark:3.5.x` does; if yours doesn't, set `SPARK_IMAGE` to a + Python-enabled Spark image (or your hardened OCIR image). +- **Demo 2 (HDFS+Spark):** Part A (direct HDFS via the NameNode pod) is the + rock-solid proof of Kerberos + HDFS. Part B wires a keytab into the Spark pods + to read secured HDFS — a real production pattern, but Kerberos-on-Spark is + fiddly; if Part B needs tuning for your image, Part A still proves the data + path. +- **Demo 3 (Object Storage):** Spark reaches `oci://` via the **oci-hdfs-connector** + pulled with `--packages` and authenticated by OKE Workload Identity. The + connector **version** (`CONNECTOR_VERSION`) and **authenticator class** + (`AUTHENTICATOR`) are version-sensitive — adjust them to match your connector. + `--packages` downloads from Maven Central, which needs internet egress + (available via the NAT gateway). +- **NetworkPolicy / egress:** the namespace ships a default-deny-egress policy, + but it is **inert until Calico is installed** (flannel doesn't enforce it). If + you *have* installed Calico, allow Maven egress for Demo 3 or pre-bake the + connector into a custom image. +- These are demos: cleanup commands are printed at the end of each run. diff --git a/hadoop-spark-oci-sample/opensource/use-cases/lib/common.sh b/hadoop-spark-oci-sample/opensource/use-cases/lib/common.sh new file mode 100755 index 0000000..a60a248 --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/use-cases/lib/common.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash + +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ +############################################################################### +# Shared helpers for the use-case demos. Source this from each run.sh. +# +# Run the demos from the OPERATOR host (it has kubectl + a working kubeconfig). +# Reach the operator through the OCI Bastion - see use-cases/README.md. +############################################################################### +set -euo pipefail + +# ---- Config (override via environment) -------------------------------------- +NS="${NS:-bigdata}" # namespace == cluster_name +SPARK_IMAGE="${SPARK_IMAGE:-docker.io/apache/spark:3.5.3}" # fully-qualified: OKE CRI-O enforces short-name mode +SPARK_VERSION="${SPARK_VERSION:-3.5.3}" +SPARK_SA="${SPARK_SA:-spark}" +TIMEOUT="${TIMEOUT:-900}" # seconds to wait for a SparkApplication + +# ---- Pretty output ---------------------------------------------------------- +log() { printf '\n\033[1;34m== %s ==\033[0m\n' "$*"; } +info() { printf ' %s\n' "$*"; } +ok() { printf '\033[1;32m[PASS]\033[0m %s\n' "$*"; } +die() { printf '\033[1;31m[FAIL]\033[0m %s\n' "$*" >&2; exit 1; } + +need() { command -v "$1" >/dev/null 2>&1 || die "missing required command: $1"; } + +require_namespace() { + need kubectl + kubectl get ns "$NS" >/dev/null 2>&1 || die "namespace '$NS' not found (set NS=)" +} + +# Wait for a SparkApplication to reach a terminal state. +wait_sparkapp() { + local name="$1" state="" i=0 + log "waiting for SparkApplication/$name (timeout ${TIMEOUT}s)" + while (( i < TIMEOUT )); do + state="$(kubectl -n "$NS" get sparkapplication "$name" \ + -o jsonpath='{.status.applicationState.state}' 2>/dev/null || true)" + info "state=${state:-PENDING} (${i}s)" + case "$state" in + COMPLETED) ok "SparkApplication/$name COMPLETED"; return 0 ;; + FAILED|FAILING|INVALIDATING|UNKNOWN) + driver_logs "$name"; die "SparkApplication/$name -> $state" ;; + esac + sleep 10; (( i += 10 )) + done + driver_logs "$name"; die "SparkApplication/$name timed out after ${TIMEOUT}s" +} + +driver_logs() { + echo "------------------------- driver logs ($1-driver) -------------------------" + kubectl -n "$NS" logs "$1-driver" 2>/dev/null || info "(driver pod not found yet)" + echo "---------------------------------------------------------------------------" +} + +# Print the transformation story + evidence the job emitted. STEP/SCHEMA/SAMPLE +# narrate input -> transform -> output (with real data tables); PROOF/RESULT are +# the machine-checkable evidence. +show_proof() { + local name="$1" + log "transformation + proof from $name driver logs" + kubectl -n "$NS" logs "$1-driver" 2>/dev/null | grep -E '^(STEP|SCHEMA|SAMPLE|PROOF|RESULT):' \ + || die "no STEP/PROOF/RESULT lines found - the job did not produce evidence" +} + +# Submit a PySpark job from a local .py file (delivered via a ConfigMap). +# Usage: submit_pyspark [extra "spark.conf.key=value"...] +submit_pyspark() { + local name="$1" pyfile="$2"; shift 2 + [ -f "$pyfile" ] || die "job file not found: $pyfile" + + local conf="" + for kv in "$@"; do conf+=" ${kv%%=*}: \"${kv#*=}\""$'\n'; done + + log "submitting SparkApplication/$name (image=$SPARK_IMAGE)" + kubectl -n "$NS" delete sparkapplication "$name" >/dev/null 2>&1 || true + kubectl -n "$NS" create configmap "$name-code" --from-file=job.py="$pyfile" \ + --dry-run=client -o yaml | kubectl apply -f - >/dev/null + + kubectl apply -f - </dev/null +apiVersion: sparkoperator.k8s.io/v1beta2 +kind: SparkApplication +metadata: + name: $name + namespace: $NS +spec: + type: Python + pythonVersion: "3" + mode: cluster + image: "$SPARK_IMAGE" + imagePullPolicy: IfNotPresent + mainApplicationFile: "local:///opt/spark/jobs/job.py" + sparkVersion: "$SPARK_VERSION" + restartPolicy: + type: Never + sparkConf: +$conf + volumes: + - name: job-code + configMap: + name: $name-code + driver: + cores: 1 + memory: "1g" + serviceAccount: $SPARK_SA + volumeMounts: + - name: job-code + mountPath: /opt/spark/jobs + executor: + cores: 1 + instances: 2 + memory: "1g" + volumeMounts: + - name: job-code + mountPath: /opt/spark/jobs +YAML + + wait_sparkapp "$name" +} diff --git a/hadoop-spark-oci-sample/opensource/variables.tf b/hadoop-spark-oci-sample/opensource/variables.tf new file mode 100644 index 0000000..07ade6e --- /dev/null +++ b/hadoop-spark-oci-sample/opensource/variables.tf @@ -0,0 +1,342 @@ +# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved. +# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/ + +############################################################################### +# Input variables - Secure Hadoop & Spark on OKE +# +# Security posture is fixed and strict (see README.md). What is configurable is +# WHICH storage backends to deploy (HDFS, Object Storage, or both), sizing, and +# the container image source. +############################################################################### + +# --------------------------------------------------------------------------- +# Resource Manager / tenancy context (auto-populated by RM) +# --------------------------------------------------------------------------- +variable "tenancy_ocid" { + type = string + description = "OCID of the tenancy. Auto-populated by Resource Manager." +} + +variable "compartment_ocid" { + type = string + description = "OCID of the compartment where all resources are created." +} + +variable "region" { + type = string + description = "OCI region identifier. Auto-populated by Resource Manager." +} + +# --------------------------------------------------------------------------- +# General +# --------------------------------------------------------------------------- +variable "cluster_name" { + type = string + default = "bigdata" + description = "Short name used as a prefix for every resource." + + validation { + condition = can(regex("^[a-z][a-z0-9-]{1,24}$", var.cluster_name)) + error_message = "cluster_name must start with a letter, be 2-25 chars, lowercase letters/digits/hyphens." + } +} + +variable "admin_cidr" { + type = string + description = "The only network range allowed to reach the Kubernetes API endpoint and open OCI Bastion sessions. Set this to your own IP, e.g. 203.0.113.4/32." + + validation { + condition = can(cidrhost(trimspace(var.admin_cidr), 0)) + error_message = "admin_cidr must be a valid CIDR block." + } + validation { + condition = trimspace(var.admin_cidr) != "0.0.0.0/0" + error_message = "admin_cidr must not be 0.0.0.0/0. Restrict access to a specific network." + } +} + +variable "ssh_public_key" { + type = string + description = "SSH public key installed on the worker nodes (for break-glass access via the OCI Bastion)." + + validation { + condition = length(trimspace(var.ssh_public_key)) > 0 + error_message = "An SSH public key is required." + } +} + +# --------------------------------------------------------------------------- +# OKE cluster +# --------------------------------------------------------------------------- +variable "kubernetes_version" { + type = string + default = "v1.35.2" + description = "Kubernetes version for the OKE cluster and node pool. Must be a FULL version (vMAJOR.MINOR.PATCH) that OKE publishes - a partial version like 'v1.36' has no matching worker image. Default is the latest GA version; v1.36.0 exists but is a preview release (not for production, OC1 realm only). List valid versions with: oci ce cluster-options get --cluster-option-id all" + + validation { + condition = can(regex("^v[0-9]+\\.[0-9]+\\.[0-9]+$", var.kubernetes_version)) + error_message = "kubernetes_version must be a full version like v1.34.2 (vMAJOR.MINOR.PATCH). A partial version such as v1.36 has no matching OKE worker image." + } +} + +variable "cluster_endpoint_is_public" { + type = bool + default = true + description = "If true, the Kubernetes API endpoint is public but locked by NSG to admin_cidr (lets Terraform deploy the workload layer in one run). If false, the endpoint is fully private and the workload layer must be applied through the Bastion." +} + +# --------------------------------------------------------------------------- +# Worker node pool +# --------------------------------------------------------------------------- +variable "node_count" { + type = number + default = 3 + description = "Number of worker nodes in the node pool." + + validation { + condition = var.node_count >= 2 && var.node_count <= 100 + error_message = "node_count must be between 2 and 100." + } +} + +variable "node_shape" { + type = string + default = "VM.Standard.E5.Flex" + description = "Compute shape for the worker nodes." +} + +variable "node_ocpus" { + type = number + default = 4 + description = "OCPUs per worker node (flexible shapes only)." + + validation { + condition = var.node_ocpus >= 2 && var.node_ocpus <= 128 + error_message = "node_ocpus must be between 2 and 128." + } +} + +variable "node_memory_gbs" { + type = number + default = 64 + description = "Memory (GB) per worker node (flexible shapes only)." + + validation { + condition = var.node_memory_gbs >= 16 && var.node_memory_gbs <= 2048 + error_message = "node_memory_gbs must be between 16 and 2048." + } +} + +variable "node_boot_volume_gbs" { + type = number + default = 100 + description = "Boot volume size (GB) per worker node." + + validation { + condition = var.node_boot_volume_gbs >= 50 && var.node_boot_volume_gbs <= 32768 + error_message = "node_boot_volume_gbs must be between 50 and 32768." + } +} + +# --------------------------------------------------------------------------- +# Storage backends (deploy either, or both) +# --------------------------------------------------------------------------- +variable "deploy_hdfs" { + type = bool + default = true + description = "Deploy HDFS (Kerberos-secured NameNode + DataNode StatefulSets on the cluster)." +} + +variable "deploy_object_storage" { + type = bool + default = true + description = "Create an OCI Object Storage bucket and wire Spark to use it (oci:// paths) via OKE Workload Identity." +} + +variable "force_destroy_bucket" { + type = bool + default = true + description = "On `terraform destroy`, empty the data bucket (all objects AND versions) so it can be deleted - otherwise destroy fails with 409-BucketNotEmpty. Set false to protect bucket data from teardown." +} + +# --------------------------------------------------------------------------- +# HDFS sizing +# --------------------------------------------------------------------------- +variable "hdfs_replication" { + type = number + default = 3 + description = "HDFS block replication factor. Automatically capped to the DataNode count." + + validation { + condition = var.hdfs_replication >= 1 && var.hdfs_replication <= 10 + error_message = "hdfs_replication must be between 1 and 10." + } +} + +variable "hdfs_namenode_storage_gbs" { + type = number + default = 100 + description = "Persistent volume size (GB) for the HDFS NameNode metadata." + + validation { + condition = var.hdfs_namenode_storage_gbs >= 50 && var.hdfs_namenode_storage_gbs <= 32768 + error_message = "hdfs_namenode_storage_gbs must be between 50 and 32768." + } +} + +variable "hdfs_datanode_count" { + type = number + default = 3 + description = "Number of HDFS DataNode replicas (StatefulSet)." + + validation { + condition = var.hdfs_datanode_count >= 1 && var.hdfs_datanode_count <= 100 + error_message = "hdfs_datanode_count must be between 1 and 100." + } +} + +variable "hdfs_datanode_storage_gbs" { + type = number + default = 200 + description = "Persistent volume size (GB) for each HDFS DataNode." + + validation { + condition = var.hdfs_datanode_storage_gbs >= 50 && var.hdfs_datanode_storage_gbs <= 32768 + error_message = "hdfs_datanode_storage_gbs must be between 50 and 32768." + } +} + +# --------------------------------------------------------------------------- +# Spark +# --------------------------------------------------------------------------- +variable "deploy_spark" { + type = bool + default = true + description = "Deploy the Apache Spark Operator so Spark applications run natively on Kubernetes." +} + +variable "spark_operator_chart_version" { + type = string + default = "1.4.6" + description = "Version of the kubeflow spark-operator Helm chart." +} + +# --------------------------------------------------------------------------- +# In-cluster storage +# --------------------------------------------------------------------------- +variable "storage_class" { + type = string + default = "oci-bv" + description = "Kubernetes StorageClass for HDFS / KDC PersistentVolumes (oci-bv = OCI Block Volume CSI, available on OKE by default)." +} + +# --------------------------------------------------------------------------- +# Kerberos +# --------------------------------------------------------------------------- +variable "kerberos_realm" { + type = string + default = "HADOOP.INTERNAL" + description = "Kerberos realm. A KDC is deployed in-cluster when HDFS is enabled." + + validation { + condition = can(regex("^[A-Z][A-Z0-9.-]{2,48}$", var.kerberos_realm)) + error_message = "kerberos_realm must be uppercase letters/digits/dots/hyphens, e.g. HADOOP.INTERNAL." + } +} + +# --------------------------------------------------------------------------- +# Container images +# --------------------------------------------------------------------------- +variable "image_source" { + type = string + default = "upstream" + description = "Where container images come from: 'upstream' (public Apache images, one-click) or 'ocir' (custom hardened images you have pushed to OCI Registry)." + + validation { + condition = contains(["upstream", "ocir"], var.image_source) + error_message = "image_source must be 'upstream' or 'ocir'." + } +} + +# NOTE: OKE worker nodes run CRI-O with short-name mode = enforcing, which +# rejects unqualified image names. Always use a FULLY-QUALIFIED registry path +# (e.g. docker.io/apache/spark:3.5.3), including for image_source=ocir images. +variable "hadoop_image" { + type = string + default = "docker.io/apache/hadoop:3.3.6" + description = "Hadoop/HDFS container image (fully-qualified). With image_source=ocir, set this to your OCIR image." +} + +variable "spark_image" { + type = string + default = "docker.io/apache/spark:3.5.3" + description = "Spark container image (fully-qualified). With image_source=ocir, set this to your OCIR image." +} + +variable "kdc_image" { + type = string + default = "docker.io/library/oraclelinux:8" + description = "Base image for the Kerberos KDC pod (fully-qualified; krb5-server is installed at startup on the upstream path). With image_source=ocir, set this to a prebuilt KDC image." +} + +# --------------------------------------------------------------------------- +# Networking +# --------------------------------------------------------------------------- +variable "vcn_cidr" { + type = string + default = "10.20.0.0/16" + description = "CIDR block for the new VCN." + + validation { + condition = can(cidrhost(var.vcn_cidr, 0)) + error_message = "vcn_cidr must be a valid CIDR block." + } +} + +variable "endpoint_subnet_cidr" { + type = string + default = "10.20.0.0/28" + description = "CIDR block for the Kubernetes API endpoint subnet." + + validation { + condition = can(cidrhost(var.endpoint_subnet_cidr, 0)) + error_message = "endpoint_subnet_cidr must be a valid CIDR block." + } +} + +variable "nodes_subnet_cidr" { + type = string + default = "10.20.1.0/24" + description = "CIDR block for the private worker-node subnet." + + validation { + condition = can(cidrhost(var.nodes_subnet_cidr, 0)) + error_message = "nodes_subnet_cidr must be a valid CIDR block." + } +} + +variable "lb_subnet_cidr" { + type = string + default = "10.20.2.0/24" + description = "CIDR block for the private internal load-balancer subnet. OKE requires a service load-balancer subnet; this stack keeps it private and creates no public LBs." + + validation { + condition = can(cidrhost(var.lb_subnet_cidr, 0)) + error_message = "lb_subnet_cidr must be a valid CIDR block." + } +} + +# --------------------------------------------------------------------------- +# Software versions (used for image tags and in-cluster config) +# --------------------------------------------------------------------------- +variable "hadoop_version" { + type = string + default = "3.3.6" + description = "Apache Hadoop version (must match the hadoop_image tag)." +} + +variable "spark_version" { + type = string + default = "3.5.3" + description = "Apache Spark version (must match the spark_image tag)." +}