Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .github/workflows/build_hbase.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:
branches:
- main
tags:
- '\d+.\d+.\d+*'
- '[0-9]+.[0-9]+.[0-9]+*'
paths:
- hbase/**
- kubedoop-base/**
Expand Down Expand Up @@ -103,6 +103,15 @@ jobs:
action_debug: 'false' # enable debug for the build-and-push action if needed


- name: Run HBase lifecycle tests
env:
IMAGE: ${{ env.REGISTRY }}/hbase:${{ matrix.version }}-kubedoop${{ env.KUBEDOOP_VERSION }}-${{ matrix.arch }}
run: |
bash hbase/tests/source-entrypoint-framework.sh
bash hbase/tests/image-entrypoint-framework.sh "$IMAGE"
bash hbase/tests/image-native-launchers.sh "$IMAGE"
bash hbase/tests/image-hbase-lifecycle.sh "$IMAGE"

build-manifest:
name: Create manifest for ${{ matrix.version }}
runs-on: ubuntu-latest
Expand Down
12 changes: 9 additions & 3 deletions hbase/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@
<!-- Parent: ../AGENTS.md -->

## Overview
Apache HBase NoSQL database. Largest Dockerfile in the repo (291 lines) with 4 build stages, including a separate hbase-operator-tools build from a specific git commit SHA.
Apache HBase NoSQL database. Multi-stage source build including separate hbase-operator-tools from a specific git commit SHA.

## Build Details
- Build command: `make hbase-build`
- Build stages: (1) hbase-builder -- builds HBase from source with `-Dhadoop.profile=3.0`, (2) hbase-operator-tools-builder -- builds from specific git commit SHA, (3) hadoop -- copies from hadoop image, (4) hadoop-s3-builder -- extracts AWS JARs
- Build stages: (1) hbase-builder -- source with `-Dhadoop.profile=3.0`, (2) hbase-operator-tools-builder -- specific git commit SHA, (3) hadoop -- named source-built dependency, (4) hadoop-s3-builder -- AWS JARs, (5) tini-downloader -- checksum-pinned init, (6) final Java runtime
- Build system: Maven
- Product/operator-tools source downloads use bounded curl retries, gzip integrity validation and fail-fast pipelines, so truncated archives cannot continue into Maven.
- Runtime adds an independent checksum-pinned Tini downloader stage and product-local universal lifecycle. Source builders and native launchers remain unchanged.
- Base images: java-devel -> java-base

## Version Schema
Expand All @@ -26,6 +28,7 @@ See `versions.yaml` for current values. Structure:
| jmx-exporter | JMX exporter version |
| async-profiler | Async-profiler version |
| oauth2-proxy | OAuth2 proxy version |
| tini | Init version, with architecture-specific checksums |

## Kubedoop Customizations
### Patches (13 total across 3 components)
Expand All @@ -43,6 +46,9 @@ Note: JMX is conditional -- empty JMX_EXPORTER_VERSION for 2.6 which has native

### Custom Scripts
- `kubedoop/patches/apply_patches.sh` -- applies versioned patches during build
- `kubedoop/bin/entrypoint.sh`, `kubedoop/lib/{log,run,signal}.sh` -- root-owned hooks, retained stdin, stderr framework diagnostics, bounded TERM/INT shutdown and exit propagation.
- ENTRYPOINT is Tini plus the universal entrypoint; existing CMD `hbase master start` and UID 1001 are preserved. Native foreground paths already exec Java unless HBASE_NOEXEC is set; do not set it for a managed container main process.
- `tests/` -- source/image generic regressions, six native role paths, real standalone default-CMD data/identity persistence and restart/TERM/INT. Fixtures never enter production.

## Extras
- Downloads hbase-operator-tools from specific git commit SHA (not release tag)
Expand All @@ -57,7 +63,7 @@ Note: JMX is conditional -- empty JMX_EXPORTER_VERSION for 2.6 which has native
- Used by: spark-k8s (commented out, planned)

## Key Files
- `Dockerfile` -- 291 lines, 4 stages (hbase-builder, operator-tools-builder, hadoop, hadoop-s3-builder)
- `Dockerfile` -- product/tools source builders, Hadoop AWS dependency, Tini downloader and final runtime
- `kubedoop/patches/hbase/{ver}/` -- per-version HBase patches
- `kubedoop/patches/hbase-operator-tools/{ver}/` -- per-version operator-tools patches
- `kubedoop/jmx/config/` -- 4 JMX configs (master, regionserver, restserver, base)
Expand Down
40 changes: 38 additions & 2 deletions hbase/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ EOF
WORKDIR /build
COPY /kubedoop/patches /build/patches
RUN <<EOF
set -eo pipefail
mkdir -p /build/hbase-src
pushd /build/hbase-src

Expand All @@ -93,7 +94,10 @@ RUN <<EOF

# Download and extract source
echo "Downloading HBase from: ${SOURCE_URL}"
curl -sSfL "${SOURCE_URL}" | tar xzf - --strip-components=1
curl -sSfL --retry 3 --retry-all-errors --connect-timeout 20 --max-time 180 "${SOURCE_URL}" -o /tmp/hbase-source.tar.gz
gzip -t /tmp/hbase-source.tar.gz
tar xzf /tmp/hbase-source.tar.gz --strip-components=1
rm /tmp/hbase-source.tar.gz

# Apply patches if exists
PATCH_DIR="${PATCHES_PATH}/${PATCH_PATH}"
Expand Down Expand Up @@ -160,6 +164,7 @@ WORKDIR /build
COPY /kubedoop/patches /build/patches
# setup hbase-operator-tools
RUN <<EOF
set -eo pipefail
mkdir -p /build/hbase-operator-tools-src
pushd /build/hbase-operator-tools-src

Expand All @@ -182,7 +187,10 @@ RUN <<EOF

# Download and extract source
echo "Downloading hbase-operator-tools from: ${SOURCE_URL}"
curl -sSfL "${SOURCE_URL}" | tar xzf - --strip-components=1
curl -sSfL --retry 3 --retry-all-errors --connect-timeout 20 --max-time 180 "${SOURCE_URL}" -o /tmp/hbase-operator-tools-source.tar.gz
gzip -t /tmp/hbase-operator-tools-source.tar.gz
tar xzf /tmp/hbase-operator-tools-source.tar.gz --strip-components=1
rm /tmp/hbase-operator-tools-source.tar.gz

# Apply patches if exists
PATCHES_DIR="${PATCHES_PATH}/hbase-operator-tools/${PATCH_PATH}"
Expand Down Expand Up @@ -247,6 +255,22 @@ COPY --from=hadoop \
/kubedoop/hadoop/share/hadoop/tools/lib/


## stage: tini-downloader
FROM zncdatadev/image/java-devel AS tini-downloader
ARG TARGETARCH
ARG TINI_VERSION
RUN <<EOF
set -e
curl -sSfL -o /tmp/tini "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini-static-${TARGETARCH}"
case "${TARGETARCH}" in
amd64) expected_sha="c5b0666b4cb676901f90dfcb37106783c5fe2077b04590973b885950611b30ee" ;;
arm64) expected_sha="eae1d3aa50c48fb23b8cbdf4e369d0910dfc538566bfd09df89a774aa84a48b9" ;;
*) echo "Unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;;
esac
echo "${expected_sha} /tmp/tini" | sha256sum -c
chmod 0755 /tmp/tini
EOF

## stage: final
FROM zncdatadev/image/java-base

Expand Down Expand Up @@ -275,6 +299,17 @@ COPY --from=hbase-operator-tools-builder --chown=kubedoop:kubedoop /kubedoop/ /k
# setup hadoop-s3
COPY --from=hadoop-s3-builder --chown=kubedoop:kubedoop /kubedoop/ /kubedoop/

COPY --from=tini-downloader /tmp/tini /usr/bin/tini
COPY --chown=root:root kubedoop/lib/ /kubedoop/lib/
COPY --chown=root:root kubedoop/bin/entrypoint.sh /kubedoop/bin/entrypoint.sh
RUN chown root:kubedoop /kubedoop \
&& chmod 1770 /kubedoop \
&& chown root:root /kubedoop/bin /kubedoop/lib \
&& chmod -R 0755 /kubedoop/bin /kubedoop/lib \
&& mkdir -p /kubedoop/mount/pre-script /kubedoop/mount/post-script /kubedoop/run \
&& chown -R root:root /kubedoop/mount && chmod -R 0755 /kubedoop/mount \
&& chown kubedoop:kubedoop /kubedoop/run && chmod 1775 /kubedoop/run

RUN find /kubedoop/hadoop/share/ -name *.jar | grep 'aws' | xargs -I {} ln -s {} /kubedoop/hbase/lib/

ENV HBASE_HOME=/kubedoop/hbase
Expand All @@ -287,4 +322,5 @@ WORKDIR /kubedoop/hbase

USER kubedoop

ENTRYPOINT ["tini", "--", "/kubedoop/bin/entrypoint.sh"]
CMD ["hbase", "master", "start"]
2 changes: 2 additions & 0 deletions hbase/kubedoop/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
!/bin/
!/lib/
56 changes: 56 additions & 0 deletions hbase/kubedoop/bin/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/bin/bash
# /kubedoop/bin/entrypoint.sh -- Universal container entrypoint
#
# Note: Uses set -uo pipefail WITHOUT -e.
# Signal-aware scripts must not use set -e because trap handlers
# and set -e interact unpredictably. All error handling is explicit.
#
# Lifecycle logic (cleanup, signal handling) is in lib/signal.sh.
# This script is a thin orchestrator: config → load → pre-script → lifecycle → exit.
set -uo pipefail

# Paths are locked at build time — not overridable via environment variables
readonly KUBEDOOP_HOME="/kubedoop"
readonly KUBEDOOP_MOUNT_DIR="${KUBEDOOP_HOME}/mount"
readonly KUBEDOOP_RUN_DIR="${KUBEDOOP_HOME}/run"

# Graceful shutdown timeout (seconds). Validated and locked at startup.
KUBEDOOP_KILL_TIMEOUT="${KUBEDOOP_KILL_TIMEOUT:-30}"
if ! [[ "$KUBEDOOP_KILL_TIMEOUT" =~ ^[1-9][0-9]*$ ]]; then
echo "[kubedoop] [ERROR] KUBEDOOP_KILL_TIMEOUT must be a positive integer >= 1, got: $KUBEDOOP_KILL_TIMEOUT" >&2
exit 1
fi
readonly KUBEDOOP_KILL_TIMEOUT

# Ensure run directory exists with correct permissions
mkdir -p "${KUBEDOOP_RUN_DIR}"

# Load common libraries.
# Libraries load in glob sort order. If a library depends on another,
# it must document this requirement (e.g., signal.sh depends on log.sh and run.sh).
for lib in "${KUBEDOOP_HOME}/lib/"*.sh; do
if [[ -f "$lib" ]]; then
# shellcheck source=/dev/null
source "$lib"
fi
done

# Arguments may contain credentials; never include them in framework logs.
log_info "Entrypoint starting"

# --- Step 1: Operator pre-scripts (auto-discovered from mount) ---
# Note: pre-script failure prevents main process startup (fail-fast).
# Operators must test injected scripts in non-production environments first.
if ! run_pre_scripts; then
log_error "Pre-scripts failed, aborting startup"
exit 1
fi

# --- Step 2: Start main process and manage lifecycle ---
if [[ $# -eq 0 ]]; then
log_error "No command specified"
exit 1
fi

run_lifecycle "$@"
exit "$EXIT_CODE"
57 changes: 57 additions & 0 deletions hbase/kubedoop/lib/log.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/bin/bash
# /kubedoop/lib/log.sh -- Unified log output
#
# Provides structured logging with ISO 8601 timestamps.
# Log prefix is configurable via __KUBEDOOP_LOG_PREFIX (default: "kubedoop").
# Debug output is gated by KUBEDOOP_DEBUG=true.

__KUBEDOOP_LOG_PREFIX="${__KUBEDOOP_LOG_PREFIX:-kubedoop}"

# Internal log formatter. Strips control characters to prevent log injection.
#
# Arguments:
# $1 - Log level (INFO, WARN, ERROR, DEBUG)
# $2.. - Message strings
#
# Output:
# stdout - Formatted log line: "[TIMESTAMP] [PREFIX] [LEVEL] MESSAGE"
_kubedoop_log() {
local level="$1"; shift
local timestamp
timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
# Sanitize: strip control characters to prevent log injection
local msg
msg=$(printf '%s' "$*" | tr -d '[:cntrl:]')
printf "[%s] [%s] [%s] %s\n" "$timestamp" "$__KUBEDOOP_LOG_PREFIX" "$level" "$msg"
}

# Log at INFO level to stderr, preserving the main command's stdout.
#
# Arguments:
# $@ - Message strings
log_info() { _kubedoop_log "INFO" "$@" >&2; }

# Log at WARN level to stderr.
#
# Arguments:
# $@ - Message strings
log_warn() { _kubedoop_log "WARN" "$@" >&2; }

# Log at ERROR level to stderr.
#
# Arguments:
# $@ - Message strings
log_error() { _kubedoop_log "ERROR" "$@" >&2; }

# Log at DEBUG level to stderr. No-op unless KUBEDOOP_DEBUG=true.
#
# Arguments:
# $@ - Message strings
#
# Returns:
# 0 - Always (no error when debug is disabled)
log_debug() {
if [[ "${KUBEDOOP_DEBUG:-false}" == "true" ]]; then
_kubedoop_log "DEBUG" "$@" >&2
fi
}
Loading
Loading